Learning Scalable ℓ∞-constrained Near-lossless Image Compression via Joint Lossy Image and Residual Compression
Yuanchao Bai1, Xianming Liu1,2,∗ Wangmeng Zuo1,2, Yaowei Wang1, Xiangyang Ji3 1Peng Cheng Laboratory, 2Harbin Institute of Technology, 3Tsinghua University {baiych,wangyw}@pcl.ac.cn, {csxm,wmzuo}@hit.edu.cn, [email protected]
Abstract of the decoded images, ℓ∞-constrained near-lossless image compression is developed [6, 15, 2, 47, 51] and standardized We propose a novel joint lossy image and residual in traditional codecs, e.g., JPEG-LS [47] and CALIC [51]. compression framework for learning ℓ∞-constrained near- Different from lossy image compression with Peak Signal- lossless image compression. Specifically, we obtain a lossy to-Noise Ratio (PSNR) or Multi-Scale Structural SIMilar- reconstruction of the raw image through lossy image com- ity index (MS-SSIM) [45, 46] distortion measures, ℓ∞- pression and uniformly quantize the corresponding resid- constrained near-lossless image compression requires the ual to satisfy a given tight ℓ∞ error bound. Suppose that maximum reconstruction error of each pixel to be no larger the error bound is zero, i.e., lossless image compression, than a given tight numerical bound. Lossless image com- we formulate the joint optimization problem of compress- pression is a special case of near-lossless image compres- ing both the lossy image and the original residual in terms sion, when the tight error bound is zero. of variational auto-encoders and solve it with end-to-end With the fast development of deep neural networks training. To achieve scalable compression with the error (DNNs), learning-based lossy image compression [40, 3, bound larger than zero, we derive the probability model 39, 41, 34, 24, 4, 31, 27, 22, 8, 7, 23, 25, 29] has achieved of the quantized residual by quantizing the learned prob- tremendous progress over the last four years. Most re- ability model of the original residual, instead of training cent methods for lossy image compression adopt variational multiple networks. We further correct the bias of the de- auto-encoder (VAE) architecture [17, 18] based on trans- rived probability model caused by the context mismatch be- form coding [11], where the rate is modeled by the entropy tween training and inference. Finally, the quantized resid- of quantized latent variables and the reconstruction distor- ual is encoded according to the bias-corrected probability tion is measured by PSNR or MS-SSIM. Through end-to- model and is concatenated with the bitstream of the com- end rate-distortion optimization, the state-of-the-art meth- pressed lossy image. Experimental results demonstrate that ods, such as [7], can achieve comparable performance with our near-lossless codec achieves the state-of-the-art per- the lastest compression standard Versatile Video Coding formance for lossless and near-lossless image compression, (VVC) [32]. However, the above transform coding scheme and achieves competitive PSNR while much smaller ℓ∞ er- cannot be directly employed on near-lossless image com- ror compared with lossy image codecs at high bit rates. pression, because it is difficult for DNN-based transforms to satisfy a tight bound on the maximum reconstruction error of each pixel, even without quantization. Invertible trans- 1. Introduction forms, such as integer discrete flow [13] or wavelet-like transform [25], are possible solutions to lossless compres- Image compression is a ubiquitous technique in com- sion but not to general near-lossless compression. puter vision. For certain applications with stringent de- In this paper, we propose a new joint lossy image and mands on image fidelity, such as medical imaging or image residual compression framework for learning near-lossless archiving, the most reliable choice is lossless image com- image compression, inspired by the traditional “lossy plus pression. However, the compression ratio of lossless com- residual” coding scheme [10, 26, 2]. Specifically, we obtain pression is upper-bounded by Shannon’s source coding the- a lossy reconstruction of the raw image through lossy im- orem [36], and is typically between 2:1 and 3:1 for practical age compression and uniformly quantize the corresponding lossless image codecs [50, 47, 37, 14, 5, 38]. To improve residual to satisfy the given error bound . Suppose that the compression performance while keeping the reliability ℓ∞ τ the error bound τ is zero, i.e., lossless image compression, ∗Corresponding author we formulate the joint optimization problem of compress-
11946 ing both the lossy image and the original residual in terms of further compress residuals, which is a special case of our VAEs [17, 18], and solve it with end-to-end training. Note framework. Beyond [30], our lossy image compressor and that our VAE model is novel, different from transform cod- residual compressor are jointly optimized through end-to- ing based VAEs [3, 39, 4] for simply lossy image compres- end training and are interpretable as a VAE model. sion or bits-back coding based VAEs [42, 19] for lossless Near-lossless Image Compression. Basically, previ- image compression. ous methods for near-lossless image compression can be To achieve scalable near-lossless compression with error divided into three categories: 1) pre-quantization: adjust- bound τ > 0, we derive the probability model of the quan- ing raw pixel values to the ℓ∞ error bound, and then com- tized residual by quantizing the learned probability model pressing the pre-processed images with lossless image com- of the original residual at τ = 0, instead of training mul- pression, e.g., near-lossless WebP [14]; 2) predictive cod- tiple networks. Because residual quantization leads to the ing: predicting subsequent pixels based on previously en- context mismatch between training and inference, we fur- coded pixels, then quantizing predication residuals to sat- ther propose a bias correction scheme to correct the bias of isfy the ℓ∞ error bound, and finally compressing the quan- the derived probability model. An arithmetic coder [48] is tized residuals, e.g., [6, 15], near-lossless JPEG-LS [47] adopted to encode the quantized residual according to the and near-lossless CALIC [51]; 3) lossy plus residual cod- bias-corrected probability model. Finally, the near-lossless ing: similar to 2), but replacing predictive coder with lossy compressed image is stored including the bitstreams of the image coder, and both the lossy image and the quantized encoded lossy image and the quantized residual. residual are encoded, e.g., in [2]. Our main contributions are summarized as follows: In this paper, we propose a joint lossy image and residual • We propose a joint lossy image and residual compression compression framework for learning-based near-lossless framework to realize learning-based lossless and near- image compression, inspired by lossy plus residual cod- lossless image compression. The framework is inter- ing. To the best of our knowledge, we propose the first preted as a VAE model and can be end-to-end optimized. deep-learning-based near-lossless image codec. Recently, • We realize scalable compression by deriving the proba- CNN-based soft-decoding methods [52, 53] are proposed bility model of the quantized residual from the learned to improve the reconstruction performance of near-lossless probability model of the original residual, instead of CALIC [51]. However, these methods should belong to im- training multiple networks. A bias correction scheme fur- age reconstruction rather than image compression. ther improves the compression performance. • Our codec achieves the state-of-the-art performance 3. Scalable Near-lossless Image Compression for lossless and near-lossless image compression, and achieves competitive PSNR while much smaller ℓ∞ er- 3.1. Overview of Compression Framework ror compared with lossy image codecs at high bit rates. Given a tight ℓ∞ bound τ ∈ {0, 1, 2,...}, near-lossless codecs compress a raw image satisfying the following dis- 2. Related Work tortion measure: Learning-based Lossy Image Compression. Early Dnll(x, xˆ)= kx − xˆk∞ = max |xi,c − xˆi,c|≤ τ (1) learning-based methods [40, 41] for lossy image compres- i,c sion are based on recurrent neural networks (RNNs). Fol- lowing [3, 39], most recent learned methods [34, 24, 4, 31, where x and xˆ are a raw image and its near-lossless recon- 27, 22, 8, 7, 23, 25, 29] adopt convolutional neural networks structed counterpart. xi,c and xˆi,c are the pixels of x and xˆ, (CNNs) and can be interpreted as VAEs [17, 18] based on respectively. i denotes the i-th spatial position in 2D raster- transform coding [11]. In our joint lossy image and residual scan order, and c denotes the c-th channel. compression framework, we take advantage of the advanced In order to realize the ℓ∞ error bound (1), we propose transform (network structures), quantization and entropy a near-lossless image compression framework by integrat- coding techniques in the existing learning-based methods ing lossy image compression and residual compression. We for lossy image compression. first obtain a lossy reconstruction x˜ of the raw image x Learning-based Lossless Image Compression. Given through lossy image compression. Lossy image compres- the strong connections between lossless compression and sion methods, such as traditional [44, 37, 14, 5] or learned unsupervised learning, auto-aggressive models [33, 43, 35], methods [24, 4, 31, 27, 22, 8, 7, 23, 25], can achieve high flow models [13], bits-back coding based VAEs [42, 19] and PSNR results at relatively low bit rates, but it is difficult for other specific models [28, 30, 25], are introduced to approx- these methods to ensure a tight error bound τ of each pixel imate the true distribution of raw images for entropy cod- in x˜. We then compute the residual r = x − x˜ and suppose ing. Previous work [30] uses traditional BPG lossy image that r is quantized to ˆr. Let xˆ = x˜ + ˆr, the reconstruction codec [5] to compress raw images and proposes a CNN to error x − xˆ is equivalent to the quantization error r −ˆr of r.
11947 Thus, we adopt a uniform residual quantizer whose bin size where yˆ is quantized from continuous y and y is trans- is 2τ + 1 and quantized value is [47, 51]: formed from x. Like [4], we relax the quantization of y 1 1 by adding noise from U(− 2 , 2 ), and assume qφ(yˆ|x) = rˆi,c = sgn(ri,c)(2τ + 1)⌊(|ri,c| + τ)/(2τ + 1)⌋ (2) 1 1 i,c U(yi,c− 2 ,yi,c+ 2 ). Thus, log qφ(yˆ|x) = 0 is dropped from (4). For simply lossy image compression, such as where sgn(·) denotes the sign function. ri,c and rˆi,c are the Q[4, 31, 8, 7], the first and second terms of (4) are considered elements of r and r, respectively. With (2), we now have ˆ as the distortion loss and the rate loss, respectively. Only yˆ r |ri,c − rˆi,c| ≤ τ for each rˆi,c in ˆ, satisfying the tight error needs to be encoded. r bound (1). Finally, we encode ˆ and concatenate it with the Beyond lossy image compression, we further take resid- x compressed ˜, and send them to the decoder. ual compression into consideration. For each x and all To compress both the lossy image x and the quantized ˜ (x˜, r) pairs satisfying x˜ + r = x, we have p (x|yˆ) = residual r effectively leads to a challenging joint optimiza- θ ˆ p (x˜, r|yˆ) ≥ p (x˜, r|yˆ). Hence, we substitute tion problem. In the following subsections, we first propose x˜+r=x θ θ pθ(x˜, r|yˆ) for pθ(x|yˆ), leading to the upper bound of (4). an end-to-end trainable VAE model [17, 18] to solve the P Let pθ(x˜, r|yˆ)= pθ(x˜|yˆ) · pθ(r|x˜, yˆ), we have: joint optimization problem with τ = 0. Then, we propose a scalable compression scheme with bias correction for τ > 0 ✘✘✘✿ 0 based on the learned VAE model. E E −log✘p✘(x˜|yˆ) − log p (r|x˜, yˆ) − log p (yˆ) p(x) qφ(yˆ|x) ✘ θ θ θ R Rˆ 3.2. Joint Lossy Image & Residual Compression r y (5) Before addressing the joint compression of the lossy im- The lossy reconstruction x˜ is computed| {z by the} inverse| {z trans-} age x˜ and the quantized residual ˆr with variable τ, we first form of yˆ and quantization. As the quantization is re- solve the special case of near-lossless image compression laxed by adding uniform noise from U(− 1 , 1 ), we have with , i.e., lossless image compression. 2 2 τ = 0 log pθ(x˜|yˆ) = 0. The second term Rr and the third term Problem Formulation. Assuming that raw images are Ryˆ of (5) are the rates of encoding r and yˆ, respectively. sampled from an unknown probability distribution p(x), Notice that no distortion loss of lossy image compression the compression performance of lossless image compres- is specified in (5). Therefore, we can embed arbitrary lossy sion depends on how well we can approximate p(x) with image compressors and optimize (5) to achieve lossless im- an underlying model pθ(x). We adopt the latent vari- age compression. A special case is the previous lossless able model which is formulated by a marginal distribution image compression method [30], in which the BPG lossy pθ(x) = pθ(x, y)dy. y is an unobserved latent variable image compressor [5] with a learned quantization parameter and θ denote the parameters of this model. classifier optimizes − log pθ(yˆ) and a CNN-based residual Since directlyR learning the marginal distribution x is pθ( ) compressor optimizes − log pθ(r|x˜). typically intractable, one alternative way is to optimize the Network Design and Optimization. To realize (5), we evidence lower bound (ELBO) via VAEs [17, 18]. By intro- propose our network architecture in Fig. 1(a). For lossy ducing an inference model qφ(y|x) to approximate the pos- image compression, we employ the hyper-prior model pro- terior pθ(y|x), we can rewrite the logarithm of the marginal posed in [4], where side information zˆ is extracted to model likelihood pθ(x) as: the distribution of yˆ. We employ the residual and attention blocks similar to [7, 12] in the encoder and decoder. De- E pθ(x, y) E qφ(y|x) log pθ(x)= qφ(y|x) log + qφ(y|x) log tailed structure of the lossy image compressor is described qφ(y|x) pθ(y|x) in the supplementary material. Ryˆ is thus extended by ELBO Dkl(q (y|x)||p (y|x)) φ θ E E y z z (6) (3) Ryˆ,zˆ = p(x) qφ(yˆ,zˆ|x) [− log pθ(ˆ|ˆ) − log pθ(ˆ)] where D (·||·|) is the{z Kullback-Leibler} | (KL){z divergence.} kl where Ryˆ,zˆ is the cost of encoding both yˆ and zˆ. φ denote the parameters of the inference model y x . qφ( | ) Denote by u = gu(yˆ), where the feature u is extracted Since D (q (y|x)||p (y|x)) ≥ 0 and log p (x) ≤ 0, kl φ θ θ from yˆ by gu(·). Denote by x˜ = gx˜(yˆ), where the lossy ELBO is the lower bound of log p (x). θ reconstruction x˜ is inversely transformed from yˆ by gx˜(·). From (3), we can minimize the expectation of neg- gu and gx˜ share the decoder, and only the last convolutional ative ELBO as a proxy for the expected codelength layers are different, as shown in Fig. 1(a). We interpret u as E p(x)[− log pθ(x)]. In our compression framework, we first the feature of the residual r given x˜ and yˆ (u shares the adopt lossy image compression based on transform coding same height and width with r and has 64 channels). As [11], and the expectation of negative ELBO can be refor- r = x−x˜, we can compute r and further consider the causal mulated as follows: context Cr of r during encoding. Rr is thus extended by E E E E p(x) qφ(yˆ|x) [− log pθ(x|yˆ) − log pθ(yˆ)] (4) Rr = p(x) qφ(yˆ,zˆ|x) [− log pθ(r|u,Cr)] (7)
11948 Figure 1. (a) Overview of network architecture. (b) Probability inference of the quantized residual ˆr without bias correction. (c) Probability inference of the quantized residual ˆr with bias correction. (d) Legend with descriptions for notations in (a)-(c).
and reformulate pθ(ri,1,ri,2,ri,3|ui,Cri ) as
pθ(ri,1,ri,2,ri,3|ui,Cri )= pθ(ri,1|ui,Cri )· (9)
pθ(ri,2|ri,1,ui,Cri ) · pθ(ri,3|ri,1,ri,2,ui,Cri ) As shown in Fig. 2, we introduce a sub-network to estimate k entropy parameters, including mixture weights πi,c, means k k µi,c, variances σi,c and mixture coefficients βi,t. We choose (a) (b) a mixture of K = 5 logistic distributions. k denotes the u Figure 2. Entropy parameter estimation sub-network. Given index of the k-th logistic distribution. t denotes the channel and Cr, (a) estimates parameters of discrete logistic mixture like- index of β. The channel autoregression over ri, ,ri, ,ri, lihoods corresponding to entropy model in Fig. 1(a). (b) is the 1 2 3 is implemented by updating the means using: detailed structure of estimator in (a). All 1 × 1 convolutional lay- ers except the last layer have 128 channels. The last convolutional k k k k µ˜i,1 = µi,1, µ˜i,2 = µi,2 + βi,1 · ri,1, layer has 3 · K channels. k k µ˜i,3 = µi,3 + βi,2 · ri,1 + βi,3 · ri,2 (10)
k k k With πi,c, µ˜i,c and σi,c, we have where the causal context Cri ∈ Cr is shared by ri,c of all K channels. Cri is extracted from r ri,c, we evaluate pθ(ri,c|ri, pθ(r|u,Cr)= pθ(ri,1,ri,2,ri,3|ui,Cri ) (8) K + k − k i r − µ˜i,c r − µ˜i,c Y πk S i,c − S i,c (12) i,c σk σk k=1 " i,c ! i,c !# We further model the probability mass function (PMF) of X + the residual r with discrete logistic mixture likelihood [35, where S(·) denotes the sigmoid function. ri,c = ri,c + 0.5 − 28, 30]. We use a channel autoregression over ri,1,ri,2,ri,3 and ri,c = ri,c − 0.5. 11949 Figure 3. PMF quantization corresponding to residual quantization (2) with τ = 1. Each red number is the value of the quantized residual rˆi,c. The probability of each quantized value is the sum of the probabilities of the values in the same bin. Figure 4. Conditional convolutional layer for bias correction. Dif- Besides rate terms Ryˆ,zˆ and Rr, we add a distortion term E E 2 ferent outputs can be produced conditioned on τ ∈{1, 2,..., 5}. Dls(x, x˜) = p(x) i,c(xi,c − x˜i,c) to minimize the mean square error (MSE) between raw image x and lossy recon- struction x˜. As discussed in [4], minimizing MSE loss is However, encoding ˆr with pˆθ(ˆr|u,Cr) results in unde- equivalent to learning a lossy image compressor that fits codable bitstreams, since the original residual r is unknown residual r to a zero-mean factorized Gaussian distribution. to the decoder. pˆθ(ˆri,c|ri, 11950 an approximate KL-divergence or relative entropy [9] be- Codec Kodak DIV2K CLIC.p CLIC.m tween pθ(r|u,Cr) and pϕ(r|u,Cˆr). Because the entropy PNG 4.35 4.23 3.93 3.93 model receives the context Cr extracted from the original JPEG-LS 3.16 2.99 2.82 2.53 residual r and is only trained with τ = 0, pθ(r|u,Cr) CALIC 3.18 3.07 2.87 2.59 approximates the true distribution p(r|x˜, yˆ) better than JPEG2000 3.19 3.12 2.93 2.71 pϕ(r|u,Cˆr). Thus, − log pθ(r|u,Cr) is the lower bound WebP 3.18 3.11 2.90 2.73 of − log pϕ(r|u,Cˆr) on average. BPG 3.38 3.28 3.08 2.84 The probability inference scheme with bias correction FLIF 2.90 2.91 2.72 2.48 is sketched in Fig. 1(c). For τ = 0, we choose the en- L3C − 3.09 2.94 2.64 RC − 3.08 2.93 2.54 tropy model to estimate p (r|u,C ) to encode r. For θ r Ours 3.04 2.81 2.66 2.51 τ ∈ {1, 2,..., 5}, we choose the conditional entropy model to estimate p (r|u,C ) and derive pˆ (ˆr|u,C ) with ϕ ˆr ϕ ˆr Table 1. Compression performance (bpsp) of the proposed near- (14) to encode ˆr. As pˆϕ(ˆr|u,Cˆr) approximates the ideal lossless image codec with τ =0, compared to other lossless image pˆθ(ˆr|u,Cr) better than the biased pˆθ(ˆr|u,Cˆr), the com- codecs on Kodak, DIV2K, CLIC.p and CLIC.m datasets. pression performance can be improved. Since evaluating pˆ (ˆr|u,C ) is independent of r, the resulting bitstreams ∗ ϕ ˆr Codec τ Kodak DIV2K CLIC.p CLIC.m are decodable. We analyze the efficacy of the proposed bias 1 2.90 2.62 2.34 2.44 correction scheme in Sec. 4.4. JPEG- 2 2.30 2.07 1.80 1.89 Training Strategy. LS Bias corrector is trained together 4 1.68 1.53 1.28 1.35 with the lossy image compressor and the residual compres- 1 2.75 2.45 2.18 2.28 sor, but minimizing (15) only updates the parameters of CALIC 2 2.14 1.88 1.62 1.70 the conditional entropy model as shown in Fig. 1(a). The 4 1.51 1.31 1.07 1.13 masked convolutional layer is shared with that in the resid- 1 2.41 2.45 2.26 2.11 WebP ual compressor, thus can be updated by minimizing (13). 2 2.01 2.04 1.89 1.85 nll This leads to three advantages: 1) achieving the target con- 4 1.82 1.83 1.73 1.75 1 1.84 1.72 1.60 1.53 ditional entropy model; 2) circumventing the relaxation of Ours 2 1.38 1.29 1.15 1.10 residual quantization; 3) avoiding degrading the estimation w/ bc 4 0.92 0.89 0.74 0.72 of pθ(r|u,Cr) caused by training with mixed τ. Table 2. Compression performance (bpsp) of the proposed near- 4. Experiments lossless image codec (w/ bias correction) with τ > 0, compared 4.1. Experimental Setup to near-lossless JPEG-LS, near-lossless CALIC and near-lossless WebP on Kodak, DIV2K, CLIC.p and CLIC.m datasets. ∗The er- Training. We train on DIV2K high resolution train- ror bounds of near-lossless WebP are powers of two. ing dataset [1] consisting of 800 2K resolution color im- ages. Although DIV2K is originally built for image super- resolution task, it contains large numbers of high-quality consists of 61 color images taken using mobile phones. Im- images that is suitable for training our near-lossless codec. ages in CLIC.p and CLIC.m are typically 2K resolution but The images are first cropped into 27958 samples with the some of them are of small sizes. size of 256 × 256. To augment training data and prevent 4.2. Performance Evaluation at τ = 0 overfitting on noise, we use bicubic method to downscale each sample with a random factor selected from [0.6, 1.0] We first evaluate the compression performance of the and further randomly crop the downscaled sample to 128 × proposed near-lossless image codec at τ = 0, measured 128 during training. Our near-lossless codec is optimized by bits per subpixel (bpsp). Each RGB pixel has 3 sub- for 400 epochs using Adam [16] with minibatches of size pixels. We compare with seven traditional lossless image 16. The learning rate is set to 1 × 10−4 for the first 350 codecs, i.e., PNG, JPEG-LS [47], CALIC [50], JPEG2000 epochs and decays to 1×10−5 for the remaining 50 epochs. [37], WebP [14], BPG [5], FLIF [38], and two the state-of- Evaluation. We evaluate our near-lossless codec on four the-art learned lossless image codecs, i.e., L3C [28] and RC datasets: 1) Kodak dataset [20] consists of 24 uncompressed [30]. Unlike ours, L3C [28] and RC [30] are trained on a 768 × 512 color images. 2) DIV2K high resolution val- much larger dataset (300,000 images from the Open Images idation dataset [1] consists of 100 2K resolution color im- dataset [21]). We report the compression performance of ages. 3) CLIC professional validation dataset (CLIC.p) [49] L3C [28] and RC [30] published by their authors. consists of 41 color images taken by professional photogra- As reported in Table 1, our codec achieves the best per- phers. 4) CLIC mobile validation dataset (CLIC.m) [49] formance on DIV2K validation dataset, which shares the 11951 ideal JPEG BPG (4:2:0) Balle)-hyper JPEG Lossy FLIF WebP Lossy FLIF Minnen-joint JPEG-LS VVC w/ bc JPEG2000 CALIC WebP Balle)-hyper ∞ WebP nll w/o bc L Minnen-joint BPG (4:4:4) Ours w/ bc BPG (4:2:0) τ JPEG-LS JPEG2000 WebP nll BPG (4:4:4) Lossy FLIF VVC CALIC Ours w/ bc ∞ L Figure 7. Compression performance of the proposed near-lossless (a) (b) image codec without bias correction, with bias correction and with Figure 5. Rate-distortion curves of the proposed near-lossless im- the ideal pˆθ(ˆr|u,Cr) on Kodak dataset. age codec (w/ bias correction) with τ > 0 compared to other near- lossless image codecs and lossy image codecs on Kodak dataset. (a) bpsp-ℓ∞ error curves. (b) bpsp-PSNR curves. Near-lossless WebP simply adjusts pixel values to the ℓ∞ error bound τ and compresses the pre-processed images 4 5 2 3 1 Ry,̂ ẑ Rr̂ − Rr̂ Rr̂ − Rr̂ Rr − Rr̂ losslessly. Near-lossless JPEG-LS and CALIC adopt pre- R5 R3 − R4 R1 − R2 r̂ r̂ r̂ r̂ r̂ dictive coding schemes, and encode the prediction residuals quantized by (2). The predictors and the probability mod- els used in near-lossless JPEG-LS and CALIC are hand- crafted, which are not effective enough. More effectively, our near-lossless codec is based on jointly trained lossy im- age compressor and residual compressor. We employ (2) to realize the error bound τ and the probability models of the quantized residuals can be derived from the learned residual compressor with τ = 0. Therefore, our codec outperforms near-lossless JPEG-LS, CALIC and WebP by a wide mar- gin, as shown in Table 2. The rate-distortion performance of all codecs on Kodak Figure 6. Bit rates of each image compressed by the proposed near- dataset is reported in Fig. 5. In terms of ℓ∞ error defined lossless image codec with τ = {0, 1,..., 5} on Kodak dataset. in (1), our codec consistently yields the best results among all codecs, as shown in Fig. 5(a). Besides ℓ∞ error, we also compare PSNR of all codecs, as shown in Fig. 5(b). Our same domain with the training dataset. Our codec achieves codec is better than or comparable with JPEG, near-lossless the best performance on CLIC.p, and the second best per- JPEG-LS, near-lossless CALIC, WebP, near-lossless WebP, formance on Kodak and CLIC.m, only secondary to FLIF. BPG (4:2:0) and lossy FLIF. Our codec is surpassed by The results demonstrate that the jointly learned lossy im- JPEG2000, BPG (4:4:4), VVC, Balle-hyper´ and Minnen- age compressor and residual compressor with τ = 0 can joint. Overall, our codec achieves competitive performance effectively model pθ(x) and can be generalized to various at bit rates higher than 0.8 bpsp, although PSNR is not our domains of natural images. This provides a solid foundation optimization objective. for the near-lossless image compression with τ > 0. 4.4. Ablation Study 4.3. Performance Evaluation at τ > 0 Scalability. In Fig. 6, we show the bit rates of each im- We next evaluate the compression performance of the age compressed by the proposed near-lossless image codec proposed near-lossless image codec (with bias correction) with τ = {0, 1,..., 5} on Kodak dataset. Ryˆ,zˆ, on aver- at τ > 0. We compare with near-lossless JPEG-LS [47], age, accounts for about 18% of Ryˆ,zˆ + Rr at τ = 0. With τ near-lossless CALIC [51] and near-lossless WebP (WebP the increase of τ, the bit rate Rˆr of the quantized residual ˆr nll) [14]. Besides, we compare with six traditional lossy is significantly reduced. Especially the τ = 1 mode saves image codecs, i.e., JPEG [44], JPEG2000 [37], WebP [14], about 40% bit rates compared with the τ = 0 lossless image BPG [5], lossy FLIF [38] and VVC [32]. Recent learned compression mode. lossy image codecs published online are trained at relatively Bias Correction. In Fig. 7, we demonstrate the efficacy low bit rates (≤ 2 bpp ≈ 0.67 bpsp on Kodak). We re- of bias correction at τ > 0. Because of the discrepancy implement and train two recent learned lossy image codecs, between the ideal pˆθ(ˆr|u,Cr) and the biased pˆθ(ˆr|u,Cˆr), i.e., Balle-hyper´ [4] and Minnen-joint [31], at high bit rates encoding ˆr with pˆθ(ˆr|u,Cˆr) (without bias correction) de- (≥ 0.8 bpsp on Kodak) for comparison. grades the compression performance. Instead, we encode ˆr 11952 (a) Raw Image (b) lossy x˜ (λ =0) (c) lossy x˜ (λ =0.03) (d) nll xˆ (λ,τ =0, 5) (e) nll xˆ (λ,τ =0.03, 5) (f) Raw Image (g) lossy x˜ (λ =0.01) (h) lossy x˜ (λ =0.03) (i) nll xˆ (λ,τ =0.01, 5) (j) nll xˆ (λ,τ =0.03, 5) Figure 8. Visualizing the effects of different λ’s on the lossy reconstruction x˜ and the near-lossless (nll) reconstruction xˆ. The images are best viewed in full size on screen. λ Total Rate Ryˆ,zˆ Rr x˜ (PSNR) dB, the magnitude of r = x − x˜ is large. To quantize such 0 2.93 0.05 2.88 24.09 large residuals with large τ results in blocking artifacts in 0.01 3.00 0.42 2.58 37.55 the near-lossless reconstruction xˆ. As shown in Fig. 8(b), x˜ 0.03 3.04 0.55 2.49 39.29 with λ = 0 is blurry. If τ = 5, the quantization bin size is 0.05 3.06 0.62 2.44 39.91 large, i.e., 2τ +1 = 11, resulting in blocking artifacts in xˆ in Fig. 8(d). For λ = 0.01, we also observe small artifacts in Table 3. The effects of different λ’s on the learned near-lossless xˆ caused by quantization with large τ, as shown in Fig. 8(i). image codec with τ =0 on Kodak dataset. In our codec, we choose λ = 0.03, which is the best trade- off between compression performance and visual quality in the experiments. As shown in Fig. 8(e) and Fig. 8(j), we with pˆϕ(ˆr|u,Cˆr) (with bias correction) resulting in lower observe no artifacts in xˆ compared with λ = 0, 0.01. bit rates. With the increase of τ, the PMF of r is quantized by larger bins and becomes coarser. Thus, the compression 5. Conclusion performance with bias correction approaches the “ideal”. In this paper, we propose a joint lossy image and residual However, the gap between the compression performance compression framework to learn a scalable ℓ -constrained without bias correction and the “ideal” remains large, as the ∞ near-lossless image codec. With our codec, a raw image discrepancy between ˆr u and ˆr u is also pˆθ( | ,Cr) pˆθ( | ,Cˆr) is first compressed by the lossy image compressor, and the magnified with the increasing . Note that the compression τ corresponding residual is then quantized to satisfy variable performance of our codec without bias correction is also ℓ error bounds. To compress the variably quantized resid- better than near-lossless JPEG-LS, CALIC and WebP. ∞ uals, the probability models of the quantized residuals are Discussion on different λ’s. The λ in (13) adapts the derived from the learned probability model of the original rate of losslessly encoding the raw image x and the distor- residual, which is realized by the residual compressor in tion of the lossy reconstruction x˜. In Table 3, we evaluate conjunction with the bias corrector. Experimental results the effects of different λ ∈ {0, 0.01, 0.03, 0.05} on Kodak demonstrate the state-of-the-art performance of our near- dataset. With the increase of λ, the PSNR of x˜ is improved lossless image codec. but the bpsp of encoding x also becomes higher. If we simply do lossless image compression (τ = 0), we should Acknowledgement choose λ = 0. The x˜ is learned by the network without any hand-crafted constraints, leading to the best compres- This work was supported by National Natural Science Founda- tion of China under Grants 61922027, 61827804 and U20B2052, sion performance. National Key Research and Development Project under Grant However, λ = 0 is not suitable for our near-lossless 2019YFE0109600, China Postdoctoral Science Foundation under codec with τ > 0. As the average PSNR of the x˜ is 24.09 Grant 2020M682826. 11953 References [18] Diederik P. Kingma and Max Welling. An introduction to variational autoencoders. Foundations and Trends in Ma- [1] Eirikur Agustsson and Radu Timofte. NTIRE 2017 chal- chine Learning, 12(4):307–392, 2019. 1, 2, 3 lenge on single image super-resolution: dataset and study. In [19] Friso H Kingma, Pieter Abbeel, and Jonathan Ho. Bit-swap: IEEE Conf. Comput. Vis. Pattern Recog. Worksh., July 2017. Recursive bits-back coding for lossless compression with hi- 6 erarchical latent variables. In Int. Conf. Mach. Learn., 2019. [2] Rashid Ansari, Nasir D. Memon, and Ersan Ceran. Near- 2 lossless image compression techniques. Journal of Elec- [20] Eastman Kodak. Kodak lossless true color image suite (pho- tronic Imaging, 7(3):486 – 494, 1998. 1, 2 tocd pcd0992), 1993. http://r0k.us/graphics/ [3] Johannes Balle,´ Valero Laparra, and Eero P Simoncelli. End- kodak/. 6 to-end optimized image compression. In Int. Conf. Learn. [21] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Represent., 2017. 1, 2 Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Ui- [4] Johannes Balle,´ David Minnen, Saurabh Singh, Sung Jin jlings, Stefan Popov, Andreas Veit, et al. Openimages: Hwang, and Nick Johnston. Variational image compression A public dataset for large-scale multi-label and multi-class with a scale hyperprior. In Int. Conf. Learn. Represent., 2018. image classification. Dataset available from https:// 1, 2, 3, 5, 7 github.com/openimages, 2(3):2–3, 2017. 6 [5] Fabrice Bellard. BPG image format. https://bellard. [22] Jooyoung Lee, Seunghyun Cho, and Seung-Kwon Beack. org/bpg/. 1, 2, 3, 6, 7 Context-adaptive entropy model for end-to-end optimized [6] Keshi Chen and Tenkasi V Ramabadran. Near-lossless com- image compression. In Int. Conf. Learn. Represent., May pression of medical images through entropy-coded dpcm. 2019. 1, 2 IEEE Transactions on Medical Imaging, 13(3):538–548, 1994. 1, 2 [23] Mu Li, Wangmeng Zuo, Shuhang Gu, Jane You, and David Zhang. Learning content-weighted deep image compression. [7] Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro IEEE Trans. Pattern Anal. Mach. Intell., 2020. 1, 2 Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In IEEE Conf. [24] M. Li, W. Zuo, S. Gu, D. Zhao, and D. Zhang. Learn- Comput. Vis. Pattern Recog., pages 7939–7948, 2020. 1, 2, ing convolutional networks for content-weighted image com- 3 pression. In IEEE Conf. Comput. Vis. Pattern Recog., pages [8] Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Variable 3214–3223, 2018. 1, 2 rate deep image compression with a conditional autoencoder. [25] H. Ma, D. Liu, N. Yan, H. Li, and F. Wu. End-to-end opti- In Int. Conf. Comput. Vis., pages 3146–3154, 2019. 1, 2, 3, mized versatile image compression with wavelet-like trans- 5 form. IEEE Trans. Pattern Anal. Mach. Intell., pages 1–1, [9] Thomas M. Cover and Joy A. Thomas. Elements of Informa- 2020. 1, 2 tion Theory (Wiley Series in Telecommunications and Signal [26] Paul W Melnychuck and Majid Rabbani. Survey of loss- Processing). Wiley-Interscience, USA, 2006. 6 less image coding techniques. In Digital Image Processing [10] Sharaf E Elnahas. Data compression with applications to Applications, volume 1075, pages 92–100. International So- digital radiology. Washington Univ., St. Louis, MO (USA), ciety for Optics and Photonics, 1989. 1 1986. 1 [27] F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and [11] Vivek K Goyal. Theoretical foundations of transform coding. L. V. Gool. Conditional probability models for deep image IEEE Sign. Process. Magazine, 18(5):9–21, 2001. 1, 2, 3 compression. In IEEE Conf. Comput. Vis. Pattern Recog., [12] Zongyu Guo, Yaojun Wu, Runsen Feng, Zhizheng Zhang, pages 4394–4402, 2018. 1, 2 and Zhibo Chen. 3-d context entropy model for improved [28] F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and practical image compression. In IEEE Conf. Comput. Vis. L. Van Gool. Practical full resolution learned lossless image Pattern Recog. Worksh., pages 116–117, 2020. 3 compression. In IEEE Conf. Comput. Vis. Pattern Recog., [13] Emiel Hoogeboom, Jorn Peters, Rianne van den Berg, and pages 10621–10630, 2019. 2, 4, 6 Max Welling. Integer discrete flows and lossless compres- [29] Fabian Mentzer, George Toderici, Michael Tschannen, and sion. In Adv. Neural Inform. Process. Syst., pages 12134– Eirikur Agustsson. High-fidelity generative image compres- 12144, 2019. 1, 2 sion. In Adv. Neural Inform. Process. Syst., 2020. 1, 2 [14] WebP image format. https://developers.google. [30] Fabian Mentzer, Luc Van Gool, and Michael Tschannen. com/speed/webp/. 1, 2, 6, 7 Learning better lossless compression using lossy compres- [15] Ligang Ke and Michael W Marcellin. Near-lossless image sion. In IEEE Conf. Comput. Vis. Pattern Recog., 2020. 2, 3, compression: minimum-entropy, constrained-error dpcm. 4, 6 IEEE Trans. Image Process., 7(2):225–228, 1998. 1, 2 [31] David Minnen, Johannes Balle,´ and George D Toderici. [16] Diederik P Kingma and Jimmy Ba. Adam: A method for Joint autoregressive and hierarchical priors for learned image stochastic optimization. In Int. Conf. Learn. Represent., compression. In Adv. Neural Inform. Process. Syst., pages 2015. 6 10771–10780, 2018. 1, 2, 3, 4, 7 [17] Diederik P Kingma and Max Welling. Auto-encoding vari- [32] Jens-Rainer Ohm and Gary J Sullivan. Versatile video ational bayes. In Int. Conf. Learn. Represent., 2014. 1, 2, coding–towards the next generation of video compression. 3 In Picture Coding Symposium, 2018. 1, 7 11954 [33] Aaron¨ van den Oord, Nal Kalchbrenner, and Koray [49] Workshop and Challenge on Learned Image Compression. Kavukcuoglu. Pixel recurrent neural networks. In Int. Conf. https://www.compression.cc/challenge/. 6 Mach. Learn., pages 1747–1756. JMLR.org, 2016. 2 [50] X. Wu and N. Memon. Context-based, adaptive, lossless im- [34] Oren Rippel and Lubomir Bourdev. Real-time adaptive im- age coding. IEEE Trans. Communications, 45(4):437–444, age compression. In Int. Conf. Mach. Learn., volume 70, 1997. 1, 6 pages 2922–2930, 2017. 1, 2 [51] Wu Xiaolin and P. Bao. l∞ constrained high-fidelity image [35] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P compression via adaptive context modeling. IEEE Trans. Im- Kingma. Pixelcnn++: Improving the pixelcnn with dis- age Process., 9(4):536–542, 2000. 1, 2, 3, 7 cretized logistic mixture likelihood and other modifications. [52] X. Zhang and X. Wu. Near-lossless ℓ∞-constrained image In Int. Conf. Learn. Represent., 2017. 2, 4 decompression via deep neural network. In Data Compres- [36] Claude E Shannon. A mathematical theory of communi- sion Conference, pages 33–42, 2019. 2 cation. The Bell system technical journal, 27(3):379–423, [53] X. Zhang and X. Wu. Ultra high fidelity deep image decom- 1948. 1 pression with ℓ∞-constrained compression. IEEE Trans. Im- [37] Athanassios Skodras, Charilaos Christopoulos, and Touradj age Process., 30:963–975, 2021. 2 Ebrahimi. The jpeg 2000 still image compression standard. IEEE Sign. Process. Magazine, 18(5):36–58, 2001. 1, 2, 6, 7 [38] Jon Sneyers and Pieter Wuille. Flif: Free lossless image for- mat based on maniac compression. In IEEE Int. Conf. Image Process., pages 66–70. IEEE, 2016. 1, 6, 7 [39] Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszar.´ Lossy image compression with compressive autoen- coders. In Int. Conf. Learn. Represent., 2017. 1, 2 [40] George Toderici, Sean M. O’Malley, Sung Jin Hwang, Damien Vincent, David Minnen, Shumeet Baluja, Michele Covell, and Rahul Sukthankar. Variable rate image com- pression with recurrent neural networks. In Int. Conf. Learn. Represent., 2016. 1, 2 [41] George Toderici, Damien Vincent, Nick Johnston, Sung Jin Hwang, David Minnen, Joel Shor, and Michele Covell. Full resolution image compression with recurrent neural net- works. In IEEE Conf. Comput. Vis. Pattern Recog., pages 5306–5314, 2017. 1, 2 [42] James Townsend, Tom Bird, and David Barber. Practical lossless compression with latent variables using bits back coding. In Int. Conf. Learn. Represent., 2019. 2 [43] Aaron¨ van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, and Alex Graves. Conditional image genera- tion with pixelcnn decoders. In Adv. Neural Inform. Process. Syst., pages 4790–4798, 2016. 2, 5 [44] Gregory K Wallace. The jpeg still picture compression stan- dard. IEEE Trans. Consumer Electronics, 38(1):xviii–xxxiv, 1992. 2, 7 [45] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibil- ity to structural similarity. IEEE Trans. Image Process., 13(4):600–612, 2004. 1 [46] Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multi- scale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Sys- tems & Computers, 2003, volume 2, pages 1398–1402. Ieee, 2003. 1 [47] Marcelo J Weinberger, Gadiel Seroussi, and Guillermo Sapiro. The loco-i lossless image compression algorithm: Principles and standardization into jpeg-ls. IEEE Trans. Im- age Process., 9(8):1309–1324, 2000. 1, 2, 3, 6, 7 [48] Ian H Witten, Radford M Neal, and John G Cleary. Arith- metic coding for data compression. Communications of the ACM, 30(6):520–540, 1987. 2 11955