Proc. 26th IEEE International Conference on Image Processing (ICIP 2019, Taipei, Taiwan, September 2019), 3557-3561, 2019. The final publication is available on IEEE Xplore. FAST INPAINTING-BASED COMPRESSION: COMBINING SHEPARD INTERPOLATION WITH JOINT INPAINTING AND PREDICTION Pascal Peter Mathematical Image Analysis Group, Faculty of Mathematics and Computer Science, Campus E1.7, Saarland University, 66041 Saarbrucken,¨ Germany. [email protected] ABSTRACT research goal. Even real-time decoding already constitutes a significant challenge since current inpainting-based codecs Inpainting-based compression has been suggested as a qua- consist of sophisticated and time-intensive building blocks. litative competitor to the JPEG family of transform-based They supplement advanced inpainting methods [2] with cost- codecs, specifically for high compression ratios. However, intensive strategies for the optimisation of known data [2, 8]. it also requires sophisticated interpolation, data optimisation Therefore, fast decoding without loss of quality has been only and encoding tasks that are both slow and hard to implement. achieved with state-of-the-art numerics, as well as GPU and We propose a fast and simple alternative that combines Shep- CPU parallelisation. Additionally, many inpainting-based ard interpolation with a novel joint inpainting and prediction codecs use PAQ [10] for encoding, an efficient but slow approach. It represents the image by a fraction of its pixel context-mixing approach that involves predictions by a large values on a sparse regular subgrid that are selected by an number of neural networks. Overall, current codecs suffer efficient optimisation strategy. Experiments show that our from two drawbacks of inherent complexity: Slow encoding codec is up to five orders of magnitude faster than traditional speed and non-trivial implementation. inpainting-based approaches. Qualitatively, it can surpass transform-based codecs, in particular for high compression Our contribution. We propose a proof-of-concept codec ratios and cartoon-like images. for fast inpainting-based image compression with simple in- gredients. Our framework aims at reducing complexity while Index Terms— Image compression, inpainting, predic- preserving the strengths of inpainting-based compression: tion Good quality for high compression ratios and cartoon-like images. To this end, we replace advanced inpainting and opti- 1. INTRODUCTION misation techniques by simple Shepard interpolation [11, 12] on a regular grid of known pixels. This removes storage costs Lossy inpainting-based image compression [1, 2] relies on for positional data and can be combined with a novel fast the principle of spatial sparseness: Codecs select and store a optimisation strategy for the corresponding grey values. For small fraction of image points. During decompression, a suit- encoding, we combine finite state entropy (FSE) [13, 14], able interpolation operator restores the missing data based on a fast alternative to arithmetic coding, with a new concept: this known pixel mask, thus performing inpainting. Quali- joint inpainting and prediction (JIP) estimates values from tatively, these codecs can compete with JPEG [3] and JPEG partial pixel masks during inpainting at negligible additional 2000 [4], in particular for piecewise smooth images [5, 6] and cost. We compare our new strategy to compression with ho- high compression ratios on natural images [2]. mogeneous diffusion inpainting and to the transform-based So far, quality was the focus of research on inpainting- codecs JPEG and JPEG 2000. based compression, while runtime was mostly neglected. Related work. For image reconstruction, we rely on Real-time decompression has been considered for video de- Shepard interpolation [11] that restores missing pixels as a coding [7, 8, 9], but fast encoding has not been a dedicated normalised average of known data weighted by an inverse Part of our research has been funded by the ERC Advanced Grant distance function. More recent variants of this method in- INCOVID. This is gratefully acknowledged. clude normalised convolution [15] of Knutsson and Westin, as well as the efficient filtering by adaptive normalisation 0 otherwise. Then a diffusion-based inpainting result is the (EFAN) of Achanta et al. [12]. solution u of the inpainting equation Codecs with homogeneous diffusion [5, 6, 16, 17, 18, 19] resemble our approach since they use a simple inpainting op- C(u − f) − (I − C)Lu = 0; (2) erator. However, they rely either on more sophisticated data mn×mn selection or encoding strategies. Real-time performance has where I 2 R is the unit matrix, and Lu is a discretisa- been only considered for video coding [7, 8, 9] and relies on tion of the spatial Laplacian ∆u = @xxu + @yyu with reflect- complex numerics, not on simplicity. The methods of Galic´ et ing boundary conditions. The first term of Eq. (2) ensures that al. [1] and Schmaltz et al. [2] use more advanced inpainting, the known data is preserved at locations in K, while the sec- but also inspire our new strategy for grey value optimisation. ond term imposes a smoothness constraint on the missing pix- Organisation of the paper. Section 2 describes relevant els. For our experiments, we use a conjugate gradient scheme inpainting techniques. We introduce our new codec in Section to determine the solution u of the linear system Eq. (2). 3 and evaluate it in Section 4. The paper concludes with a summary and discussion of future work in Section 5. 3. CODING IMAGES WITH SIMPLE INGREDIENTS In the following, we first describe two of our core novelties, 2. FAST AND SIMPLE INPAINTING the joint inpainting and prediction (JIP) and a fast tonal opti- misation with EFAN. In Section 3.3 we combine those com- Our codec relies on Shepard interpolation [11, 12, 15]. It is ponents to our novel codec. not only fast, but also allows us to design novel prediction and optimisation components for our compression pipeline in Section 3. For the sake of comparison we also consider homo- 3.1. Joint inpainting and prediction geneous diffusion inpainting as the most widely-used operator So far, inpainting-based codecs separate image reconstruction in inpainting-based compression [5, 6, 16, 17, 18, 19]. and efficient storage by prediction and/or entropy coding (e.g. PAQ [10]) completely. Instead, we propose to use EFAN from 2.1. Shepard interpolation Section 2.1 for prediction and image reconstruction. We as- sume that the pixel values of f have been uniformly quantised m×n Consider a discrete image f 2 R which is only known to the grey level range f0; :::; q − 1g. on the inpainting mask, a subset K of the image domain Ω = EFAN is implemented by visiting each known point xj 2 f1; :::; mg × f1; :::; ng. For xi 2 Ω, we can compute a recon- K sequentially and adding its contribution to the numerator of struction ui := u(xi) by averaging known data weighted by Eq. (1) to the value accumulation map v and its contribution a function w: to the denominator to the weight accumulation map w. For P wi := w(xi), this corresponds to the updates wi wi + w(xj − xi)fj xj 2K G(x − x ) v v + G(x − x ) ui = P : (1) i j and i i i j for all locations w(xj − xi) xj 2K xi in the truncated Gaussian neighbourhood Nj of xj 2 K. Finally, the new image is given by ui = vi=wi. The EFAN variant [12] of this general Shepard interpola- Standard EFAN traverses all points of K before comput- tion approach defines w by a truncated Gaussian G(x) := ing a reconstruction. However, as soon as the weight accu- exp((−x2 − x2)=(2σ2)) with standard deviation σ and size 1 2 mulation map wi is non-zero at a location xi, a preliminary (d4σe+1)×(d4σe+1). In Eq. (1), the Gaussian is centred at reconstruction pi = vi=wi can be defined. Even though this the point xi. The computation of a full image reconstruction is only based on adjacent, already visited known data, it can operations and is linear in the number of pixels m · n. act as a surprisingly accurate prediction. The only parameter of the algorithm, the standard devia- Therefore, our joint inpainting and prediction (JIP) ex- tion σ of the Gaussian, can be automatically adapted to the tends EFAN by a simple prediction step: We visit all points mask density by σ = p(mn)=(πjKj). We have experimen- xi 2 K in a fixed order (left to right, top to bottom) and es- tally confirmed this as a good choice for our purposes. timate them by pi if wi 6= 0, and by 0 otherwise. By storing the prediction error ei = fi − pi mod q instead of the original 2.2. Inpainting with homogeneous diffusion pixel value fi we can reduce the entropy of mask values. JIP requires only a single additional division for each prediction In our experiments, we also consider inpainting based on ho- compared to standard EFAN. The adaptation of the truncated mogeneous diffusion [20]. We write the original image as a Gaussian’s size to the mask density (see Section 2.1) ensures vector f 2 Rmn by traversing it row by row. Furthermore, an overlap with at least one adjacent known data point that has we express the set of known data K by a diagonal matrix not been visited so far. Thereby, starting with a single known mn×mn C 2 R : Its entries ci;i are 1 for known pixels fi and data point, we can predict all remaining mask points. RJIP RJIP 105 HOM RJIP (LPAQ) QAT 350 HOM JPEG 103 300 JPEG2000 250 101 MSE 200 log runtime (s) 150 −1 (a) original image trui 10 100 50 10−3 104 105 106 107 20 40 60 80 100 120 140 log number of pixels compression ratio (X:1) (b) original image sintel (c) runtime on sintel (d) MSE on trui Fig.