Inpainting-Based Compression of Visual Data

2018 Data Compression Conference M I Snowbird, UT, USA, March 27–30, 2018 A 1 2 3 4 5 6 7 8 Inpainting-Based Compression of Visual Data 9 10 11 12 13 14 15 16 Joachim Weickert 17 18 Faculty of Mathematics and Computer Science 19 20 21 22 Saarland University 23 24 Saarbr¨ucken, Germany 25 26 27 28 29 30 31 32 long term research funded by 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 Gottfried Wilhelm Leibniz Prize ERC Advanced Grant INCOVID 57 58 59 60 61 62 c 2018 Joachim Weickert 63 64 65

Collaborators M I A 1 2 3 4 joint work with 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Naoufal Amrani Sarah Andris Egil Bae Zakaria Belhachmi Matthias Berg Pinak Bheed 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 AndrésBruhn Dorin Bucur Irena Galić Sebastian Hoffmann Lena Karos Markus Mainberger 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Pascal Peter Gerlind Plonka Christian Schmaltz Joan Serra-Sagrista Ching Hoo Tang Martin Welk 61 62 63 64 65 Introduction (1) M I A 1 2 3 4 Introduction 5 6 7 8 9 10 Transform-based Codecs for Lossy Image and Video Compression 11 12 13 14 15 16 Widely-used lossy codecs such as JPEG, JPEG2000, HEVC are transform-based. 17 18 19 20 rely on sparse representation in transform domain (DCT, wavelets) 21 22 23 24 25 26 sparse approximation in terms of an orthogonal base offers many benefits: 27 28 29 30 clean theory 31 32 • 33 34 fast algorithms 35 36 • 37 38 39 40 quality deteriorates with sparsity 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Introduction (2) M I A 1 2 Quality Deterioration under JPEG Compression 3 4 5 6 7 8 original compression rate 10 : 1 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 compression rate 40 : 1 compression rate 80 : 1 61 62 63 64 65 Introduction (3) M I A 1 2 Further Improvements through Transform-Based Codecs 3 4 5 6 7 8 carefully engineered by many experts over many years 9 10 11 12 13 14 approaches have become highly sophisticated 15 16 17 18 19 20 increasingly difficult to achieve further fundamental improvements 21 22 by staying within the universe of transform-based ideas 23 24 25 26 27 28 29 30 Things that Transform-Based Codecs Ignore 31 32 33 34 35 36 more semantical, higher level representation of image structures: 37 38 39 40 – need for features (e.g. edges) rather than pixels 41 42 – filling-in effect of our visual system (Werner 1935) 43 44 45 46 47 48 approximation qualities beyond pixelwise quality measures (MSE, PSNR): 49 50 51 52 respecting e.g. perceptual similarities of stochastic textures 53 54 55 56 57 58 59 60 61 62 63 64 65

Introduction (4) M I A 1 2 An Emerging Alternative: Inpainting-Based Compression 3 4 5 6 7 8 store only a very small, carefully selected subset of the data 9 10 11 12 13 14 reconstruct missing data with a suitable inpainting process 15 16 17 18 19 20 21 22 23 24 What is Inpainting ? 25 26 27 28 29 30 interpolation strategy for missing / degraded data 31 32 33 34 so far mainly used for repairing smaller parts of an image 35 36 37 38 39 40 two classes relevant for us: 41 42 43 44 PDE-based inpainting (Masnou/Morel 1998, Bertalmio et al. 2000) 45 46 • 47 48 exemplar-based inpainting (Efros/Leung 1999, Criminisi et al. 2004) 49 50 • 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 Introduction (5) M I A 1 2 PDE-Based Inpainting 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 original inpainted 37 38 39 40 Left: Original image with severe degradations by a text. From Bertalmio et al. (2000). Right: Result of 41 42 inpainting with an anisotropic diffusion process. 43 44 45 46 47 48 49 50 fills in with a partial differential equation (PDE), e.g. a diffusion process 51 52 53 54 uncorrupted data serve as fixed boundary conditions 55 56 57 58 PDE propagates this information to inpainting domain 59 60 61 62 local approach that can be justified from the statistics of natural images 63 64 65

Introduction (6) M I A 1 2 Exemplar-Based Inpainting 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 original inpainted 37 38 39 40 Left: Original image. From Criminisi et al. (2004). Right: Inpainting result with Criminisi’s exemplar- 41 42 based approach. 43 44 45 46 47 48 49 50 clever copy-and-paste from similar patches within the image 51 52 53 54 nonlocal approach that exploits the intrinsic statistical properties of the image 55 56 57 58 good for texture-dominated images 59 60 61 62 computationally more expensive than PDE-based inpainting 63 64 65 Introduction (7) M I A 1 2 How Sparse can the Data be for PDE-based Inpainting ? 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 Image where only 2 percent of all pixels are known. Can you recognise what is depicted ? 57 58 59 60 61 62 63 64 65

Introduction (8) M I A 1 2 Is Exemplar-based Inpainting also Useful for Sparse Representations ? 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 Texture synthesis with the exemplar-based approach of Criminisi et al. (2004), using our fast algorithm 57 58 59 60 with space-ﬁlling curves (Dahmen et al. 2017). 61 62 63 64 65 Introduction (9) M I A 1 2 What Happens if We Select Semantically More Important Data ? 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 original data near edges kept inpainted 45 46 47 48 49 50 Image reconstruction from edges. Left: Original image, 237 316 pixels. Middle: Edge set. Only × 51 52 the left and right neighbours of each Canny edge are stored. Right: Inpainting with the homogeneous 53 54 55 56 diﬀusion equation. 57 58 59 60 61 62 63 64 65

Introduction (10) M I A 1 2 Chances for Inpainting-Based Compression 3 4 5 6 7 8 spatial representation more intuitive than transform-based coding 9 10 11 12 respects more properties of human visual system: 13 14 15 16 can use semantically important features such as edges 17 18 • 19 20 PDE-based inpainting resembles biological filling-in mechanisms 21 22 • 23 24 exemplar-based inpainting captures statistics of textures, 25 26 • 27 28 while renouncing pixelwise accuracy 29 30 31 32 33 34 may go beyond limits of sparse approximations in orthogonal bases 35 36 37 38 39 40 generic framework for various data types: 41 42 1D, 2D, 3D signals 43 44 • 45 46 still images and videos 47 48 • 49 50 scalar-, vector-, matrix-valued 51 52 • 53 54 surface data, graphs 55 56 • 57 58 dedicated codecs for specific applications 59 60 • 61 62 63 64 65 Introduction (11) M I A 1 2 Key Challenges for Inpaiting-Based Compression 3 4 5 6 7 8 1. Which data should be kept? 9 10 11 12 13 14 2. What are the most suitable inpainting processes? 15 16 17 18 19 20 3. How can the selected pixels be encoded in an efficient way? 21 22 23 24 4. Can one design fast algorithms for encoding and decoding? 25 26 27 28 29 30 31 32 33 34 These Problems are Highly Interrelated: 35 36 37 38 39 40 Optimality of data depends on inpainting process. 41 42 43 44 45 46 Suboptimal pixels can pay off if they are cheaper to encode. 47 48 49 50 There is a natural tradeoff between high efficiency and high quality. 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Introduction (12) M I A 1 2 How Does this Differ from Related Approaches ? 3 4 5 6 7 8 Classical Inpainting Methods: 9 10 11 12 PDE-based inpainting: 13 14 • Masnou/Morel 1998, Bertalmio et al. 2000, Chan/Shen 2000, W./Welk 2006, 15 16 17 18 Bornemann/März2007, Facciolo et al. 2009, Schönlieb2015 19 20 21 22 exemplar-based inpainting: 23 24 • Efros/Leung 1999, Wei/Leroy 2000, Criminisi et al. 2004, Levina/Bickel 2006, 25 26 27 28 Aujol et al. 2010, Arias et al. 2011 29 30 31 32 rarely used for very sparse data; do not optimise inpainting data 33 34 35 36 37 38 Scattered Data Interpolation Methods, Radial Basis Functions: 39 40 e.g. Shepard 1968, Duchon 1976, DiBlasi et al. 2010, Achanta et al. 2017 41 42 43 44 no data optimisation, usually not applied to compression 45 46 47 48 49 50 Image Compression with Subdivision: 51 52 53 54 Dyn et al. 1990, Sullivan et al. 1994, Distasi et al. 1997, Demaret et al. 2006, 55 56 57 58 Kohout 2007, Peyréet al. 2009, Cohen et al. 2012 59 60 61 62 usually not inpainting-based, uses e.g. linear splines; more localised 63 64 65 Introduction (13) M I A 1 2 Inpainting-Based Reconstruction from Image Features: 3 4 5 6 toppoints in scale-space: 7 8 • 9 10 Johansen et al. 1986, Kanters et al. 2005 11 12 edges: 13 14 • 15 16 Carlsson 1988, Hummel/Moniot 1989, Desai et al. 1996, Elder 1999, 17 18 Mainberger et al. 2011, Gautier et al. 2012 19 20 21 22 level lines: 23 24 • 25 26 Soléet al. 2004, Facciolo et al. 2006, Xie et al. 2007 27 28 29 30 derivatives: 31 32 • Lillholm et al. 2003, Brinkmann et al. 2015, Peter et al. 2016 33 34 35 36 SIFT points or local binary descriptors: 37 38 • Weinzaepfel et al. 2011, DAngelo et al. 2014 39 40 41 42 43 44 often not for compression purposes; 45 46 suboptimal for general imagery; 47 48 49 50 can be useful for specific image classes, e.g. digital elevation maps, depth maps 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Introduction (14) M I A 1 2 PDEs and Inpainting Ideas within Classical Codecs 3 4 5 6 many papers on PDE-based pre- or postprocessing 7 8 • 9 10 PDE-improved wavelet thresholding: 11 12 • Chan/Zhou 2000 13 14 15 16 inpainting within JPEG, H.264/AVC etc.: 17 18 • 19 20 Liu et al. 2007, Xiong et al. 2007, Ball´eet al. 2011, Ndjiki-Nya et al. 2012, 21 22 Racape et al. 2013 23 24 25 26 27 28 Does this exploit the full potential of inpainting? 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 Introduction (15) M I A 1 2 Goals of This Talk 3 4 5 6 7 8 survey achievements and challenges in inpainting-based compression 9 10 11 12 13 14 philosophy: 15 16 17 18 focus mainly on PDE-based impainting, 19 20 • use exemplar-based inpainting when necessary 21 22 23 24 aim at designing dedicated inpainting-based codecs from scratch, 25 26 • 27 28 rather than embedding inpainting into existing frameworks 29 30 31 32 show that they can reproduce important features of transform-based codecs 33 34 • 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Optimising the Data (1) M I A 1 2 3 4 Optimising the Data 5 6 7 8 Our Baseline Codec 9 10 11 12 13 14 15 16 Encoding: 17 18 19 20 store (greyscale) image f :Ω R only in some small subset K Ω. 21 22 • → ⊂ 23 24 25 26 Decoding: 27 28 29 30 In this subset K (inpainting mask), the reconstruction u is known: 31 32 • 33 34 u(x) = f(x). 35 36 37 38 39 40 In the inpainting domain Ω K where the data are unknown: 41 42 • \ 43 44 Compute the steady state (t ) of the homogeneous diﬀusion equation 45 46 →∞ 47 48 ∂tu = ∆u 49 50 51 52 with the known data as ﬁxed boundary conditions 53 54 55 56 57 58 In short: Inpaint with the Laplace equation ∆u = 0. 59 60 • 61 62 63 64 65 Optimising the Data (2) M I A 1 2 Continous Spatial Data Optimisation 3 4 5 6 7 8 Belhachmi et al. (2009): analytic theory based on shape optimisation 9 10 11 12 13 14 result: choose density of the data points as an increasing function of ∆f 15 16 | | 17 18 19 20 real-time encoding: just compute Laplacian and apply dithering 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 Felix Klein Laplacian magnitude 10% mask by dithering inpainting 55 56 57 58 59 60 61 62 63 64 65

Optimising the Data (3) M I A 1 2 Analytic Approach Clearly Outperforms Random Selection 3 4 5 6 7 8 randomly selected analytic approach 1 9 10 (5% pixels) (5% pixels) 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 mask 27 28

original 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 reconstruction 53 54 55 56 57 58 MSE: 189.90 MSE: 49.47 59 60 61 62 [1] with electrostatic dithering, Schmaltz et al. (2010) 63 64 65 Optimi1ing the Data (4) M I A 1 2 Discrete Spatial Data Optimisation 3 4 5 6 7 8 9 10 Why Discrete? 11 12 13 14 15 16 Althought the analytic approach is optimal in theory, 17 18 dithering may introduce substantial errors. 19 20 21 22 23 24 Does a genuinely discrete data optimisation perform better? 25 26 27 28 29 30 31 32 What Happens if we Discretise our Inpainting Problem ? 33 34 35 36 (Mainberger et al. 2011) 37 38 39 40 41 42 finite differences lead to sparse linear system of equations: 43 44 one unknown per unknown pixel 45 46 47 48 49 50 has a unique solution, if at least one pixel is specified 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Optimising the Data (5) M I A 1 2 Good and Bad News about Discrete Data Selection 3 4 5 6 Good News: Problem is finite, thus a global optimum exists. 7 8 9 10 Bad News: Selecting the best 5% pixels of a 256 256 image offers 11 12 × 13 14 65536 15 16 1.72 105648 possibilities. 17 18 3277 ≈ · 19 20 21 22 23 24 Well-Performing Minimisation Algorithm (Mainberger et al. 2012) 25 26 27 28 29 30 Probabilistic Sparsification: 31 32 start with full mask 33 34 • 35 36 gradually discard pixels that give only small errors when they are removed 37 38 • 39 40 41 42 Postprocessing by Nonlocal Pixel Exchange: 43 44 to avoid getting trapped in local minima 45 46 • 47 48 exchange randomly selected mask pixels with non-mask pixels with large errors, 49 50 • 51 52 if this improves the total error 53 54 55 56 Alternative Optimisation Strategies: 57 58 59 60 e.g. bilevel ideas (Hoeltgen et al. 2013, Ochs et al. 2014, Bonnetini et al. 2017) 61 62 • 63 64 65 Optimising the Data (6) M I A 1 2 Substantial Improvements over Analytic Approach: 3 4 5 6 7 8 analytic approach discrete approach 9 10 (5% pixels) (5% pixels) 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 mask 27 28

original 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 reconstruction 53 54 55 56 57 58 MSE: 49.47 MSE: 23.21 59 60 61 62 63 64 65

Optimising the Data (7) M I A 1 2 Discrete Tonal Data Optimisation 3 4 5 6 (Mainberger et al. 2012) 7 8 9 10 11 12 Motivation 13 14 15 16 17 18 So far we have optimised the location of the pixels: 19 20 21 22 Can we achieve further improvements by optimising their grey values ? 23 24 25 26 We call this tonal optimisation. 27 28 29 30 Such a mechanism would not help for transform-based codecs: 31 32 33 34 Orthogonal basis approximations cannot be improved by tuning the coefficients. 35 36 37 38 39 40 How Does This Work ? 41 42 43 44 Let some image f and some optimised fixed pixel mask be given. 45 46 47 48 Perturb data f within the pixel mask by some vector α. 49 50 51 52 Minimising the MSE w.r.t. α is a least squares problem with unique solution. 53 54 55 56 resulting linear system is symmetric and dense (as many unknowns as mask pixels) 57 58 59 60 61 62 can be solved with standard algorithms such as Cholesky, QR, or CG 63 64 65 Optimising the Data (8) M I A 1 2 Tonal Optimisation Leads to Significant Quality Gains: 3 4 5 6 7 8 randomly selected analytic approach discrete approach 9 10 11 12 (5 % pixels) (5 % pixels) (5 % pixels) 13 14 15 16 MSE: 189.90 MSE: 49.47 MSE: 23.21 17 18 19 20 21 22 23 24 25 26 27 28 29 30

before 31 32 original 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 after 49 50 51 52 53 54 55 56 57 58 MSE: 106.17 MSE: 31.62 MSE: 17.17 59 60 61 62 63 64 65

Outline M I A 1 2 3 4 Outline 5 6 7 8 9 10 Optimising the Data 11 12 13 14 Optimising the Inpainting Operator 15 16 17 18 19 20 Optimising the Encoding 21 22 23 24 25 26 Fast Algorithms and Real-Time Aspects 27 28 29 30 31 32 Connections, Extensions, and Applications 33 34 35 36 Summary and Outlook 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 Optimising the Inpainting Operator (1) M I A 1 2 3 4 Optimising the Inpainting Operator 5 6 7 8 so far: inpainting with the homogeneous diffusion operator ∆u; 9 10 11 12 requires many data to represent edges 13 14 15 16 17 18 Galićet al. 2005, Schmaltz et al. 2009, Chen et al. 2014: 19 20 experimental evaluations of various differential operators 21 22 • 23 24 motivated from spline theory or inpainting literature 25 26 • 27 28 29 30 31 32 interpolation strategy authors differential oper. max–min 33 34 35 36 homogeneous diffusion Iijima 1959 ∆u yes 37 38 39 40 biharmonic interpol. Duchon 1976 ∆2u no 41 42 − 3 43 44 triharmonic interpol. ∆ u no 45 46 47 48 AMLE Caselles et al. 1998 u (η u) yes ηη k ∇ 49 50 u 51 52 TV inpainting Rudin et al. 1992 div ∇u yes 53 54 |∇ | 55 56 anisotropic diff. (EED) W. 1996 div (D( uσ) u) yes 57 58 ∇ ∇ 59 60 61 62 63 64 65

Optimising the Inpainting Operator (2) M I A 1 2 Comparison of Different Inpainting Operators 3 4 5 6 7 8 input image inpainting mask homogeneous diff. biharmonic 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 triharmonic AMLE TV anisotr. diff. (EED) 53 54 55 56 57 58 TV performs always bad for sparse data (prefers small edge lengths). 59 60 61 62 Edge-enhancing anisotropic diffusion (EED) is consistently good. 63 64 65 Optimising the Inpainting Operator (3) M I A 1 2 Best performing inpainting operator in this and other experiments: 3 4 5 6 7 8 Edge-Enhancing Anisotropic Diffusion (EED) 9 10 11 12 13 14 uses nonlinear inpainting operator 15 16 17 18 19 20 div (D( u ) u) 21 22 ∇ σ ∇ 23 24 25 26 27 28 originally for denoising (W. 1996), later for inpainting (W./Welk 2006) 29 30 31 32 33 34 uσ is a Gaussian-smoothed version of u 35 36 37 38 39 40 diffusion tensor D( uσ): 41 42 ∇ 43 44 symmetric 2 2 matrix 45 46 • × 47 48 adapts itself to (semi-) local image structure 49 50 • 51 52 prefers inpainting along edges over inpainting across them 53 54 • 55 56 57 58 requires to solve sparse nonlinear systems of equations 59 60 61 62 63 64 65

Optimising the Inpainting Operator (4) M I A 1 2 Why Does Anisotropic Diffusion (EED) Work so Well ? 3 4 5 6 7 8 9 10 Two Simple but Instructive Experiments 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 dipole occluded disk 53 54 55 56 57 58 grey initialisation of inpainting domain 59 60 61 62 63 64 65 Optimising the Inpainting Operator (5) M I A 1 2 EED has Interesting Shape Reconstruction Qualities 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 Binary image, where data are specified within 24 small disks. The unknown pixels are initialised with 53 54 55 56 uniform noise. Reconstruction is performed with anisotropic diffusion (EED). Author: W. (2012). 57 58 59 60 61 62 63 64 65

Optimising the Inpainting Operator (6) M I A 1 2 What is so Special about EED ? 3 4 5 6 7 8 anisotropy due to the diffusion tensor: 9 10 11 12 – makes it well-suited for reconstructing edges 13 14 15 16 – no need for high mask pixel density at edges 17 18 19 20 21 22 nonlocality due to Gaussian convolution: 23 24 25 26 – allows to propagate edge structures 27 28 – creates a curvature-reducing effect (cf. Merriman et al. 1994) 29 30 31 32 33 34 very natural: 35 36 37 38 – EED-like operators reflect statistics of natural images 39 40 41 42 (Peter et al. 2015) 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 Outline M I A 1 2 3 4 Outline 5 6 7 8 9 10 Optimising the Data 11 12 13 14 Optimising the Inpainting Operator 15 16 17 18 19 20 Optimising the Encoding 21 22 23 24 25 26 Fast Algorithms and Real-Time Aspects 27 28 29 30 31 32 Connections, Extensions, and Applications 33 34 35 36 Summary and Outlook 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Optimising the Encoding (1) M I A 1 2 3 4 Optimising the Encoding 5 6 7 8 9 10 Lessons Learnt 11 12 13 14 15 16 Homogeneous diffusion requires careful data optimisation. 17 18 19 20 Anisotropic diffusion (EED) can be a better alternative: 21 22 23 24 It needs fewer data to reconstruct edges (and texture). 25 26 • 27 28 Thus, the data distribution can be more evenly. 29 30 • 31 32 This means that the data location is a bit less critical. 33 34 • 35 36 37 38 39 40 Key Aspect Ignored so Far 41 42 43 44 45 46 encoding costs: storing a freely optimised mask can be expensive 47 48 49 50 51 52 53 54 Balancing all Requirements 55 56 57 58 59 60 use EED and find a slightly suboptimal mask that is much cheaper to encode 61 62 63 64 65 Optimising the Encoding (2) M I A 1 2 Finding a Useful Mask that is Inexpensive to Encode 3 4 5 6 7 8 rectangular subdivision where approximation quality of EED is insufficient 9 10 11 12 (cf. Dyn et al. 1990, Sullivan et al. 1994, Distasi et al. 1997) 13 14 15 16 encoding in a binary tree costs about 1 bit per pixel location 17 18 19 20 21 22 23 24 Depth 0 25 26 27 28 29 30 31 32 33 34 Depth 1 35 36 37 38 39 40 41 42 43 44 45 46 Depth 2 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Rectangular subdivision. Each subdivision step splits the white area and inserts one point. 61 62 63 64 65

Optimising the Encoding (3) M I A 1 2 Designing a Full Codec 3 4 5 6 7 8 beneﬁt from established concepts on the encoding side: 9 10 – coarser quantisation of grey values 11 12 13 14 – remove redundancy from entire bitstring by entropy coding (e.g. PAQ) 15 16 17 18 19 20 decoding: inpaint with EED 21 22 23 24 25 26 27 28 Example at a Compression Rate of 57 : 1 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 original image data from subdivision EED-based decoding 59 60 61 62 63 64 65 Optimising the Encoding (4) M I A 1 2 Comparison to JPEG and JPEG 2000 at 57 : 1 3 4 5 6 7 8 original JPEG, MSE = 111.01 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 JPEG 2000, MSE = 70.79 EED, MSE = 42.77 61 62 63 64 65

Optimising the Encoding (5) M I A 1 2 What Happens for Extremely High Compression Rates ? 3 4 5 6 7 8 JPEG 2000, 162 : 1 JPEG 2000, 202 : 1 JPEG 2000, 261 : 1 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 EED, 163 : 1 EED, 211 : 1 EED, 264 : 1 61 62 63 64 65 Outline M I A 1 2 3 4 Outline 5 6 7 8 9 10 Optimising the Data 11 12 13 14 Optimising the Inpainting Operator 15 16 17 18 19 20 Optimising the Encoding 21 22 23 24 25 26 Fast Algorithms and Real-Time Aspects 27 28 29 30 31 32 Connections, Extensions, and Applications 33 34 35 36 Summary and Outlook 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Fast Algorithms and Real-Time Aspects (1) M I A 1 2 3 4 Fast Algorithms and Real-Time Aspects 5 6 7 8 9 10 distinguish between real-time encoding and real-time decoding: 11 12 13 14 e.g. video decoding must be real-time, video encoding not necessarily 15 16 • 17 18 19 20 encoding requires fast data optimisation 21 22 23 24 simple strategies real-time capable, e.g. analytic approach with dithering 25 26 • 27 28 29 30 decoding with homogeneneous diffusion: 31 32 requires to solve sparse linear systems of equations 33 34 • 35 36 adapted full multigrid schemes 37 38 • 39 40 real-time performance on CPUs (Mainberger et al. 2011) 41 42 • 43 44 45 46 decoding with edge-enhancing anisotropic diffusion (EED): 47 48 49 50 requires to solve sparse nonlinear systems of equations 51 52 • 53 54 developed novel, so-called fast explicit diffusion schemes (W. et al. 2016) 55 56 • 57 58 real-time performance on GPUs (Peter et al. 2015) 59 60 • 61 62 63 64 65 Fast Algorithms and Real-Time Aspects (2) M I A 1 2 Real-Time Demo: Analytic Approach with Web Cam (640 480 Pixels) 3 4 × 5 6 7 8 data selection: Floyd-Steinberg dithering of Laplacian magnitude 9 10 11 12 inpainting: homogeneous diffusion with full multigrid algorithm 13 14 15 16 computations done on the Laptop CPU (Intel i5–6300HQ, 2.30 GHz, 4 cores) 17 18 19 20 21 22 solves three systems of 307,200 linear equations with ca. 13 fps 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 Start (320 240) Start (640 480) 59 60 × × 61 62 63 64 65

Fast Algorithms and Real-Time Aspects (3) M I A 1 2 Real-Time Decoding with EED-Based Inpainting (Peter et al. 2015) 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 Nosferatu – Eine Symphonie des Grauens is a silent movie directed by Friedrich Wilhelm Murnau in 53 54 1922. We can play it in full length (94 minutes) with our EED-based decoder. Inpainting 25 frames per 55 56 57 58 second with 640 480 pixels on a PC requires highly optimised parallel algorithms for GPUs. We use × 59 60 our fast explicit diﬀusion schemes (W. et al. 2016). 61 62 63 64 65 Outline M I A 1 2 3 4 Outline 5 6 7 8 9 10 Optimising the Data 11 12 13 14 Optimising the Inpainting Operator 15 16 17 18 19 20 Optimising the Encoding 21 22 23 24 25 26 Fast Algorithms and Real-Time Aspects 27 28 29 30 31 32 Connections, Extensions, and Applications 33 34 35 36 Summary and Outlook 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Connections, Extensions, and Applications (1) M I A 1 2 3 4 Connections, Extensions, and Applications 5 6 7 8 9 10 Connections to Sparsity (Hoffmann et al. 2015) 11 12 13 14 15 16 consider Green’s functions of a discrete linear differential operator D 17 18 19 20 21 22 discrete Green’s function gk,` in a pixel (k, `) solves 23 24 25 26 27 28 (D gk,`)i,j = δ(k,`),(i,j) for all (i, j) 29 30 31 32 33 34 where δ is the Kronecker function. 35 36 (k,`),(i,j) 37 38 39 40 inpainting solution via superposition of Green’s functions in mask points 41 42 43 44 45 46 Green’s functions are atoms in a dictionary 47 48 49 50 51 52 allows also fast algorithms for sparse inpainting problems 53 54 55 56 57 58 59 60 61 62 63 64 65 Connections, Extensions, and Applications (2) M I A 1 2 Discrete Green’s Functions for Operators with Reflecting Boundary Conditions 3 4 5 6 7 8 Laplace Operator in (25, 30) Laplace Operator in (45, 10) 9 10 11 12 13 14 15 16 0.6 1 0.8 17 18 0.4 0.6 19 20 0.2 0.4 21 22 0.2 0 23 24 0

−0.2 −0.2 25 26 50 50 40 60 40 60 27 28 30 50 30 50 40 40 20 30 20 30 29 30 10 20 10 20 10 10 0 0 0 0 31 32 i j i j 33 34 35 36 37 38 39 40 150 10 41 42 100 5 43 44 50

0 45 46 0 47 48 −5 −50 49 50 50 50 51 52 40 60 40 60 30 50 30 50 40 40 53 54 20 30 20 30 10 20 10 20 10 10 55 56 0 0 0 0 i j i j 57 58 59 60 Biharmonic Operator in (25, 30) Biharmonic Operator in (45, 10) 61 62 63 64 65

Connections, Extensions, and Applications (3) M I A 1 2 Adaptation to Progressive Mode Coding (Peter et al. 2015) 3 4 5 6 7 8 JPEG 2000, 33 % JPEG 2000, 67 %, JPEG 2000, 100 % 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 EED, 33 % EED, 67 % EED, 100 % 55 56 57 58 Illustration of progressive mode capacities at a compression rate of 80 : 1. The percentage refers to 59 60 the size of the compressed image. 61 62 63 64 65 Connections, Extensions, and Applications (4) M I A 1 2 Edge-based Codecs for Piecewise Almost Constant Images 3 4 5 6 7 8 (Mainberger et al. 2011) 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 original data kept reconstruction 39 40 41 42 43 44 Left: Original image. Middle: Stored data with tonal optimisation. The data are subsampled along the 45 46 contour, requantised and entropy coded. Right: Reconstruction with homogeneous diﬀusion in each 47 48 colour channel. Compression rate: 200:1. 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Connections, Extensions, and Applications (5) M I A 1 2 Comparison to JPEG and JPEG 2000 at 200 : 1 3 4 5 6 7 8 original image JPEG, MSE=441.20 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 JPEG 2000, MSE=164.53 diffusion, MSE=25.97 61 62 63 64 65 Connections, Extensions, and Applications (6) M I A 1 2 Segmentation-based Codecs for Piecewise Smooth Images 3 4 5 6 7 8 (Hoffmann et al. 2013) 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 depth map stored data 43 44 45 46 47 48 Left: Zoom into a depth map image. It is a good example of a piecewise smooth image. Right: 49 50 Segment boundaries and data used for homogeneous diffusion inpainting. We store the segmentation 51 52 53 54 boundaries, the grey values on a hexagonal grid, and the locations and grey values of a few extra pixels. 55 56 57 58 59 60 61 62 63 64 65

Connections, Extensions, and Applications (7) M I A 1 2 Comparison at 177 : 1 (0.045 bpp) 3 4 5 6 7 8 original depth map segmentation-based hom. diﬀ. HEVC 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 JPEG 2000 JPEG 55 56 57 58 59 60 61 62 63 64 65 Connections, Extensions, and Applications (8) M I A 1 2 Exemplar-based Inpainting for Highly Textured Images 3 4 5 6 7 8 Hybrid Codec with EED- and Exemplar-Based Inpainting (Peter/W. 2015) 9 10 11 12 13 14 original EED hybrid 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 MSE=118.70 MSE=77.35 53 54 55 56 57 58 We use EED-optimised data and decide locally if EED-based inpainting or Facciolo’s exemplar-based 59 60 inpainting for sparse data is better. Top: Images in full size. Bottom: Zoom. Compression rate: 18:1. 61 62 63 64 65

Connections, Extensions, and Applications (9) M I A 1 2 Data Optimisation for Exemplar-Based Inpainting 3 4 5 6 (Karos et al. 2018) 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 original image optimised data, 5 % inpainted 41 42 43 44 For highly textured images, it can be beneﬁcial to use nonlocal, patch-based inpainting methods for 45 46 sparse data (Facciolo et al. 2009). Currently we are developing novel data optimisation strategies that 47 48 49 50 improve the quality substantially. 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 Connections, Extensions, and Applications (10) M I A 1 2 Extension to Colour Images (Peter et al. 2014) 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 original JPEG 2000, MSE=151.31 EED, MSE=77.53 35 36 37 38 Compression of a 256 256 subimage of peppers with compression rate 110:1. Left: Original image. 39 40 × 41 42 Middle: JPEG 2000. Right: EED in Luma Preference Mode. This mode works in the YCbCr space. It 43 44 45 46 compresses the luma channel Y with higher quality. The resulting diﬀusion tensor steers the inpainting 47 48 in the chroma channels Cb and Cr. 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Connections, Extensions, and Applications (11) M I A 1 2 Extension to Hyperspectral Data (Amrani et al. 2017) 3 4 5 6 7 8 homogeneous diﬀusion inpainting in space 9 10 11 12 13 14 biharmonic inpainting in spectral dimension 15 16 17 18 additional linear prediction model 19 20 21 22 qualitatively between JPEG 2000 Part II with DWT 9/7 and JPEG 2000 with PCA 23 24 25 26 27 28 29 30 55 31 32 33 34 35 36 50 37 38 39 40 45 41 42 43 44 45 46

SNR (dB) 40 47 48 49 50 Inpainting 2D+1D+h( ) 51 52 35 · JPEG2000 part II (DWT 97) 53 54 JPEG2000 part II (PCA) 55 56 30 57 58 0.0 0.5 1.0 1.5 2.0 2.5 59 60 Bitrate (bpp) 61 62 63 64 65 Connections, Extensions, and Applications (12) M I A 1 2 Extension to Three-Dimensional Data Sets 3 4 5 6 (Peter 2013) 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 original DICOM, MSE=65.02 EED, MSE=38.66 41 42 43 44 Compression of a 3-D data set with a compression rate of 207:1. Left: Original slice from a 3-D CT 45 46 47 48 data set of a trabecular bone. Middle: Result after compression in the DICOM standard, based on 49 50 JPEG 2000. Right: Compression with 3-D EED with cuboidal subdivision. 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Connections, Extensions, and Applications (13) M I A 1 2 Extension to Surface Data (Bae/W. 2010) 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 Left: Original data set. Right: Reconstruction with optimised 10 % of the points and geometric linear 55 56 57 58 diffusion. 59 60 61 62 63 64 65 Connections, Extensions, and Applications (14) M I A 1 2 Adaptation to Audio Compression 3 4 5 6 (Peter et al. 2018) 7 8 9 10 11 12 sparse representation of audio signal in sample domain 13 14 15 16 17 18 homogeneous diffusion inpainting becomes linear interpolation 19 20 21 22 data sparsification approach, coding with LPAQ 23 24 25 26 competitive to mp3, AAC, and Vorbis in terms of SNR, 27 28 29 30 inferior w.r.t. perceptual quality measures such as PEAQ 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Connections, Extensions, and Applications (15) M I A 1 2 Extension to Video Compression 3 4 5 6 (Andris et al., PCS 2016 Best Poster Award) 7 8 9 10 11 12 inpainting-based coding of selected images (intraframes) 13 14 15 16 prediction of images inbetween (interframes) with variational optic flow 17 18 19 20 additional coding of the residual errors 21 22 23 24 25 26 original frame inpainting mask (intraframe) 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 subsampled motion field (interframe) interframe residual 63 64 65 Connections, Extensions, and Applications (16) M I A 1 2 Example: MPI Sintel Data Set 3 4 5 6 7 8 9 10 stored data prediction 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 residue final reconstruction 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Connections, Extensions, and Applications (17) M I A 1 2 Steganographic Application: Censoring 3 4 5 6 7 8 (Mainberger et al. 2012) 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 original censored restored 43 44 45 46 47 48 Left: Original image. Middle: Censored version. The censored part is EED-encoded and hidden in the 49 50 least signiﬁcant bits of the non-censored part. Right: Reconstruction. 51 52 53 54 55 56 57 58 Try it with your own image: stego.mia.uni-saarland.de 59 60 61 62 63 64 65 Outline M I A 1 2 3 4 Outline 5 6 7 8 9 10 Continuous Spatial Data Optimisation 11 12 13 14 Discrete Spatial Data Optimisation 15 16 17 18 19 20 Discrete Tonal Data Optimisation 21 22 23 24 25 26 Better PDEs 27 28 29 30 31 32 Optimising the Encoding 33 34 35 36 Connections, Extensions, and Applications 37 38 39 40 41 42 Summary and Outlook 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Summary and Outlook M I A 1 2 3 4 Summary 5 6 7 8 inpainting offers novel and promising ways for compressing visual data 9 10 11 12 store subset of pixels and inpaint inbetween 13 14 15 16 17 18 can achieve results of competitive quality, 19 20 if data, inpainting process, and coding are optimised 21 22 23 24 25 26 flexible and widely applicable 27 28 29 30 31 32 33 34 Outlook 35 36 37 38 39 40 theoretical underpinning, quality guarantees 41 42 43 44 learning the optimal data and inpainting operators 45 46 47 48 49 50 treating all optimisation problems simultaneously 51 52 53 54 inpainting-based codecs for highly textured images 55 56 57 58 highly efficient numerical algorithms for parallel architectures 59 60 61 62 63 64 65 Thanks M I A 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Thank you! 15 16 17 18 19 20 21 22 23 24 25 26 www.mia.uni-saarland.de/Research/IP Compress.shtml 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65