2018 Data Compression Conference M I Snowbird, UT, USA, March 27–30, 2018 A 1 2 3 4 5 6 7 8 Inpainting-Based Compression of Visual Data 9 10 11 12 13 14 15 16 Joachim Weickert 17 18 Faculty of Mathematics and Computer Science 19 20 21 22 Saarland University 23 24 Saarbr¨ucken, Germany 25 26 27 28 29 30 31 32 long term research funded by 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 Gottfried Wilhelm Leibniz Prize ERC Advanced Grant INCOVID 57 58 59 60 61 62 c 2018 Joachim Weickert 63 64 65 Collaborators 1 2 3 4 joint work with 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Naoufal Amrani Sarah Andris Egil Bae Zakaria Belhachmi Matthias Berg Pinak Bheed 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 Andr´esBruhn Dorin Bucur Irena Gali´c Sebastian Hoffmann Lena Karos Markus Mainberger 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Pascal Peter Gerlind Plonka Christian Schmaltz Joan Serra-Sagrista Ching Hoo Tang Martin Welk 61 62 63 64 65 Introduction (1) M I A 1 2 3 4 Introduction 5 6 7 8 9 10 Transform-based Codecs for Lossy Image and Video Compression 11 12 13 14 15 16 Widely-used lossy codecs such as JPEG, JPEG2000, HEVC are transform-based. 17 18 19 20 rely on sparse representation in transform domain (DCT, wavelets) 21 22 23 24 25 26 sparse approximation in terms of an orthogonal base offers many benefits: 27 28 29 30 clean theory 31 32 • 33 34 fast algorithms 35 36 • 37 38 39 40 quality deteriorates with sparsity 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 Introduction (2) 1 2 Quality Deterioration under JPEG Compression 3 4 5 6 7 8 original compression rate 10 : 1 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 compression rate 40 : 1 compression rate 80 : 1 61 62 63 64 65 Introduction (3) M I A 1 2 Further Improvements through Transform-Based Codecs 3 4 5 6 7 8 carefully engineered by many experts over many years 9 10 11 12 13 14 approaches have become highly sophisticated 15 16 17 18 19 20 increasingly difficult to achieve further fundamental improvements 21 22 by staying within the universe of transform-based ideas 23 24 25 26 27 28 29 30 Things that Transform-Based Codecs Ignore 31 32 33 34 35 36 more semantical, higher level representation of image structures: 37 38 39 40 – need for features (e.g. edges) rather than pixels 41 42 – filling-in effect of our visual system (Werner 1935) 43 44 45 46 47 48 approximation qualities beyond pixelwise quality measures (MSE, PSNR): 49 50 51 52 respecting e.g. perceptual similarities of stochastic textures 53 54 55 56 57 58 59 60 61 62 63 64 65 Introduction (4) 1 2 An Emerging Alternative: Inpainting-Based Compression 3 4 5 6 7 8 store only a very small, carefully selected subset of the data 9 10 11 12 13 14 reconstruct missing data with a suitable inpainting process 15 16 17 18 19 20 21 22 23 24 What is Inpainting ? 25 26 27 28 29 30 interpolation strategy for missing / degraded data 31 32 33 34 so far mainly used for repairing smaller parts of an image 35 36 37 38 39 40 two classes relevant for us: 41 42 43 44 PDE-based inpainting (Masnou/Morel 1998, Bertalmio et al. 2000) 45 46 • 47 48 exemplar-based inpainting (Efros/Leung 1999, Criminisi et al. 2004) 49 50 • 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 Introduction (5) M I A 1 2 PDE-Based Inpainting 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 original inpainted 37 38 39 40 Left: Original image with severe degradations by a text. From Bertalmio et al. (2000). Right: Result of 41 42 inpainting with an anisotropic diffusion process. 43 44 45 46 47 48 49 50 fills in with a partial differential equation (PDE), e.g. a diffusion process 51 52 53 54 uncorrupted data serve as fixed boundary conditions 55 56 57 58 PDE propagates this information to inpainting domain 59 60 61 62 local approach that can be justified from the statistics of natural images 63 64 65 Introduction (6) 1 2 Exemplar-Based Inpainting 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 original inpainted 37 38 39 40 Left: Original image. From Criminisi et al. (2004). Right: Inpainting result with Criminisi’s exemplar- 41 42 based approach. 43 44 45 46 47 48 49 50 clever copy-and-paste from similar patches within the image 51 52 53 54 nonlocal approach that exploits the intrinsic statistical properties of the image 55 56 57 58 good for texture-dominated images 59 60 61 62 computationally more expensive than PDE-based inpainting 63 64 65 Introduction (7) M I A 1 2 How Sparse can the Data be for PDE-based Inpainting ? 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 Image where only 2 percent of all pixels are known. Can you recognise what is depicted ? 57 58 59 60 61 62 63 64 65 Introduction (8) 1 2 Is Exemplar-based Inpainting also Useful for Sparse Representations ? 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 Texture synthesis with the exemplar-based approach of Criminisi et al. (2004), using our fast algorithm 57 58 59 60 with space-filling curves (Dahmen et al. 2017). 61 62 63 64 65 Introduction (9) M I A 1 2 What Happens if We Select Semantically More Important Data ? 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 original data near edges kept inpainted 45 46 47 48 49 50 Image reconstruction from edges. Left: Original image, 237 316 pixels. Middle: Edge set. Only × 51 52 the left and right neighbours of each Canny edge are stored. Right: Inpainting with the homogeneous 53 54 55 56 diffusion equation. 57 58 59 60 61 62 63 64 65 Introduction (10) 1 2 Chances for Inpainting-Based Compression 3 4 5 6 7 8 spatial representation more intuitive than transform-based coding 9 10 11 12 respects more properties of human visual system: 13 14 15 16 can use semantically important features such as edges 17 18 • 19 20 PDE-based inpainting resembles biological filling-in mechanisms 21 22 • 23 24 exemplar-based inpainting captures statistics of textures, 25 26 • 27 28 while renouncing pixelwise accuracy 29 30 31 32 33 34 may go beyond limits of sparse approximations in orthogonal bases 35 36 37 38 39 40 generic framework for various data types: 41 42 1D, 2D, 3D signals 43 44 • 45 46 still images and videos 47 48 • 49 50 scalar-, vector-, matrix-valued 51 52 • 53 54 surface data, graphs 55 56 • 57 58 dedicated codecs for specific applications 59 60 • 61 62 63 64 65 Introduction (11) M I A 1 2 Key Challenges for Inpaiting-Based Compression 3 4 5 6 7 8 1. Which data should be kept? 9 10 11 12 13 14 2. What are the most suitable inpainting processes? 15 16 17 18 19 20 3. How can the selected pixels be encoded in an efficient way? 21 22 23 24 4. Can one design fast algorithms for encoding and decoding? 25 26 27 28 29 30 31 32 33 34 These Problems are Highly Interrelated: 35 36 37 38 39 40 Optimality of data depends on inpainting process. 41 42 43 44 45 46 Suboptimal pixels can pay off if they are cheaper to encode. 47 48 49 50 There is a natural tradeoff between high efficiency and high quality. 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 Introduction (12) 1 2 How Does this Differ from Related Approaches ? 3 4 5 6 7 8 Classical Inpainting Methods: 9 10 11 12 PDE-based inpainting: 13 14 • Masnou/Morel 1998, Bertalmio et al.