DOI 10.1515/jisys-2013-0033 Journal of Intelligent Systems 2013; 22(3): 299–315 Research Article Rajkumar L. Biradar* and Vinayadatt V. Kohir Texture Inpainting Using Covariance in Wavelet Domain Abstract: In this article, the covariance of wavelet transform coefficients is used to obtain a texture inpainted image. The covariance is obtained by the maximum likelihood estimate. The integration of wavelet decomposition and maximum likelihood estimate of a group of pixels (texture) captures the best-fitting texture used to fill in the inpainting area. The image is decomposed into four wavelet coefficient images by using wavelet transform. These wavelet coefficient images are divided into small square patches. The damaged region in each coefficient image is filled by searching a similar square patch around the damaged region in that particular wavelet coefficient image by using covariance. Keywords: Inpainting, texture, maximum likelihood estimate, covariance, wavelet transform. *Corresponding author: Rajkumar L. Biradar, G. Narayanamma Institute of Technology and Science, Electronics and Tel ETM Department, Shaikpet, Hyderabad 500008, India, e-mail: [email protected] Vinayadatt V. Kohir: Poojya Doddappa Appa College of Engineering, Gulbarga 585102, India 1 Introduction Image inpainting is the technique of filling in damaged regions in a non-detect- able way for an observer who does not know the original damaged image. The concept of digital inpainting was introduced by Bertalmio et al. [3]. In the most conventional form, the user selects an area for inpainting and the algorithm auto- matically fills in the region with information surrounding it without loss of any perceptual quality. Inpainting techniques are broadly categorized as structure inpainting and texture inpainting. The structure filling algorithms rely on filling the inner area with information from a structured region, which is the boundary of the region to be inpainted. Texture inpainting techniques fill in the damaged or missed regions using the similar neighborhood region in the image. They try to match the statistics of damaged regions to the statistics of known regions in the neighborhood of a damaged area. 300 R. L. Biradar and V. V. Kohir In this article, we address the issue of texture inpainting by integrating the maximum likelihood estimate (MLE) of covariance and wavelet transform. The original damaged texture image is decomposed into four quarter-size wavelet coefficient images by using wavelet transform. Then, each wavelet coefficient image is divided into small square patches. The damaged regions in all wavelet coefficient images are filled by searching a similar patch around the damaged region in that particular image. The covariance is used to find similarity between two patches. After filling in the damaged region of each wavelet coeffi- cient image in the wavelet domain, inverse discrete wavelet transform (DWT) is used to get the inpainted image. The proposed algorithm achieves better results for removing of objects and inpainting of a larger damaged region in the texture image. 2 Previous Work Inpainting techniques can be classified into two broad categories: structure inpainting and texture inpainting. 2.1 Structure Inpainting In structure inpainting, the damaged region is filled by diffusing with the neigh- boring information of the image. The diffusion of neighboring information is achieved using a partial differential equation (PDE). Bertalmio et al. [3] have developed a digital inpainting technique algorithm based on the PDE, and it is an extension of the level lines based disocclusion method proposed by Masnou and Morel [16]. In Bertalmio et al.’s inpainting, the direction of arrival of isophotes is maintained by computing the direction of the largest spatial change. The direc- tion is obtained by computing a discretized gradient vector. Bertalmio et al. [4] used the ideas from computational fluid dynamics to propagate isophote lines continuously from the exterior into the region to be inpainted. Chan and Shen proposed two image inpainting algorithms. The total variation (TV) inpainting model [6, 13] uses Euler–Lagrange modeling. Inside the inpainting domain, this model employs anisotropic diffusion [18] based on the contrast of the isophotes. The curvature-driven diffusion model [7, 8] is an extended TV algorithm in which the geometric information of isophotes is taken into account while defining the strength of the diffusion process, thus allowing the inpainting to proceed over inpainting regions. Texture Inpainting Using Covariance 301 Zhang [21] introduced fractional-order image inpainting (a projection inter- polation method) into metal artifacts reduction in computer tomography (CT) images. They introduced a fast non-iterative method based on the fast marching method and coherence transport for metal artifacts reduction in CT. Zhang [22] also proposed an image inpainting by combining the TV model with a fractional derivative called fractional-order TV image inpainting model. They introduced a new class of fractional-order variational image inpainting models in both space and wavelet domains. All PDE-based methods require a difficult implementation process. Some steps are numerically unstable and the inpainting process is slow. For large damaged regions, the results show a blocky effect. 2.2 Texture Inpainting Texture inpainting is relatively difficult as texture is to be either synthesized or pasted from the surrounding area of the damaged region based on some similar properties. The statistical properties of texture play an important role in syn- thesizing and measuring the similarity between two textures. The damaged or missing regions are filled in using similar neighborhood texture in the image. The statistics of the boundary of damaged regions should be similar to the statistics of known neighborhood regions. Hirani and Totsuka [15] combined frequency and spatial domain information to fill a given region with a selected texture. Other texture synthesis algorithms, e.g., see refs. [1, 20], can be used as well to recreate a preselected texture to fill in a square region to be inpainted. The algorithm mainly deals with texture synthesis and not with structured background. Efros and Leung [11] proposed a non-par- ametric texture synthesis model-based Markov random field to inpaint textural images. In their method, first a neighborhood around a damaged pixel is selected and then the known regions of the image are searched to find the most similar region to the selected neighborhood. Finally, the central pixel found in the neigh- borhood is copied to the damaged pixel. This method is time consuming and does not produce good results in structured regions. Criminisi and Perez [10] modified the model in ref. [11] to achieve better results. Chang [9] showed that the priority function adopted in ref. [11] is not a well-defined function and may become unreliable after a number of iterations. Therefore, they proposed a new function to assign priority to pixels providing a robust exemplar-based algorithm. Bertalmio et al. [5] proposed to decompose the original image into textural and structural subimages. The structural subimage is reconstructed using a structure algorithm and the textural subimage is restored 302 R. L. Biradar and V. V. Kohir by a texture synthesis approach. Then, two processed components are combined to obtain an inpainted image. A similar approach is proposed by Grossauer [14], in which instead of decomposing the image, the original image is segmented into two subregions. Bai et al. [2] proposed completion of missing parts by structure propaga- tion and synthesizing the regions along the salient structures specified by the user. After structure completion, a finer algorithm is used to fill in the remain- ing unknown regions. It can prevent erroneous matching blocks and reduce the breaking of salient structures. The proposed algorithm copies the texture in wavelet domain by using the MLE of covariance in a localized window around the inpainting area. 3 Wavelet Transform Wavelet transforms are based on small waves, called wavelets, of varying fre- quency and limited time duration. This allow us to analyze both time and fre- quency domain information of a function simultaneously. A wavelet basis starts with two orthogonal functions: the scaling function φ(x) or father wavelet, and the wavelet function ψ(x) or mother wavelet, which allows detecting frequency components and their location in the time or spatial domain. The one-dimension [12] scaling function φ(x) and the wavelet function are defined as φ =−φ ()xh∑ φ()nx2(2)n , (1) n ψφ=− ()xh∑ ψ()nx2(2)n , (2) n where hφ(n) and hψ(n) are called the scaling function coefficient vector and the wavelet function coefficient vector, respectively. In two dimensions, a two-dimension scaling function φ(x, y) and three two- dimension wavelet functions, ψH(x, y), ψV(x, y), and ψD(x, y), are required. Each is the product of a one-dimension scaling function φ(.) and the corresponding wavelet function ψ(.). Excluding the products that produce one-dimensional results, such as φ(x)ψ(x), the four remaining products produce the separable scaling function φ(x, y) = φ(x)φ( y), (3) ψH(x, y) = ψ(x)φ( y), (4) Texture Inpainting Using Covariance 303 ψV(x, y) = φ(x)ψ( y), (5) ψD(x, y) = ψ(x)ψ( y). (6) These wavelets measure variations in intensity or gray level of image along different directions. ψH(.), ψV(.), and ψD(.) measure variations along columns (horizontal edges), rows (vertical edges), and diagonals, respectively. By scaling and translation of these orthogonal functions, we obtain a complete basis set and they are defined as j/2 j j φj, m, n(x, y) = 2 φ(2 x – m, 2 y – n), (7) i j/2 i j j ψj, m, n(x, y) = 2 ψ (2 x – m, 2 y – n), (8) where i = {H, V, D} and identifies the directional wavelets of Eqs. (3)–(6). The two-dimensional DWT of image f(x, y) of size N × N is 1 NN−−11 Wj(,mn,)= fx(,yx)(φ ,)y , φ 0,∑∑ jm0 ,n (9) N xy==00 1 NN−−11 ii= ψ Wjψ(,mn,) ∑∑fx(,yx)(jm,,n ,)y .