Structure Tensor Based Image Interpolation Method
Ahmadreza Baghaie and Zeyun Yu University of Wisconsin-Milwaukee, WI, USA
Abstract — Feature preserving image interpolation is an active area in image processing field. In this paper a new direct edge directed image super-resolution algorithm based on structure tensors is proposed. Using an isotropic Gaussian filter, the structure tensor at each pixel of the input image is computed and the pixels are classified to three distinct classes; uniform region, corners and edges, according to the eigenvalues of the structure tensor. Due to application of the isotropic Gaussian filter, the classification is robust to noise presented in image. Based on the tangent eigenvector of the structure tensor, the edge direction is determined and used for interpolation along the edges. In comparison to some previous edge directed image interpolation methods, the proposed method achieves higher quality in both subjective and objective aspects. Also the proposed method outperforms previous methods in case of noisy and JPEG compressed images. Furthermore, without the need for optimization in the process, the algorithm can achieve higher speed 1. . Index Terms — Local structure tensor, Image interpolation, Super-Resolution, Edge-directed interpolation
1. INTRODUCTION Feature preserving image interpolation is an active area in the image processing field, from everyday digital pictures to application-oriented medical and satellite images. Many methods have been proposed in the past decades to tackle this problem [1-20]. Generally speaking, the methods for image interpolation/super-resolution can be divided in 3 different categories: 1) Direct Interpolation methods, 2) PDE based interpolation methods and 3) Optimization based interpolation methods. All of these methods have their pros and cons regarding their simplicity of implementation, computational complexity and performance. The proposed method in this paper is a direct interpolation method without the need for any optimization in the process. Also in terms of computational time, the proposed method can achieve the result in less than one second for an image of typical size using MEX based implementation. This feature along with not using any optimization procedure, as well as being robust in case of noisy images, make this method a suitable choice for implementation in everyday used electronic devices. Nearest neighbor and bilinear interpolation are two simple methods for image interpolation [1]. Despite the simplicity in implementation and very low computational cost, these methods suffer from severe blocky artifacts, as well as blurring and ringing artifacts near the edges. Although better performance can be achieved by using higher order splines, rather than 0 and 1 order splines as in the nearest neighbor and bilinear methods, higher order spline methods still contain oscillatory edges and ringing artifacts [2]. The main reason is that these methods don’t take into consideration any information other than intensity values. In other words, they are intensity based and not feature (edge) based. So even though they are easy to implement and need low computational cost, they are not suitable for most of applications. The final recipient of any image processing algorithm is the human visual system which is
1 Ahmadreza Baghaie (Corresponding author) is with the Department of Electrical Engineering at University of Wisconsin - Milwaukee, Milwaukee, WI 53211 ([email protected]). Zeyun Yu is with the Department of Computer Science at University of Wisconsin - Milwaukee, Milwaukee, WI 53211 ([email protected]). very feature sensitive. These features are mostly edges and corners within the image. Also sharpness of the final image is of high importance. Based on these criteria, the previously mentioned methods, despite their technical advantages, are not satisfactory. So the need for introducing new approaches and novel models for image interpolation which satisfy the human visual system has been emerged in the past decades and many methods have been proposed. Some of these methods will be mentioned here. Edge directed methods usually are the first ones that come to notice when dealing with image interpolation problem. In 2001 a method called NEDI was proposed which performs based on the assumption that every image can be modeled as a locally stationary Gaussian process [3]. Based on this assumption, the local covariance coefficients from the low resolution (LR) image is estimated and then interpolation is done based on the geometric duality between the LR covariance and the high resolution (HR) covariance. An improved version of NEDI algorithm called iNEDI is proposed later which achieves higher scores in terms of subjective and objective image quality measures relative to NEDI with the cost of needing more computational time [4]. Another edge directed image interpolation method is ICBI which works based on an estimation of the edge orientation using second order derivative of the image [5]. DFDF method [6] is another method in this category which utilizes directional filtering and data fusion. In DFDF, at first, two observation sets are defined in two orthogonal directions for each pixel to be interpolated. Then these two estimates will be fused using Linear Minimum Mean Square Error (LMMSE) in order to achieve a more robust estimate for the missing pixel. Methods proposed in [7-15] also are good examples of edge directed image interpolation. In [7], the method is based on partitioning the input image into homogeneous and edge areas with regard to local structure of the image and then, interpolating each parts differently, bilinear interpolation for homogeneous regions and an adaptive edge oriented method for edge pixels. In [8], a modified edge adaptive bilinear image interpolation method called EASE is proposed. This modified version is achieved using the classical interpolation error theorem. In [9], a new directional cubic convolution (CC) interpolation scheme is proposed. In [10], an interpolation framework is proposed in which denoising and image sharpening are embedded together. In this method, bilateral filtering method is used to partition the input image into detail and base layers, and then edge preserving interpolation method is applied to each layer. In [11], the edge information of the LR image is first estimated using the modified Leung-Malik filter bank, and then this information is converted into that of HR image by using a mapping function. In [12], a fast image interpolation method with adaptive weights is proposed motivated by Inverse Distance Weighting (IDW). The use of Radial Basis Functions (RBFs) to solve image interpolation problem is investigated in [13, 14]. In [15], a soft decision interpolation technique is proposed which estimates missing pixels in groups rather than one at a time. They use a piecewise 2-D autoregressive (AR) model to determine the local structure of the scene. Even though the above mentioned methods are of a wide range of use and discipline, still there are more methods that are not discussed here; like Partial Differential Equation (PDE) based methods [16, 17, 27], and regularization based methods [18-20]. The reader will be referred to the papers and their references for more information on these classes of image interpolation methods. As can be seen, each of the mentioned methods deals with the image interpolation problem from a different angle. But still image interpolation is an open problem and there is room for improvement. In this paper, a new edge-directed method based on structure tensor will be proposed which its strength is not only in reconstructing edges in the HR image, but also is more robust in case of noise. The proposed method is very simple and easy to implement and based on the conducted experiments, outperforms the most common image interpolation methods. For comparison, five well-known image interpolation methods are considered: NEDI [3], DFDF [5], ICBI [6], KR [26] and iNEDI [4]. Tests were conducted for noise-free, noisy and JPEG compressed images. For completeness of the comparison another structure tensor-based method by Roussos and Maragos [27] is also considered. This method (RM) can be categorized as a PDE-based technique. In this method at first an initial interpolation is done by Fourier zero-padding and de-convolution. The result of this stage suffers from significant ringing artifacts. Using a tensor-driven diffusion process, the ringing artifacts are removed. The main assumption in this method is that the process of interpolation is a reversible process which cannot be hold always. Based on this assumption, interpolation is done by first applying an anti-aliasing low-pass filter followed by sampling. This assumption can be problematic especially in the case of naïve sub-sampling which is the case used in this paper. In naïve sub-sampling of factor , one pixel is chosen from each pixels of the image without any anti-aliasing filtering. This will cause high amount of ringing artifacts near edges introduced by the first stage of the method as well as sever stair-cased edges which cannot be resolved properly using the tensor-driven diffusion process. More discussion will be given in the following sections regarding this issue. Here the online implementation of this method is used implemented by Getreuer [28]. The rest of the paper is organized as follows: in Section 2 a brief introduction will be given on structure tensor computation and its theoretical aspects. Then in Section 3, the proposed interpolation method will be described in more detail. Section 4 contains the implementation aspects, image quality measures that being used and tables of objective and subjective comparison between the five above mentioned methods and the proposed method, as well as some of the final results. Section 5 concludes the paper.
2. LOCAL STRUCTURE TENSOR Local structure tensors have been used in image processing to solve problems such as anisotropic filtering [21, 22] and motion detection [23]. This method uses the gradient information of an image in order to determine the orientation information of the edges and corners. The structure tensor is defined as: