Single Image Super-resolution using Vectorization and Texture Synthesis Kaoning Hu1, Dongeun Lee1 and Tianyang Wang2 1Department of Computer Science and Information Systems, Texas A&M University - Commerce, Commerce, Texas, U.S.A. 2Department of Computer Science and Information Technology, Austin Peay State University, Clarksville, Tennessee, U.S.A. Keywords: SISR, Image Vectorization, Texture Synthesis, KS Test. Abstract: Image super-resolution is a very useful tool in science and art. In this paper, we propose a novel method for single image super-resolution that combines image vectorization and texture synthesis. Image vectorization is the conversion from a raster image to a vector image. While image vectorization algorithms can trace the fine edges of images, they will sacrifice color and texture information. In contrast, texture synthesis techniques, which have been previously used in image super-resolution, can reasonably create high-resolution color and texture information, except that they sometimes fail to trace the edges of images correctly. In this work, we adopt the image vectorization to the edges of the original image, and the texture synthesis based on the Kolmogorov–Smirnov test (KS test) to the non-edge regions of the original image. The goal is to generate a plausible, visually pleasing detailed higher resolution version of the original image. In particular, our method works very well on the images of natural animals. 1 INTRODUCTION on the test set of low-resolution images to create a high-resolution image (Yang et al., 2019; Zhang et al., Image super-resolution is a technique that enhances 2019). With deep neural networks, they can perform the resolution of digital images. It has a various very well on general images. However, these methods range of applications including medical image pro- rely largely on the quality of the similarity between cessing, satellite image processing, art, and entertain- the training set and the test set, and also require a large ment. In this paper, we focus on single image super- training set (Wang et al., 2013). The reconstruction- resolution (SISR), which is a classic computer vision based methods assume that the downsampled version problem to create a visually pleasing high-resolution of the target high-resolution image is similar to the image from a single low-resolution input image. In low-resolution image that we have (Wang et al., 2013; many situations, a single low-resolution image is the Damkat, 2011). However, it is observed that the re- only source of data when a video sequence or high- constructed edges sometimes are unnaturally sharp resolution footage does not exist, thus SISR tech- and are often accompanied by undesired artifacts such niques are highly demanded. However, since the as ringing (Wang et al., 2013). The edge-directed ground truth (i.e. the high-resolution information) methods place emphasis on processing edges in the does not exist in the low-resolution image, SISR is target high-resolution image by tracing the edge with still a challenging problem. the gradient of the input low-resolution image (Wang In general, there are four types of super- et al., 2013). While they can create smooth and nat- resolution methods (Sun et al., 2008): interpolation- ural edges in the target image, these methods can- based, learning-based, reconstruction-based, and not always properly handle certain texture regions edge-directed. The interpolation-based methods, such such as grass and animal hair. Therefore, the edge- as bilinear and bicubic interpolations, are simple directed methods work better in upscaling of video and fast, and have been widely used in commercial game pixel arts (Stasik and Balcerek, 2017) and depth software. However, these methods typically make images (Liu and Ling, 2020; Xie et al., 2015), where the edges of the image blurry. The learning-based there is not much texture information. Since each type methods first learn the correspondences between low- of super-resolution methods has advantages and dis- resolution and high-resolution image patches from the advantages, methods that combine two or more types training dataset, and then apply the learned model also exist (Yang et al., 2017; Liu et al., 2019). 512 Hu, K., Lee, D. and Wang, T. Single Image Super-resolution using Vectorization and Texture Synthesis. DOI: 10.5220/0010325505120519 In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 4: VISAPP, pages 512-519 ISBN: 978-989-758-488-6 Copyright c 2021 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved Single Image Super-resolution using Vectorization and Texture Synthesis Due to the different types of image capturing de- 2 RELATED WORK vices, and objects in the images, there are many dif- ferent categories of images. As such, SISR methods In recent years, deep learning has been successfully have been developed to particularly work on certain applied to SISR, and some non-deep learning meth- types of images. Besides the previously mentioned ods have also been successfully implemented in com- game pixel arts and depth images, there are also meth- mercial software. Successful commercial software in- ods that work on remote sensing images (Jiang et al., cludes DCCI2x (Zhou et al., 2012) and ON1 Resize. 2019), medical images (Chen et al., 2018), and docu- DCCI2x is an edge-directed interpolation method ment and text images (Wang et al., 2019). that adapts to different edge structures of images. We have observed that most of the previous work ON1 Resize uses the technology of the U.S. patent emphasize enhancing edges in the images. Few meth- “Genuine Fractals R ” (Faber and Dougherty, 2007), ods exist that put emphasis on texture regions (Saj- which exploits the self-similarity of an image to in- jadi et al., 2017). In this paper, we propose a novel crease its size while preserving its details. Typical approach that combines texture synthesis and image deep learning-based methods include (Zhang et al., vectorization to upscale a single digital image. In par- 2019; Zhang et al., 2018), and (Ledig et al., 2017). ticular, we focus attention on SISR for natural animal In (Zhang et al., 2019), a novel degradation model images, as our method works very well in processing for SISR was proposed, leveraging the classical blind the hair and feather parts of those images. deblurring methods. In (Zhang et al., 2018), the au- A digital image of natural animals or objects typ- thors aimed to improve framework practicability by ically contains edge regions, where the brightness designing a single convolutional network that can take changes sharply along contours, and non-edge re- both blur kernel and noise level as input. Ledig et gions, where there is no significant edge. Although al. (Ledig et al., 2017) pioneered to propose a genera- texture regions still have small edges as the colors of tive adversarial network (GAN) for SISR, namely SR- neighboring pixels could be different, these edges are GAN. All these methods achieved competitive quan- usually too tiny and irregular to be traced. For this titative results (in PSNR/SSIM) and appealing qual- reason, we consider texture regions as non-edge re- itative results. However, most of these methods em- gions. Edge regions and non-edge regions show sig- phasized processing of the edge regions, but not the nificantly different visual characteristics. Therefore, texture regions. In 2017, Sajjadi et al. (Sajjadi et al., it is reasonable to treat them separately and apply dif- 2017) proposed a learning-based method that uses ferent upscaling operations. convolutional neural networks in an adversarial train- Specifically, for the edge regions, we will adopt ing setting with a texture loss function. Their goal Potrace (Selinger, 2003), an image vectorization was to upscale low-resolution images by a 4 × 4 fac- method, to trace the edge and upscale the regions. Po- tor with synthesized realistic textures. trace approximates the edges in an image first with It is understandable that in many applications, the polygons, and then sharp corners are approximated primary goal is not to faithfully reconstruct a high- with sharp angles, while non-sharp corners are ap- resolution version of an input low-resolution image, proximated with Bezier´ curves. For the non-edge but to improve its appearance (Damkat, 2011). In regions, we will assume they contain texture infor- 2011, Damkat (Damkat, 2011) proposed an SISR mation, and will exploit the self-similarity of the method using example-based texture synthesis. The texture and apply a texture synthesis method based assumption is that even in a low-resolution image, on Kolmogorov–Smirnov Test (KS test) (Massey Jr, there are still a number of scale-invariant elements 1951) to upscale the texture regions. Our work uses and self-similarity across scale. For example, animal the KS test since it can naturally compare the proba- hair looks similar at different scales, so upscaling may bility distributions of an upscaled texture patch and a be achieved by generating more hairs instead of mak- low-resolution texture patch whose sizes are different. ing hairs thicker. Their method achieved relatively The paper is organized as follows. In Section 2, good result in the texture regions except some salt- related work, including conventional algorithms and and-pepper noises. deep learning algorithms, is discussed. In Section 3, The idea of self-similarity can be also explained we will introduce the proposed method in detail. In by fractals, whose applications include fractal com- Section 4, we will demonstrate results and evalu- pression (Jacquin et al., 1992; Wohlberg and De Jager, ate the performance of our image super-resolution 1999). Fractal compression takes advantage of self- method. The conclusion and future work will be similarity in images by analyzing and representing stated in Section 5.