SketchGAN: Joint Sketch Completion and Recognition with Generative Adversarial Network Fang Liu1;2 Xiaoming Deng1 Yu-Kun Lai3 Yong-Jin Liu4 ∗ Cuixia Ma1;2 ∗ Hongan Wang1;2 ∗ 1 State Key Laboratory of Computer Science and Beijing Key Lab of Human-Computer Interaction, Institute of Software, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences 3 Cardiff University 4 Tsinghua University Abstract Hand-drawn sketch recognition is a fundamental prob- lem in computer vision, widely used in sketch-based image and video retrieval, editing, and reorganization. Previous methods often assume that a complete sketch is used as in- put; however, hand-drawn sketches in common application scenarios are often incomplete, which makes sketch recog- nition a challenging problem. In this paper, we propose SketchGAN, a new generative adversarial network (GAN) Figure 1. Illustration of our joint sketch completion and recog- based approach that jointly completes and recognizes a nition pipeline. We use sketch completion as an intermediate sketch, boosting the performance of both tasks. Specifically, step for recognizing incomplete sketches. Experiments show that the sketch recognition performance can be improved greatly with we use a cascade Encode-Decoder network to complete the completed sketch. input sketch in an iterative manner, and employ an auxiliary sketch recognition task to recognize the completed sketch. Experiments on the Sketchy database benchmark demon- ceptually closure contour and uses a sparse set of lines to strate that our joint learning approach achieves competitive capture main shape characteristics. Although sketch recog- sketch completion and recognition performance compared nition [17] and classification [47, 54] have achieved much with the state-of-the-art methods. Further experiments us- progress in the last decade, little attention is paid to sketch ing several sketch-based applications also validate the per- completion. Hand-drawn sketches lack texture and contex- formance of our method. tual information, and are generally known to be more am- biguous than natural images. Therefore, image completion methods designed for color images do not work well for 1. Introduction sketches directly. Moreover, the sketches of the same object may be drawn in diverse styles due to the nature of freehand Sketch is a natural symbol to express abstract ideas in a sketches, which makes sketch completion very challenging. straightforward way, which is widely used in computer vi- Sketch is closely related to contour, both of which are sion [14, 27, 45], multimedia reorganization [7, 31] and hu- visually closed outlines of objects using black and white man computer interactions [39]. Hand-drawn sketch recog- pixels. The major difference between sketch and contour is nition is a fundamental problem in many sketch-based ap- that contour always keeps consistent with the correspond- plications. However, they are often incomplete, e.g., quick ing real image, while sketch has more diversity [13] and ab- interim sketches, and sketches of multiple overlapping ob- straction levels [36] in appearance. A lot of effort has been jects where some objects are partially occluded. In this pa- made on contour completion [34, 42, 50], aiming to either per, we explore the sketch completion problem, which will extract perceptually-salient contours from images or to find benefit related research such as sketch recognition, sketch the boundaries of objects and surfaces in a scene. The con- editing and sketch-based image retrieval (SBIR). tour completion methods cannot be applied to sketch com- Sketch completion aims to infer reasonable lines to fill pletion directly since contour completion enforces accurate in missing strokes in a sketch. An ideal sketch contains per- alignment with object boundaries in original images, while ∗Corresponding authors sketches are not associated with specific images. Another related problem is image completion. How- 2. Related Work ever, even the state-of-the-art convolutional neural network (CNN) based methods [18, 20] designed for image comple- Contour completion. Contour completion plays an impor- tion have poor performance on sketch completion due to the tant role in visual perception, and it aims to group frag- lack of color and texture context. mented low-level edge elements into perceptually coherent and salient contours [34]. Several learning based meth- In this paper, we propose a generative adversarial net- ods for edge detection are proposed recently [2, 57]. Xie work (GAN) for sketch completion, called SketchGAN. et al. [48] use a convolutional network to pool informa- Our method is not category specific, and can complete in- tion from the whole image. These local [23] or mid-level put sketches of different categories. Our key idea is to contour detection models [34, 42, 50] often ignore the im- jointly conduct sketch completion and recognition tasks. portant problem of contour closure, which has been widely The generator of our SketchGAN consists of a multi-stage studied more recently [35]. Instead of using segmentation, cascade network, where each stage consists of a conditional they propose a novel approach for contour completion that GAN [21], and the cascade structure can further boost the results in closure contour in the contour domain. performance of a single stage. The first stage uses an in- Our paper focuses on sketch completion, which aims to complete sketch as input, and transforms it to a roughly inpaint corrupted sketches or hand-drawn lines with miss- completed output. A later stage concatenates the original ing lines instead of extracting edges from natural images in input and all the output sketches of the previous stages, and contour completion. Sketch completion is more challenging feeds them to the generator network in the current stage. than contour completion, due to the ambiguity nature, large Since we aim to design a general sketch completion frame- variations and different abstraction levels among sketches. work suitable for multi-category sketch completion, moti- Image completion. Another related problem to sketch vated by multi-task learning in which auxiliary tasks can completion is image completion, both of which aim to fill often improve the performance of the main task, we adopt in missing strokes or regions. Previous image completion sketch recognition as the auxiliary task. Conceptually, the methods can be classified into diffusion-based image syn- high level recognition task benefits from the global context thesis and patch-based methods. Diffusion-based methods structure of the sketch completion results, and in turn train- apply filters to propagate the local image appearance near ing them together promotes better sketch completion. the target regions to fill them [4, 25]. Patch-based ap- Experiments on a widely-used dataset show that our proaches are proposed for larger missing areas and com- method is superior to state-of-the-art methods, and the plicated image completion [12, 18, 24]. Unfortunately, de- sketch completion and recognition tasks benefit each other. pending on the hand-crafted features, these traditional im- An example of our joint sketch completion and recognition age completion approaches can only repair small corrupted is illustrated in Figure 1, which demonstrates the mutual areas, and cannot generate new objects which do not exist benefits of addressing both tasks simultaneously. We fur- in the original corrupted images. Recently, convolutional ther demonstrate that our method is very helpful to improve networks have been applied to image completion [37, 51]. the performance of downstream applications such as incom- A joint multiplanar autoregressive and low-rank based ap- plete sketch recognition and sketch editing. proach has been proposed for image completion from ran- Our contributions are summarized as follows. dom sampling [26]. Beak et al. [3] present a multiview im- age completion method that provides geometric consistency 1. To the best of our knowledge, we are the first to solve among different views by propagating spatial structures. the problem of sketch completion, which can inspire Sketch completion and image completion both belong to further sketch-based research. generative models. Compared to image completion, sketch completion is more challenging due to the fact that hand- 2. We propose a new network architecture for sketch drawn sketches lack color and contextual information for completion, namely SketchGAN, which handles sketch understanding. Although these image completion sketches of different categories. Our method jointly methods work very well on incomplete natural images, they conducts sketch completion and an auxiliary sketch are not suitable for the sketch completion task due to the recognition task, and we find that sketch completion style and content gap between natural images and sketches. and recognition tasks benefit each other. Generative adversarial networks (GANs). GANs [15] achieved impressive results for image generation [5], im- 3. Experiments demonstrate that the output of Sketch- age completion [20], and image editing [38]. GAN trains GAN can enhance the performance of typical sketch two networks, a generative model G and a discriminative applications such as incomplete sketch recognition and model D. Upon convergence, D can reject images that look sketch editing. fake, and G can produce high-quality images, which can fool D. A variety of GANs have been proposed for im- Figure 2. Overview of our SketchGAN architecture. Our network is built upon conditional GANs with a cascade encode-decoder architec- ture. The network uses an incomplete sketch as input, and predicts a completed sketch and its sketch classification label. We illustrate a three-stage network which is the chosen architecture. In the first stage, we use the original incomplete sketch x as input and get a roughly completed sketch output y1. In the next stage, we feed x and y1 to the generator and get the output of the second cascade stage, y2. Finally, the outputs of all the previous cascade stages y1 and y2, together with the original x, are fed to the generator. We feed the output of the final cascade stage y3 to the discriminator and the sketch classifier.