User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks
Total Page:16
File Type:pdf, Size:1020Kb
User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks Yuanzheng Ci Xinzhu Ma Zhihui Wang∗† DUT-RU International School of DUT-RU International School of Key Laboratory for Ubiquitous Information Science & Engineering Information Science & Engineering Network and Service Soware of Dalian University of Technology Dalian University of Technology Liaoning Province [email protected] [email protected] Dalian University of Technology [email protected] Haojie Liy Zhongxuan Luoy Key Laboratory for Ubiquitous Key Laboratory for Ubiquitous Network and Service Soware of Network and Service Soware of Liaoning Province Liaoning Province Dalian University of Technology Dalian University of Technology [email protected] [email protected] ABSTRACT Scribble colors based line art colorization is a challenging computer vision problem since neither greyscale values nor semantic informa- tion is presented in line arts, and the lack of authentic illustration- line art training pairs also increases diculty of model generaliza- tion. Recently, several Generative Adversarial Nets (GANs) based methods have achieved great success. ey can generate colorized illustrations conditioned on given line art and color hints. However, these methods fail to capture the authentic illustration distributions and are hence perceptually unsatisfying in the sense that they oen lack accurate shading. To address these challenges, we propose a novel deep conditional adversarial architecture for scribble based anime line art colorization. Specically, we integrate the condi- tional framework with WGAN-GP criteria as well as the perceptual loss to enable us to robustly train a deep network that makes the synthesized images more natural and real. We also introduce a local features network that is independent of synthetic data. With GANs Figure 1: Our proposed method colorizes a line art composed conditioned on features from such network, we notably increase by artist (le ) based on guided stroke colors (top center, best the generalization capability over ”in the wild” line arts. Further- viewed with grey background) and learned priors. Line art more, we collect two datasets that provide high-quality colorful image is from our collected line art dataset. illustrations and authentic line arts for training and benchmarking. With the proposed model trained on our illustration dataset, we demonstrate that images synthesized by the presented approach are arXiv:1808.03240v2 [cs.CV] 10 Aug 2018 CCS CONCEPTS considerably more realistic and precise than alternative approaches. •Computing methodologies ! Image manipulation; Computer vision; Neural networks; ∗Corresponding author. †Also with DUT-RU International School of Information Science & Engineering, Dalian University of Technology. KEYWORDS Interactive Colorization; GANs; Edit Propagation Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation 1 INTRODUCTION on the rst page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permied. To copy otherwise, or republish, Line art colorization plays a critical role in the workow of artis- to post on servers or to redistribute to lists, requires prior specic permission and/or a tic work such as the composition of illustration and animation. fee. Request permissions from [email protected]. Colorizing the in-between frames of animation as well as simple MM ’18, Seoul, Republic of Korea © 2018 ACM. 978-1-4503-5665-7/18/10...$15.00 illustrations involves a large portion of redundant works. Never- DOI: 10.1145/3240508.3240661 theless, there is no automatic colorization pipelines for line art colorization. As a consequence, it is performed manually using of the state-of-the-art stroke-based user-guided line art colorization image editing applications such as Photoshop and PaintMan. Espe- methods in both qualitative and quantitative evaluations. cially in the animation industry, where it is even known as a hard In summary, the key contributions of this paper are summarized labor. It’s a challenging task to develop a fast and straightforward as follows. way to produce illustration-realistic imagery with line arts. • We propose an illustration synthesis architecture and a loss Several recent methods have explored approaches for guided for stroke-based user-guided anime line art colorization, image colorization [13, 14, 18, 34, 39, 49, 59, 61]. Some works whose results are signicantly beer than existing guided are mainly focused on images containing greyscale information line art colorization methods. [14, 18, 61]. A user can colorize a greyscale image by color points • We introduce a novel local features network in the cGAN or color histograms [61], or colorize a manga based on reference architecture to enhance the generalization ability of the images [18] with color points [14]. ese methods can achieve networks trained with synthetic data. impressive results but can neither handle sparse input like line arts, • e colorization network in our cGAN is dierent with ex- nor color stroke-based input which is easier for users to intervene. isting GANs’ generators in that it is much deeper, with sev- To address these issues, recently, researchers have also explored eral specially designed layers to both increase the receptive more data-driven colorization methods [13, 34, 49]. ese method eld and the network capacity. It makes the synthesized colorize a line art/sketch by scribble color strokes based on learned images more natural and real. priors over synthetic sketches. is makes colorizing a new sketch • We collect two datasets that provide quality illustration cheaper and easier. Nevertheless, the results oen contains unreal- training data and line art test benchmark. istic colorization and artifacts. More fundamentally, the overing over synthetic data is not well handled, thus the network can hardly 2 RELATED WORK perform well with ”in the wild” data. PaintsChainer[39] tries to ad- dress these issues with three models (with two models not opened) 2.1 User-guided colorization proposed. However, the eye pleasing results were achieved in the Early interactive colorization methods [20, 31] propagate stroke cost of losing realistic textures and global shading that authentic colors with low-level similarity metrics. ese methods based on illustrations preserve. the assumption that the adjacent pixels with similar luminousness In this paper, we propose a novel conditional adversarial illus- in greyscale images should have a similar color, and numerous user tration synthesis architecture trained fully on synthetic line arts. interactions are typically required to achieve realistic colorization Unlike existing approaches using typical conditional networks [23], results. Later research studies improved and extended this method we combine a cGAN with a pretrained local features network, i.e., by using chrominance blending [57], specic schemes for dierent both generator and discriminator network are only conditioned textures [42], beer similarity metrics [35] and global optimiza- on the output of local features network to increase the general- tion with all-pair constraints [1, 56]. Learning methods such as ization ability over authentic line arts. With proposed networks boosting [33], manifold learning [7] and neural networks [12, 61] trained in an end-to-end manner with synthetic line arts, we are have also been proposed to propagate stroke colors with learned able to generate illustration-realistic colorization from sparse, au- priors. In addition to local control, some approaches proposed to thentic line art boundaries and color strokes. is means we do not colorize images by transferring the color theme [14, 18, 32] or color suer from overing issue over synthetic data. In addition, we palee [4, 61] of the reference image. While these methods make randomly simulate user interactions during training, allowing the use of the greyscale information of the source image which is not network to propagate the sparse stroke colors to semantic scene available for line art/sketch, Scribbler [48] developed a system to elements. Inspired by [27] which obtained state-of-the art results transform sketches of specic categories to real images with scrib- in motion deblurring, we fuse the conditional framework [23] with ble color strokes. Frans [13] and Liu et al. [34] proposed methods WGAN-GP [17] and perceptual loss [24] as the criterion in the for guided line art colorization but can hardly produce plausible GAN training stage. is allows us to robustly train a network with results based on arbitrary man-made line arts. Concurrently, Zhang more capacity, thus makes the synthesized images more natural et al. [59] colorize man-made anime line arts with a reference im- and real. Moreover, we collected two cleaned datasets with high age. PaintsChainer [39] rst developed an online application that quality color illustrations and hand-drawn line arts. ey provide can generate pleasing colorization results for man-made anime line a stable training data source as well as a test benchmark for line arts with stroke colors as hints, they provide three models (named art colorization. tanpopo, satsuki, canna) with one of them open-sourced (tanpopo). By training with the proposed illustration dataset and adding However, these models