AdversarialNAS: Adversarial Neural Architecture Search for GANs Chen Gao1,2,4, Yunpeng Chen4, Si Liu3,∗ Zhenxiong Tan5, Shuicheng Yan4 1Institute of Information Engineering, Chinese Academy of Sciences 2University of Chinese Academy of Sciences 3School of Computer Science and Engineering, Beihang University 4Yitu Technology 5Beijing Forestry University [email protected], [email protected], [email protected] a) Manual Abstract Search Search Generator Discriminator Space Space Design Design via Manual via Manual Neural Architecture Search (NAS) that aims to automate Update Evaluation Update the procedure of architecture design has achieved promis- (Inception Score, etc.) ing results in many computer vision fields. In this paper, b) AutoGAN we propose an AdversarialNAS method specially tailored Search Fixed Search Generator Space Discriminator Space for Generative Adversarial Networks (GANs) to search for Sample Design a superior generative model on the task of unconditional via RNN Controller via Manual Update Evaluation image generation. The AdversarialNAS is the first method via reward (Inception Score, etc.) that can search the architectures of generator and discrim- inator simultaneously in a differentiable manner. Dur- c) AdversarialNAS-GAN Search Search Generator Discriminator ing searching, the designed adversarial search algorithm Space Space Sample Sample does not need to comput any extra metric to evaluate the via Distribution via Distribution performance of the searched architecture, and the search Update Update Evaluation Evaluation via Gradient via Gradient paradigm considers the relevance between the two network (Adversarial Loss) (Adversarial Loss) architectures and improves their mutual balance. There- fore, AdversarialNAS is very efficient and only takes 1 GPU Figure 1. Comparisons of different ways of designing GAN archi- day to search for a superior generative model in the pro- tectures. a) The previous hand-crafted GAN architectures depend posed large search space (1038). Experiments demonstrate on the experience of human experts. b) AutoGAN [9] adopts IS or the effectiveness and superiority of our method. The discov- FID as reward to update the architecture controller via reinforce- ered generative model sets a new state-of-the-art FID score ment learning. c) The proposed AdversarialNAS searches archi- of 10.87 and highly competitive Inception Score of 8.74 on tecture in a differentiable way with an adversarial search mecha- CIFAR-10. Its transferability is also proven by setting new nism, which achieves better performance with higher efficiency. state-of-the-art FID score of 26.98 and Inception score of 9.63 on STL-10. Code is at: https://github.com/ literature has very few types and can be simply divided into chengaopro/AdversarialNAS. two styles: DCGANs-based [32] and ResNet-based [14]. On the other hand, the benefits of specially designing the 1. Introduction network architecture have been proven through lost of dis- criminative networks, such as ResNet [14], DenseNet [17], Image generation is a fundamental task in the field of MobileNet [34], ShuffleNet [46], EfficientNet [36] and HR- computer vision. Recently, GANs [10] have attracted much Net [35]. Therefore, the research about the backbone archi- attention due to their remarkable performance for generat- tecture of GANs needs more attention to further improve ing realistic images. Previous architectures of GANs are the performance of the generative model. designed by human experts with laborious trial-and-error Recently, Neural Architecture Search (NAS) has been testings (Fig. 1 a)) and the instability issue in GAN train- studied heatedly owing to its ability to automatically dis- ing extremely increases the difficulty of architecture design. cover the optimal network architecture, which significantly Therefore, the architecture of the generative model in GAN reduces human labor. However, on generation tasks, specif- ∗Corresponding author ically GANs-based generation, only AutoGAN [9] and 5680 AGAN [38] have explored the application of NAS. achieves state-of-art performance with much higher ef- To design a NAS algorithm specially tailored for GANs ficiency. We design a large architecture search space on the unconditional image generation task, there are two (1038) for GAN and make it feasible to search in. Our main challenges. First, it is expected to utilize an efficient AdversarialNAS can only tasks 1 GPU day for search- supervision signal to guide the search process in this un- ing an optimal architecture in the large search space. supervised task. However, the existing works [9, 38] both adopt the Inception Score (IS) [33] or FID to evaluate the • Considering GAN is an unique competition between architecture performance and take IS or FID as a reward to two networks, the proposed AdversarialNAS alter- update the architecture controller via reinforcement learn- nately searches the architecture of both of them under ing strategy. Obtaining IS or FID needs to generate hun- an adversarial searching strategy to improve their mu- dreds of images and use the statistics produced by an In- tual balance, which is specifically tailored for GAN. ception network to calculate the final score. Thus it is ex- • The searched architecture has more advanced transfer- tremely time-consuming, e.g. 200 GPUs over 6 days [38]. ability and scalability while achieving state-of-the-art Second, the relevance and balance between generator and performance on both CIFAR-10 and STL-10 datasets. discriminator need to be considered during searching since the training process of GANs is a unique competition. How- 2. Related Work ever, AutoGAN search for a generator with a pre-defined growing discriminator (Fig. 1 b)), where the architecture of 2.1. Generative Adversarial Networks the discriminator can be regarded as fixed and may limit the Although Restricted Boltzmann Machines [15] and flow- algorithm to search for an optimal architecture of generator. based generative models [6] are all capable of generating In this work, we propose an Adversarial Neural natural images, GANs [10] are still the most widely used Architecture Search (AdversarialNAS) method to address methods in recent years due to their impressive generation the above challenges (Fig. 1 c)). First, we design a large 38 ability. GANs based approaches have achieved advanced search space (10 ) for fragile GAN and relax the search results in various generation tasks, such as image-to-image space to be continuous. Thus the architecture can be translation [18, 5, 19, 48], text-to-image translation [43, 45] represented by a set of continuous variables obeying cer- and image inpainting [29]. However, the potential of GANs tain probability distribution and searched in a differentiable has not been fully explored since there is rare work [32] manner. Second, we propose to directly utilize the exist- studying the impact of architecture design on the perfor- ing discriminator to evaluate the architecture of generator mance of GANs. In this work, we aim to search for a power- in each search iteration. Specifically, when searching for ful and effective network structure specifically for the gen- the generator architecture, the discriminator provides the erative model via an automatic manner. supervision signal to guide the search direction, which is technically utilized to update the architecture distribution of 2.2. Neural Architecture Search generator through gradient descent. Therefore, our method Automatic Machine Learning (AutoML) has attracted is much more efficient since the extra computation cost for lots of attention recently, and Neural Architecture Search calculating evaluation metric is eliminated. Third, in order (NAS) is one of the most important direction. The goal of to consider the relevance and balance between the generator NAS is to automatically search for an effective architecture and discriminator, we propose to dynamically change the that satisfies certain demands. The NAS technique has ap- architecture of discriminator simultaneously during search- plied to many computer vision tasks such as image classifi- ing. Accordingly, we adopt the generator to evaluate the ar- cation [2, 25, 26, 31, 49], dense image prediction [24, 47, 3] chitecture of discriminator and comput the loss to update the and object detection [8, 30]. discriminator architecture through ascending the stochas- Early works of NAS adopt heuristic methods such as re- tic gradient. The two architectures play against each other inforcement learning [49] and evolutionary algorithm [41]. in a competition to continually improve their performance, Obtaining an architecture with remarkable performance us- which is essentially an adversarial searching mechanism. ing such methods requires huge computational resources, Therefore, the AdversarialNAS gets rid of calculating extra e.g., 2000 GPUs days [41]. Therefore, lots of works de- evaluation metric and solves the unsupervised task through sign various strategies to reduce the expensive costs includ- an adversarial mechanism. It adequately considers the mu- ing weight sharing [31], performance prediction [1], pro- tual balance between the two architectures, which benefits gressive manner [25] and one-shot mechanism [26, 42]. for searching a superior generative model. The DARTS [26] in one-shot literature is the first approach To sum up, our main contributions are three-fold. that relaxes the search space to be continuous and conducts • We propose a novel AdversarialNAS method, which is searching in a differentiable