Learning to Generate and Edit Hairstyles

Weidong Yin1, Yanwei Fu1,†, Yiqiang Ma1 Yu-Gang Jiang2, Tao Xiang3, Xiangyang Xue1,2 1School of Data Science, Fudan University; 2School of Computer Science, Fudan University; 3 Queen Mary University of London , † Corresponding author. Email: [email protected]

ABSTRACT One of the reasons is that there are large variations in hairstyles Modeling hairstyles for classication, synthesis and image edit- and in order to model these variations, large-scale datasets are ing has many practical applications. However, existing hairstyle needed. Unfortunately, such a large-scale hairstyle dataset does not datasets, such as the Beauty e-Expert dataset, are too small for exist. In Multimedia and computer vision communities, hairstyles developing and evaluating computer vision models, especially the are often labeled as attributes for face datasets. However, such recent deep generative models such as generative adversarial net- annotation is often crude, focusing mostly hair length and color. work (GAN). In this paper, we contribute a new large-scale hairstyle On the other hand, existing specialized hairstyle datasets such as dataset called Hairstyle30k, which is composed of 30k images con- Beauty e-Expert dataset [18] are too small to represent the diversity taining 64 dierent types of hairstyles. To enable automated gener- of human hairstyles in the wild. ating and modifying hairstyles in images, we also propose a novel In this paper, we introduce the rst large-scale hairstyle dataset GAN model termed Hairstyle GAN (H-GAN) which can be learned – Hairstyle30K to the community and hope that this will greatly eciently. Extensive experiments on the new dataset as well as boost the research into hairstyle modeling. Images in the dataset existing benchmark datasets demonstrate the eectiveness of pro- (see Fig. 1 for examples) are collected from the Web via search posed H-GAN model. engines using keywords corresponding a hairstyle ontology. This results in 64 dierent types of hairstyles in 30K images. On average, KEYWORDS each hairstyle class has around 480 images. The newly proposed dataset is used to train the H-GAN model proposed in this paper. Hairstyle dataset, Hairstyle Classication, Generative Adversarial Importantly, with 64 hairstyle classes, this is a ne-grained dataset Networks presenting a challenging recognition task, as veried by our exper- ACM Reference format: iments. 1 1,† 1 2 3 Weidong Yin , Yanwei Fu , Yiqiang Ma and Yu-Gang Jiang , Tao Xiang , Apart from releasing a new dataset, we also present a Hairstyle 1,2 Xiangyang Xue . 1997. Learning to Generate and Edit Hairstyles. In Pro- Generative Adversarial Network (H-GAN) model for automati- ceedings of ACM Conference, Washington, DC, USA, July 2017 (Conference’17), cally generating or modifying/editing hairstyles given an input 9 pages. image. Our H-GAN has three components: an encoder-decoding DOI: 10.475/123_4 sub-network, a GAN and a recognition subnetwork. Particularly, the encoder-decoding network is a variant of Variational Auto- 1 INTRODUCTION Encoders (VAE) [12]; the recognition sub-network shares the same Hairstyle can express one’s personalities, self-condence, and at- networks as the discriminator of GAN as in InfoGAN [5]. The titudes. It is thus an important aspect of personal appearance. A model is unique in that once trained, it can be used to perform computer vision model that enables recognition, synthesis, and various tasks including recognition, synthesis and modication. modication of hairstyles in images is of great practical use. For ex- Extensive experiments of our H-GAN algorithm on the proposed ample, with such as model, customer can take a photo of him/herself dataset and other general-purpose benchmark datasets validate the and then synthesize dierent hairstyles before going to the hair- ecacy of our model. dresser’s to make the most satisfactory one a reality. In addition, an Contributions. We make several contributions in this paper. Firstly, automated hairstyle recognition model can be used for recognizing to study the hairstyle related problems, we contribute a new large- person’s identity for security applications. scale hairstyle dataset – Hairstyle30k to the community. To the best Existing eorts on hairstyle modeling have been focused on rec- of our knowledge, this is the largest hairstyle dataset, especially in ommending the most suitable hairstyles [18], or interactively users’ terms of the number of hairstyle classes. Secondly, we present a editing [7, 22, 32]. However, there is no attempt so far to systemat- new deep generative model called – H-GAN which can eectively ically study hairstyles in images and no model available that can and eciently generate and modify the hairstyles of person images. address various hairstyle modeling task in a comprehensive manner. Extensive experiments demonstrate that our H-GAN is superior to a number of state-of-the-art alternative models. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation 2 RELATED WORK on the rst page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). 2.1 Image Editing and Synthesis Conference’17, Washington, DC, USA © 2016 Copyright held by the owner/author(s). 123-4567-24-567/08/06...$15.00 Editing image with interaction. Recent advances in interactive DOI: 10.475/123_4 image segmentation have signicantly simplied the tasks of object Conference’17, July 2017, Washington, DC, USA Yin et al.

Figure 1: Examples of our Hairstyle30K dataset with corresponding hairstyle class labels. segmentations [3, 4, 16, 25, 38]. Existing interactive segmentation information between attributes and generated images to generate approaches such as lazy snapping [16] and grab cut [25] as well as images with specied attributes. recent Generative Adversarial Networks (GAN) related methods [44] enable the users to achieve good quality object cutout with 2.2 Attribute Analysis a few of strokes. In comparison, the existing eorts on hairstyles Attribute-based people search. Hairstyles can be considered as and makeup editing are very primitive [7, 22, 32]. In theory, these a special type of person attributes. Such attributes can be used in interactive image editing works can be used for editing hairstyles. the applications of surveillance environments, including but not However, it can be tedious and time-consuming to manually modify limited in attribute-based people search [29, 34, 36] and person hairstyles via user-interaction. Fully automated image editing thus re-identication [15, 21, 31, 42]. Recent studies on person attributes becomes desirable. are focused on clothing. These include clothes recognition [37], Automated image editing. There are some recent eorts on fully clothing parsing [17] as well as clothing retrieval [2, 41]. The study automated images editing [10, 24, 28, 43]. In particular, two recent presented in this paper complements the existing clothing oriented studies [24, 28] propose approaches to modify the attribute of facial attribute analysis. images. The proposed H-GAN is an automated image editing model Face attribute analysis. It is another important research topic but focuses on hairstyle images. related to our hairstyle analysis. Facial attribute analysis was rst Image Editing and Synthesis. Our work is also related to previ- studied by Kumar et al. [13]. In terms of dierent visual features ous work on joint image editing and synthesis [10, 14, 24, 28, 39]. and distinctive learning paradigm, facial attribute analysis has been Shen et al. [28] manipulated the facial attributes by a GAN-based developed into three categories: (1) the methods using hand-crafted image transformation networks; nevertheless each trained model visual features [13], such as SIFT [20] and LBP [23]; (2) the meth- in [28] can only modify one special type of facial attribute im- ods utilizing the recent deep features [19, 36]; and (3) multi-task ages. In contrast, one trained model of our H-GAN can modify all methods for learning facial attribute [1, 6, 26]. hairstyles presented in the training data. In model proposed by [39] , conditional variational auto-encoder is used to generate facial 3 DATASET COLLECTION images of dierent attributes; due to the lack of the adversarial loss, Our hairstyle30k dataset is designed for studying the problem of the generative images are often blurry. In VAEGAN[14], VAE is hairstyle classication as well as other hairstyle related tasks includ- combined with GAN to generate more realistic image. However, ing synthesis and editing. To construct the dataset, we had down- compared to our H−GAN, it does not use attribute information and loaded more than 1 million images using various web search engines modication is achieved by calculating ¨residual attribute vector¨. (Google, Flicker, and Bing, etc) with hairstyle related search words, The main dierence between VAEGAN and H_GAN is thus the e.g. Beehive hairstyle. The full set of class names of hairstyles are fact that we add attribute information explicitly into the generator listed in Fig. 1. The initial downloaded images were rstly ltered so that we can specify dierent attributes that we want to change. by face detection algorithm. We subsequently pruned some irrele- Also we introduce a recognition network to maximize the mutual vant or erroneous images which has neither faces nor hairstyles. We then manually ltered out the irrelevant images for some hairstyles Learning to Generate and Edit Hairstyles Conference’17, July 2017, Washington, DC, USA that come without faces, e.g., Ducktail. We carefully annotated the the parameters of G and D with the loss functions LD = LGAN pruned images and classied them into dierent hairstyle classes. and LG = −LGAN for generator and discriminator respectively. Finally, we obtained 30k images with 41 types of male hairstyles and The generator can draw a sample z ∼ pprior (z) = N (0, 1) and 42 types of female hairstyles. Among them, 19 kinds of hairstyle utilized the generator network G, i.e., G(z) to generate an image. have both male and female versions. Thus totally, the dataset has InfoGAN [5] further models the noise variable z in Eq (1) by de- 64 dierent types of hairstyles. composing it into a latent representationy and incompressible noise z. To ensure no loss information of latent representation in the gen- 3.1 Statistics eration, InfoGAN maximizes the mutual information I (y;G (z,y)) The number of images of each hairstyle class are varied in term as the recognition loss, of how popular this hairstyle is. In general, similar to most object h i Lrg = −Ex∼G(z,y) Ey∼p (y |x) [logQ(y | x)] (2) classication dataset [33], we also observe a long-tailed distribution dat a of the number of hairstyle instances over classes as illustrated in where Q (y | x) is an approximation of the posterior pdata (y | x). Fig. 2. On average, each hairstyle has around 480 images. InfoGAN can unsupervisedly learn disentangled, interpretable and meaningful representations with the loss function of the generator L L − L 3.2 Uniqueness G as GI nf oGAN = G rg Existing publicly available datasets for academic research either VAEGAN [14] integrates the Variational Auto-Encoders (VAE) into have too few image (e.g., the Beauty e-Experts dataset); or too GAN. It uses the feature-wise errors to replace the element-wise few hairstyle classes (e.g., the CelebA dataset). Specically, The errors of original GAN in the data space. The VAE part encodes the Beauty e-Expert dataset [18] has only 1505 female gures in distinct data sample x to latent representation z: z ∼ Enc (x) = penc (z | x) fashions; in contrast, our hairstyle dataset has around 30K instances. and decodes the z back to data space: x˜ ∼ Dec (x | z) = pdec (x | z) The general-purpose face dataset CelebA [19] has around 200K by two loss functions: (1) the regularization of the latent space
celebrity gures with 40 annotated attributes. Nevertheless, in Lprior = KL qenc (z | x) k pprior (z) , where qenc (z | x) is the CelebA, only very few and very simple hairstyles are labeled as approximation to the true posterior pdec (z | x); (2) the reconstruc- the attributes, e.g., wavy hair. Generally, the targets of our dataset tion error is also dierent from CelebA, since ours is a benchmark dataset Dl Lrecon = −Eq (z |x) [logpdec (Dl (x) | z)] (3) for recognizing dierent hairstyles; and the images within each enc hairstyle class can cover large pose variations and background where Dl (x) is hidden representation of l − th layer of the dis- clutter. Importantly, The images of the same person with dierent criminator. Thus the loss functions of VAEGAN are updated as Dl hairstyles should be categorized into dierent hairstyle classes. LD = LGAN , Ldec = LG = −LGAN +λ ·Lrecon and the encoder Dl Lenc = Lprior + Lrecon, where λ is the coecient. However the 3.3 Applications latent representation z is unsupervised learned and not explicitly The datasets can be used to develop dierent applications. Specif- associated with any nameable attributes. ically, the task of recognizing dierent hairstyles belongs to the CVAE [30, 39] is short for the conditional VAE. CVAE introduces category of ne-grained classication, which is an active and yet an independent attribute y to control the generating process of x very challenging research topic in the multimedia community, e.g. by sampling from p (x | y, z); where p (y, z) = p (y)p (z). [11, 35]. Potentially, this dataset can serve as the benchmark dataset The encoder and decoder networks are thus z ∼ Enc (x) = qenc (z | x) for many real-world applications and tasks such as hairstyle re- and x˜ ∼ Dec (z,y) = pdec (x | z,y). The variable y is introduced trieval and recommendation systems [18], and the research of rec- to control the generate process of x by sampling from p (x | y, z); ognizing ne-grained hairstyles, and automatically generating and where p (y, z) = p (y)p (z). Nevertheless, y is still sampled from changing the hairstyles. In this next section, we propose a frame- data, but not directly optimized and learned from the data. work that enables three tasks, i.e. recognition, generation and modication of hairstyles, by using a single model. 4.2 Hairstyle GAN (H-GAN) Hairstyle GAN model is formulated for generating and modifying 4 HAIRSTYLE GAN (H-GAN) hairstyles in a single framework. Particularly, besides the input 4.1 Background noise z in GAN, we utilize the independent hairstyle variables y of ∼ ( | ) GAN [8] targets at learning to discriminate real data samples from each image x, i.e. y pdata y x . Mathematically, we have i.e., ( ) ( ) ( ) generated samples by training the generator network G to fool the p y, z = p y p z . discriminator network D. It is formulated to optimize the following As overviewed in Fig. 3, the network of H-GAN has three compo- objective functions, nents: an encoder-decoding sub-network, a GAN sub-network and a recognition sub-network. The network structure is explained in Tab. 1. The whole H-GAN network is trained together in an end-to-end minmaxLGAN = Ex∼p (x) [logD (x)] (1) fashion. Once trained, the encoder-decoding sub-network enables G D dat a generating and changing hairstyles; the recognition sub-network +E ∼ ( ) [log (1 − D (G (z)))] , z ppr ior z can be used for hairstyle classication. Note that the recognition where pdata (x) and pprior (z) are the distributions of real data and sub-network and discriminator of GAN share the same network Gaussian prior N (0, 1). The training process iteratively updates structures except the last softmax classication layer, since both Conference’17, July 2017, Washington, DC, USA Yin et al.

Figure 2: The statistics of the hair-style Dataset. In the X-axis, we list the hairstyles. The number of pictures of each hairstyle class are compared in Y-axis. We use three color attributes to annotate each hairstyle: red and green colors indicate the hairstyle for male only and female only respectively. The blue color means the hairstyle both for male and female.

Figure 3: Overview of our H-GAN.

sub-networks are assigned the classication task with the same where Dl (x) is the hidden representation of l − th layer of the input. The loss functions of the generator GH −GAN , discriminator discriminator. It measures the loss of reconstructing generated

DH −GAN , encoder and decoder are thus updated as, images by sampling the z. Lrgz is the recognition loss on z, which is dened as   Dl LGH −GAN = LG + λ1 Lrgz + Lrgy −G + λ2Lrecon (4) L = −E [log (Q (z | x))] (9)   rgz x∼pdat a (x),z∼penc (z |x) LD − = LD + Lrg −D + Lrgz (5) −E [ ( ( | ))] H GAN y x∼pdec (x |z,y),y∼pdat a (y |x),z∼penc (z |x) log Q z x Dl −E ∼ ( | ) ∼ ( ) ∼ ( ) [log (Q (z | x))] Lenc = LVAE = Lprior + Lrecon (6) x pdec x y,z ,y pdat a y ,z ppr ior z −E ∼ ( | , ), ∼ ( ), ∼ ( | ) [log (Q (z | x))] , Ldec = LGH −GAN (7) x pdec x z y y pdat a y z penc z x where the rst term measures the loss of predicting errors on real where λ1 and λ2 are the weight of corresponding term; The recon- struction loss of Eq (3) is updated as data; and rest three terms are the loss functions on generated data. Q (·) is an approximation of the corresponding posterior data distri- LDl −E [ ( ( ) | )] ( | ) recon = z∼penc (z |x),y∼pdat a (y |x) logpdec Dl x z,y (8) bution. penc zla x is the distribution of z given x parameterized Learning to Generate and Edit Hairstyles Conference’17, July 2017, Washington, DC, USA

× † × † Encoder (128 128) Decoder (128 128) Discrim/Recog pdata(x) and the hairstyle y ∼ pdata(y), z ∼ penc (z | x), to update A 64@C (6 × 6) | y | +256 FC 64@C(6 × 6) the decoder and discriminator by minimizing Ldec and LDSL−GAN 8064FC -> respectively. 128@C(4×4) 128@C(4 × 4) 4 × 4 × 7 × 72 FC (3) Learning image generation: We sample a batch of latent vectors 256@C(4×4) 288@DeC(3 × 3) 128@C(4 × 4) z ∼ pprior (z) and the hairstyle y ∼ pdata(y); to minimize the L L 256@C(4×4) 216@DeC(3 × 3) 256@C(4 × 4) decoder dec and discriminator with DSL−GAN iteratively. 256 × 2 FC 144@DeC(5 × 5) + | y | 1 FC/256 FC 4.3 Generation and modication of hairstyles c 72@DeC(5 × 5) 3@DeC(6 × 6) Our H-GAN can be used to perform both tasks. Table 1: The details of networks. Stride is 2 for all layers. C Generation of hairstyles. To generate a new hairstyle image, we ( ) and Dec indicate the convolutional and deconvolutional l- can sample z from pprior z and setting y to any desired hairstyle. 0 ∼ ( , ) ter individually. 64@C (4 × 4) means 64 convolutional lters The image can be generated as x G z y . Modication of hairstyles with size 4 × 4. FC means the fully connected layer. The Dis- . For ecient image editing, we pro- criminator and Recognition networks share the same four posed to utilize the residual dierences between the desired hairstyle layers (the rst four layers in the third column). The last and all hairstyles. The resultant hairstyle modication algorithm layer of Discriminator and Recognition networks are 1 and takes two steps. (1) Given an image x and the desired hairstyle y , we rst sample z ∼ p (z | x). (2) We employ the en- 256+ | y | neurons individually. Note that: (1) † and indi- desired enc coder to compute the corresponding z of all images. We compute cate the activation of ReLU and Leaky ReLU (ratio: 0A.2) for the the corresponding networks. (2) indicates the reparame- z¯y = Ez∼p (z |x),x∼p(x |y ) [z] (12) terization trick [12]; specically,c we take z ∼ N (µz , σz ); and desir ed enc desir ed 256 and 256 neurons to regress µ and σ respectively. On de- coding part, z = µ+ϵσ, ϵ ∼ N (0, 1). (3) | y | means the number E [ ] z¯ = z∼penc (z |x),x∼pdat a (x) z (13) of hairstyles. (4) -> denotes the reshape operation. where z¯ydesir ed is the mean vector of images with the desired hairstyle; and z¯ is the mean vector for all the images. We then

compute the 4 = z¯ydesir ed − z¯; and the modied image can be 0 generated by x ∼ pdec (x | z + 4,ydesired ). by the encoder network; pdata (y) is the data distribution of y on real data; pdata (y | x) is the data distribution of y given x on the 5 EXPERIMENTS real data; pdata (x) is the data distribution of x on the real data; pprior (z) is the prior distribution of z and we use the Gaussian dis- 5.1 Experimental setup tribution N (0, 1); pdec (x | z,y) is the distribution of x given z and Dataset. We conduct the experiments on three datasets. y, and the distribution is parameterized by the decoder network; (1) Hairstyle30K is the newly proposed dataset in this paper. This penc (z | x) is the distribution of z given x and the distribution dataset has totally 30911 images of 64 dierent ne-grained male parameterized by the encoder network [40]. and female hairstyles. On average, each hairstyle has 480 images. For the recognition loss ony, the loss functions for the discriminator (2) CelebA is a facial attribute dataset of approximately 200k images and generator are dened as, of 10k identities [19]. Each image is annotated with 5 landmarks L −E [ ( ( | ))] (two eyes, the nose tips, the mouth corners) and binary labels of rgy −D = x∼pdat a (x),y∼pdat a (y |x) log Q y x (10) 40 attributes. Among all these attributes, 8 attributes are related to hairstyle and thus can be used to evaluate our H-GAN algorithm. L = −E [log (Q (y | x))] rgy −G x∼pdec (x |y,z),z∼penc (z |x),y∼pdat a (y) (3) Male hairstyle dataset combines the male hairstyles from both −E [ ( ( | ))] x∼pdec (x |y,z),z∼penc (z |x),y∼pdat a (y |x) log Q y x Hairstyle and CelebA dataset. Particularly we divide the male styles −E [ ( ( | ))] x∼pdec (x |y,z),z∼ppr ior (z),y∼pdat a (y |x) log Q y x into 6 dierent male hairstyle classes, namely, bald, bang, curly, (11) long, undercut-short, and undercut-long. We re-annotate the images where for the discriminatorLrgy −D , only the real data is used to from Hairstyle and CelebA dataset into these 6 categories. Totally, train the model since the quality of generated data in the training this dataset has 38293 images, and each class on average contains process is unreliable. 6382 images. Compare to the original Hairstyle30K dataset, this Training algorithms. The training of H-GAN is optimized by dataset has much few classes but each class has much more samples many epochs; each epoch is divided into three stages, namely, and suers less the class imbalance problem. (1) Learning image reconstruction: we update the encoder-decoder Evaluation. We employ several dierent evaluation metrics to subnetwork and learn to reconstruct the image given the desired evaluate our proposed dataset and H-GAN algorithm. (1) Classi- hairstyle. Specically, we sample a batch of images x ∼ pdata(x) cation of hairstyles. We evaluate the tasks of the classication and hairstyle y ∼ pdata(y|x), z ∼ penc (z | x) to update the encoder, tasks of the Hairstyle30K dataset. Particularly, in each hairstyle and decoder, by minimizing Lenc and Ldec individually. class, the images are equally sliced into two halves. 50% images are (2) Learning image modication: Given an image x and desired used for training while the rest images are saved as the testing data. hairstyle y, this stage learns of modifying image x with the y The mean accuracy (i.e., the mean of the diagonal of the confusion hairstyle. In particular, we rstly sample a batch of images x ∼ matrix) is employed as the metrics for evaluation due to the data Conference’17, July 2017, Washington, DC, USA Yin et al.

Hairstyle Male Hairstyle

30 60

25 50

20 40

15 30

Mean Accuracy 10 Mean Accuracy 20

5 10

0 0 RBF-SVM Random Forest Logistic Regression Neural Network RBF-SVM Random Forest Logistic Regression Neural Network

Figure 4: The results of classication on two hairstyle datasets. The chance-level are 1.56% and 16.7% respectively for the hairstyle30k and male hairstyle datasets respec- Figure 5: Qualitative results on the hairstyle generation task tively. using the Male Hairstyle dataset.

Dataset H-GAN VAEGAN Attrb2img Male hairstyle 2.28 2.26 2.31 CelebA 2.29 2.08 2.32 the insucient training instances for some rare types of hairstyles Table 2: Inception Scores on CelebA and Male hairstyle as shown in Fig. 2. In contrast, the male hairstyle dataset has 6 datasets. The higher values, the better results. types of hairstyles, and each hairstyle averagely has more than 3000 images. Thus on this dataset, the DNN achieves the best results on Male hairstyle dataset and outperforms the SVM results by 2.5%. imbalance between classes. The experiments are repeated for 10 5.3 Hairstyle generation round to reduce the variance. (2) Hairstyle generation. we employ to Competitors. We compare various open-source methods on this evaluate the hairstyle generation, the inception scores [27] which task, including VAEGAN [14], and Attrb2img [39]. Attrb2img is an aims at measuring whether the varied images are generated and advanced version of CVAE. To make the results more comparable, whether the generated images contains meaningful objects. (3) all the methods are trained with the same experimental settings. Modifying hairstyles. A user-study is carried out to evaluate the We conduct the generation of hairstyles on both the CelebA and eects of changing hairstyles by dierent methods. male hairstyle datasets. Implementation details. Our model is implemented based on the The results of inception scores on CelebA and Male hairstyle dataset tensor ow platform and our model get converged in 4-5 hours on are compared and shown in Fig. 2. Both the inception scores of the Male hairstyle dataset on GeForce GTX 1080; and our model needs generated and reconstructed images are compared. Totally 3000 around 8GB GPU memory. We use the Adam optimizer to train the images are generated for each method. Salimans et al. [27] proposed generator and the discriminator is trained by the Rmsprop optimizer the inception score for evaluating image generation quality. Higher with a mini-batch size of 64. The learning rate is set as 1e − 4 and inception scores indicate better visual quality of samples generated. 2e − 4 for generator/encoder and discriminator respectively. The To make a fair comparison, we make all three methods to generate input size of images is 128 × 128. Our H-GAN is trained by 10 the same hairstyle. epochs, and each epoch has 5000 iterations. As we can see from Table 2, our H-GAN achieves a 2.28 inception score, outperforming the VAEGAN on both datasets. This validates 5.2 Classication of Hairstyles that images generated using our H-GAN have better visual quality To investigate the classication performance of some baselines, than those of VAEGAN. Table 2 also shows that the inception score we extract the deep features of all the images by Resnet-50 net of Attrb2img is marginally higher than ours. After a close inspection [9]. On the Hairstyle30k and Male hairstyle datasets, we compare of the qualitative results generated on CelebA (see Fig. 7) and Male several baselines: (1) SVM: a SVM with RBF kernel is trained for the dataset (see Fig. 5), we conclude that our H-GAN’s results are still classication. (2) Random Forest: 100 estimators are used and the better than those of Attrb2img in term of the resolution, and clarity minimum number of samples required to split an internal node is of generated hairs. Particularly, the images generated by Attrb2img set as 10. (3) Logistic Regression (LR): a LR classier is also learned has very blurred hairstyles but sharp human faces (which have for classication; (4) Deep neural network (DNN): we use two fully contributed to the high inception scores). connected layers with the 1024 and 512 neurons respectively. We Qualitative results. Some qualitative examples of the generated use 10-fold cross validation to estimate the key parameters of each images of VAEGAN [14], Attrb2img [39] and H-GAN are illustrated method. in Fig. 7 and Fig. 5 for CelebA and Male hairstyle respectively. The The results are compared in Fig. 4. The mean accuracy of each generation results of Attrb2img show again sharp human faces and method is reported. We observe that on Hairstyle30k, SVM can yet blurred hairstyles. The VAEGAN can generated hairstyles with beat the other three methods and achieves the accuracy of 29.1%. ne details. Nevertheless, the overall quality of generated images This reveals the challenging nature of our dataset. Note that the of VAEGAN is worse than our H-GAN. For example, the second second best result 26.8% is obtained by DNN method, partly due to image of the VAEGAN results contains a very distorted face. Learning to Generate and Edit Hairstyles Conference’17, July 2017, Washington, DC, USA

Figure 6: Modication of hairstyles on Male hairstyle dataset.

Figure 7: Qualitative results of the generation on CelebA dataset.

5.4 Modifying hairstyles Competitors. We compare various open source methods on edit- ing hairstyles, including Attrib2img [39], and VAEGAN[14]. Though Attrb2img is not designed for modifying the images, we can actu- ally employ the “attribute-conditioned image progression” strategy suggested in [39] to interpolate one type of hairstyle gradually changing the values along the dimension of hairstyle. The same experimental settings are used for all the experiments. Figure 8: Results of modifying hairstyles on the CelebA User-study experiments. Since the task of modifying hairstyles dataset. Each column indicates modifying the hairstyles for is essentially only modifying some parts of images, we notice that one method. the modied images often have almost similar visual quality and thus the inception score used in generation task is not suitable as the evaluation metrics any more in this task. Instead, a user study as suggested in [40] is employed to compare these methods. Partic- three methods are utilized to modify the hairstyle of images and we ularly, ten participants unaware of the project goals are invited for can obtain two images. Totally 100 images are randomly sampled the user study. Given each image and one target hairstyle, these from the CelebA and Male hairstyle dataset respectively, and we Conference’17, July 2017, Washington, DC, USA Yin et al.

Metric Saliency Quality Similarity Guess REFERENCES Attrb2img 3.40 4.40 4.50 40.0% [1] Abrar H Abdulnabi, Gang Wang, Jiwen Lu, and Kui Jia. Multi-task cnn model for attribute prediction. IEEE TMM, 2015. VAEGAN 4.15 4.23 4.12 65.0% [2] D. Anguelov, K.-C. Lee, S. B. Gokturk, and B. Sumengen. Contextual identity H-GAN 4.40 4.40 4.65 70.0% recognition in personal photo albums. In CVPR, 2007. [3] Xue Bai, Jue Wang, David Simons, and Guillermo Sapiro. Video snapcut: Robust Table 3: The user-study of modication of user-dened at- video object cutout using localized classiers. In ACM SIGGRAPH, 2009. tributes. The “Guess” results are reported as the accuracy of [4] Dhruv Batra, Adarsh Kowdle, Devi Parikh, Jeibo Luo, and Tsuhan Chen. icoseg: guessing. Interactive co-segmentation with intelligent scribble guidance. In IEEE Conference on Computer Vision and Pattern Recognition, 2010. [5] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximiz- ing generative adversarial nets. In ICML, 2016. [6] Ankur Datta, Rogerio Feris, and Daniel Vaquero. Hierarchical ranking of facial change the hairstyles of sampled images. The participants have attributes. In IEEE FG, 2011. [7] D.Guo and T.Sim. Digital face makeup by example. In CVPR, 2009. been required to compare the modied images with the original [8] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, DavidWarde- image and give their judgment on a ve-point scale system from Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial the less to the most likely by answering the following questions? nets. In NIPS, 2014. [9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning (1) Saliency: how salient is the hairstyle has been changed in the for image recognition. arXiv preprint arXiv:1512.03385, 2015. image? (2) Quality: How is the overall quality of image modied? [10] I. Kemelmacher-Shlizerman, S. Suwajanakorn, and S. M. Seitz. Illumination- (3) Identity: how much degree do you think the modied image aware age progression. In CVPR, 2014. [11] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. and original image is of the same person? (4) Guess: Additionally, Novel dataset for ne-grained image categorization. In First Workshop on Fine- a guessing game is introduced as the fourth metrics. Given a pair Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011. of modied and original images, the participants will be asked to [12] Diederik P. Kingma and MaxWelling. Auto-encoding variational bayes. In ICLR, guess which hairstyle has been modied from the four candidate 2014. choices, which are randomly sampled from the hairstyle names and [13] Neeraj Kumar, Alexander C. Berg, Peter N. Belhumeur, and Shree K. Nayar. Attribute and simile classiers for face verication. In ICCV, 2009. of course, the correct hairstyle name should be included. [14] Anders Boesen Lindbo Larsen, Soren Kaae Sonderby, Hugo Larochelle, and Quantitative results. We list the user-study results in Tab. 3. On OleWinther. Autoencoding beyond pixels using a learned similarity metric. In ICML, 2016. all metrics, our H-GAN beats the other compared methods. Thus [15] Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang. Deepreid: Deep lter pairing we can draw the conclusion that our H-GAN can more eectively neural network for person re-identication. In CVPR, 2014. [16] Yin Li, Jian Sun, Chi-Keung Tang, and Heung-Yeung Shum. Lazy snapping. In modify the hairstyles whilst keeping the person’s identity. Interest- ACM SIGGRAPH, 2004. ingly, even though Attrb2img has relative good visual quality, the [17] Xiaodan Liang, Liang Lin, Wei Yang, Ping Luo, Junshi Huang, and Shuicheng Yan. strategy of modifying hairstyle employed by Attrb2img is relative Clothes co-parsing via joint image segmentation and labelingwith application to clothing retrieval. IEEE TMM, 2016. less ecient and the scores “Guess” is signicantly lower than the [18] L. Liu, J. Xing, S. Liu, H. Xu, X. Zhou, and S. Yan. Wow! you are so beautiful other two methods. today! ACM TMCCA, 2014. [19] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face Qualitative results. Some visualization results are compared in attributes in the wild. In ICCV, pages 3730–3738, 2015. Fig. 8 and Fig. 6. Each row is corresponding to one type of hairstyle. [20] David G. Lowe. Distinctive image features from scale-invariant keypoints. Inter- national Journal of Computer Vision, 60, 2004. Each column indicates the results of one method. We highlight that [21] L. Ma, X. Yang, and D. Tao. Person re-identication over camera networks using in general, the results of our H-GAN are always better than or at multi-task distance metric learning. In IEEE TIP, 2014. least comparable to those of the other two methods. The results of [22] Yuki Nagai, Kuniko Ushiro, Yoshiro Matsunami, Tsuyoshi Hashimoto, Yuusuke Kojima, and Weniger. Hairstyle suggesting system, hairstyle suggesting method, Attrb2img still suer from the problem of very blurred hair. We also and computer program product. In US Patent US20050251463, 2005. notice that in Fig. 8, the hairstyle of Gray hair is highly correlated [23] Timo Ojala, Matti Pietikäinen, and Topi Mäenpää. Gray scale and rotation with the “age” and “eyeglasses” attribute in CelebA, since usually invariant texture classication with local binary patterns. IEEE TPAMI, 2002. [24] Guim Perarnau, Joost van deWeijer, Bogdan Raducanu, and Jose M. Álvarez. the senior person may wear glasses and have gray hair. Invertible conditional gans for image editing. In NIPS Workshop on Adversarial Training, 2016. [25] C. Rother, V. Kolmogorov, and A. Blake. Grabcut: Interactive foreground extrac- 6 CONCLUSION tion using iterated graph cuts. ACM Trans. on Graphics, 2004. [26] Ethan M. Rudd, Manuel Gunther, and Terrance E. Boult. Moon:a mixed objective In this paper, we aim to present a comprehensive study on various optimization network for the recognition of facial attributes. In ECCV, 2016. hairstyle-related problems including classication, generation and [27] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In nips, 2016. modication of hairstyles. To promote the study of this topical issue, [28] Wei Shen and Rujie Liu. Learning residual images for face attribute manipulation. we introduce a new large-scale hairstyle dataset – Hairstyle30k In arxiv, 2017. with extensive hairstyle annotation. To automatically generate [29] Behjat Siddiquie, Rogerio Feris, and Larry Davis. Image ranking and retrieval based on multi-attribute queries. In IEEE Conference on Computer Vision and and change the hairstyle, we also propose a new – H-GAN model. Pattern Recognition, 2011. Extensive experiments on several benchmark datasets had validated [30] Kihyuk Sohn, Xinchen Yan, and Honglak Lee. Learning structured output the eectiveness of the proposed H-GAN over the existing methods. representation using deep conditional generative models. In NIPS, 2016. [31] Xiao T, W.Ouyang, H. Li, and X. Wang. Learning deep feature representations with domain guided dropout for person re-identication. In CVPR, 2016. 7 ACKNOWLEDGMENT [32] Wai-Shun Tong, Chi-Keung Tang, Michael S. Brown, and Ying-Qing Xu. example- based cosmetic transfer. In FG, 2007. This work was supported in part by two NSFC projects (#61572138 [33] A. Torralba, R. Fergus, and W.T. Freeman. 80 million tiny images: A large data and#61572134), one project from STCSM (#16JC1420401) and Shang- set for nonparametric object and scene recognition. IEEE TPAMI, 2008. hai Sailing Program (#17YF1427500) Learning to Generate and Edit Hairstyles Conference’17, July 2017, Washington, DC, USA

[34] D.A. Vaquero, R.S. Feris, D. Tran, L. Brown, A. Hampapur, and M. Turk. Attribute- based people search in surveillance environments. In IEEE WACV, 2009. [35] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. [36] Jing Wang, Yu Cheng, and Rogerio Feris. Walk and learn: Facial attribute representation learning from egocentric video and contextual data. CVPR, 2016. [37] Xianwang Wang and Tong Zhang. Clothes search in consumer photos via color matching and attribute learning. In ACM International Conference on Multimedia, 2011. [38] Chih-Yu Yan, Ming-Chun Tien, and Ja-Ling Wu. Interactive background blurring. In ACM MM, 2009. [39] Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. attribute2image: conditional image generation from visual attributes. In ECCV, 2016. [40] Weidong Yin, Yanwei Fu, Leonid Sigal, and Xiangyang Xue. Semi-latent gan: Learning to generate and modify facial images from attributes. In arxiv, 2017. [41] Bo Zhao, Xiao Wu, Qiang Peng, and Shuicheng Yan. Clothing cosegmentation for shopping images with cluttered background. IEEE TMM, 2016. [42] Liang Zheng, Hengheng Zhang, Shaoyan Sun, Manmohan Chandraker, and Qi Tian. Person re-identication in the wild. arXiv preprint arXiv:1604.02531, 2016. [43] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A. Efros. View synthesis by appearance ow. In ECCV, 2016. [44] Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros. Generative visual manipulation on the natural image manifold. In ECCV, 2016.