Image and Vision Computing 58 (2017) 214–223 Contents lists available at ScienceDirect Image and Vision Computing journal homepage: www.elsevier.com/locate/imavis Semi self-training beard/moustache detection and ଝ ଝଝ segmentation simultaneously , T. Hoang Ngan Le*, Khoa Luu, Chenchen Zhu, Marios Savvides CyLab Biometrics Center and Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA, USA ARTICLE INFO ABSTRACT Article history: This paper presents a robust, fully automatic and semi self-training system to detect and segment facial Received 22 September 2015 beard/moustache simultaneously in challenging facial images. Based on the observation that some certain Received in revised form 15 July 2016 facial areas, e.g. cheeks, do not typically contain any facial hair whereas the others, e.g. brows, often con- Accepted 25 July 2016 tain facial hair, a self-trained model is first built using a testing image itself. To overcome the limitation of Available online 21 September 2016 that facial hairs in brows regions and beard/moustache regions are different in length, density, color, etc., a pre-trained model is also constructed using training data. The pre-trained model is only pursued when Keywords: the self-trained model produces low confident classification results. In the proposed system, we employ Beard/moustache the superpixel together a combination of two classifiers, i.e. Random Ferns (rFerns) and Support Vector Detection and segmentation Machines (SVM) to obtain good classification performance as well as improve time efficiency. A feature Super pixel Random Ferns vector, consisting of Histogram of Gabor (HoG) and Histogram of Oriented Gradient of Gabor (HOGG) at dif- Support Vector Machine ferent directions and frequencies, is generated from both the bounding box of the superpixel and the super Histogram of Gabor (HoG) pixel foreground. The segmentation result is then refined by our proposed aggregately searching strategy Histogram of Oriented Gradient of Gabor in order to deal with inaccurate landmarking points. Experimental results have demonstrated the robust- (HOGG) ness and effectiveness of the proposed system. It is evaluated in images drawn from three entire databases i.e. the Multiple Biometric Grand Challenge (MBGC) still face database, the NIST color Facial Recognition Technology FERET database and a large subset from Pinellas County database. © 2016 Elsevier B.V. All rights reserved. 1. Introduction recognition systems due to the following biometric observation. There is lack of small patches under a human mouth and these fea- Facial hair analysis has recently received significant attention tures do not change during the lifetime of a person [2] (illustrated from forensic and biometric researchers due to three important in Fig. 3). Accurate beard and moustache detection and segmenta- observations as follows. Firstly, changing facial hairstyle can modify tion enable us to perform occlusion removal, gender and ethnicity a person’s appearance such that it effects facial recognition systems classification, age estimation, facial recognition, etc. as illustrated in Fig. 1. In this figure, the most left image is the gallery Nguyen et al. [3] presented a method for facial beard synthesis image whereas the right images are from the probe set. The sim- and editing. However, this method only works on high resolu- ilarities between the galley image and the probe set at pixel level tion images with the assumption that facial hair already existed. are given under each probe image. Secondly, most females do not Also, there was the requirement of initial seeds for Graphcuts-based have beard or moustache. Therefore, detecting facial hair helps to method. Their proposed method was lack of experimentally quanti- distinguish male against female with high confidence in the gender tative results. To overcome those weaknesses, Pierrard [4] presented classification problem. Finally, opposed to babies and young adults, facial hair detection using GrabCut and alpha matching. Although only male senior adults generally have beard or moustache. The facial their method was evaluated in the application of face recognition hair detection can help to improve the accuracy of an age estima- on FERET database [5], there was a lack of quantitative results in tion system as shown in Fig. 2. In addition to beard and moustache their work. Later, Le et al. [2,6] introduced the SparCLeS method to detection, segmentation also plays a crucial role, especially in facial automatically detect and segment beard/moustache with full mea- surement. Their methods are robust against illumination and obtain high accuracy of detection. However, it is time consuming and ଝ A preliminary version of this paper was presented in the International Conference highly complex because of the Multiscale Self-Quotient algorithm on Biometrics (ICB), May 19–22, 2015 in Phuket, Thailand [1]. and the dictionary learning approach. Furthermore, their facial hair ଝଝ This paper has been recommended for acceptance by Michele Nappi. * Corresponding author. detection depends on training data built the binary decision dynamic http://dx.doi.org/10.1016/j.imavis.2016.07.009 0262-8856/© 2016 Elsevier B.V. All rights reserved. T. Hoang Ngan Le et al. / Image and Vision Computing 58 (2017) 214–223 215 score. Our proposed algorithm first extracts features consisting of Histogram of Gabor (HoG) and Histogram of Oriented Gradient of Gabor (HOGG) at different directions and frequencies [1] form both the bounding box of the superpixel as well as from the super- pixel foreground. Then, the system is trained using a combination of two productive classifiers, i.e. Random Ferns(rFernspng) and Sup- 100.0 73.0 69.9 56.44 port Vector Machines (SVM) on the top of HoG and HOGG features to assign a score for each superpixel. To avoid the problem of incorrect Fig. 1. An illustration of how changes to facial hair can transform the appearance of alignment from the landmarker and to handle the cases where peo- a person. The left image is a target image whereas the right images are in a probe set ple have long beard, we propose a greedy searching strategy to group and the matching scores at pixel level are given below each image in the probe set. all superpixels which are close in the feature space. Unlike earlier developed systems, treating detection and seg- mentation as two separated tasks, our proposed system is able to dictionary. Recently, Le et al. [1] has developed a system that enables simultaneously detect and segment facial hair. In contrast to previ- self-learn transformation vector to separate a hair class and a non- ous works that deal with pixel level and strongly rely on landmarker, hair class from the testing image itself. In their approach, a feature- our work makes use of the superpixels and propose an aggregately based segmentation was proposed to segment the beard/moustache search strategy to overcome the presented handicaps. The proposed from a region on the face that is discovered to contain facial hair. algorithm includes the four main steps as follows, Their system has been evaluated on the large number of images from the MBGC, FERET and Pinellas databases and obtained highly accurate results of detection and segmentation with small time con- • Superpixel Generation: We start with grouping pixels into sumption. To overcome the handicaps of time consuming of Le et al. perceptually meaningful region called superpixel. Because we [2,6],Leetal.[1] presented a self-learning transformation vector for want to accurately segment object (facial hair) as well as each testing image where the training samples are extracted from reduce computational complexity, we use simple linear itera- testing image itself. Le et al. [1] is more computationally efficient and tive clustering (SLIC) [7] to generate superpixel. based on three observations: (1) certain areas, such as cheeks, do • Feature Extraction: We extract feature, which is a combination not typically contain any facial hair, (2) certain areas, such as brows, of HoG and HOGG feature [1], from both the bounding box of often contain facial hair, and (3) unlike a skin region, a facial hair the superpixel as well as from the superpixel foreground. region contains high frequency information in different orientations, • Superpixel Classification: We train the proposed system using which depends on whether the hair is straight, wavy or curly. The a classifier, which is a combination of rFerns and SVM on top experiment has shown that the later is faster but less accurate when of the high frequency features to assign a score for each super- facial hair presented in the beard/moustache region is quite different pixel. There are two models applied in the proposed system, in length, color, density from facial hair in the brows. namely, self-trained model and pre-trained model. We first use In this paper, we take advantages of both self-trained model and self-trained model to decide the label of superpixel. Based on pre-trained model. Given a testing image, the pre-trained model is the confidence of superpixel labeling, the pre-trained model called if the self-trained model gives low confidence on classification maybe used if the score is low. Fig. 2. An illustration of how beard/moustache works in the age estimation problem. Fig. 3. An illustration of patches under the mouth where facial hair is absent. 216 T. Hoang Ngan Le et al. / Image and Vision Computing 58 (2017) 214–223 • Propose an aggregately searching strategy to deal with the problem of inaccurate alignment come from the landmarker. The rest of this paper is organized as follows. Section 2 introduces the background of the main techniques employed in our method, i.e. the MASM landmarking algorithm, the two classifiers Random Ferns (rFerns) and SVM and Simple linear iterative clustering (SLIC) algorithm. Section 3 describes our proposed algorithm that consists of two key parts, i.e.