J Math Imaging Vis DOI 10.1007/s10851-017-0728-2 Principal Curvature Measures Estimation and Application to 3D Face Recognition Yinhang Tang1 · Huibin Li2 · Xiang Sun3 · Jean-Marie Morvan3,4 · Liming Chen1 Received: 4 August 2016 / Accepted: 27 March 2017 © Springer Science+Business Media New York 2017 Abstract This paper presents an effective 3D face keypoint ple curvature measures are employed to assign the canonical detection, description and matching framework based on directions. Similarity comparison between faces is accom- three principle curvature measures. These measures give a plished by matching all these curvature-based local shape unified definition of principle curvatures for both smooth descriptors using the sparse representation-based reconstruc- and discrete surfaces. They can be reasonably computed tion method. The proposed method was evaluated on three based on the normal cycle theory and the geometric mea- public databases, i.e. FRGC v2.0, Bosphorus, and Gavab. sure theory. The strong theoretical basis of these measures Experimental results demonstrated that the three princi- provides us a solid discrete estimation method on real 3D ple curvature measures contain strong complementarity for face scans represented as triangle meshes. Based on these 3D facial shape description, and their fusion can largely estimated measures, the proposed method can automatically improve the recognition performance. Our approach achieves detect a set of sparse and discriminating 3D facial feature rank-one recognition rates of 99.6, 95.7, and 97.9% on the points. The local facial shape around each 3D feature point is neutral subset, expression subset, and the whole FRGC v2.0 comprehensively described by histograms of these principal databases, respectively. This indicates that our method is curvature measures. To guarantee the pose invariance of these robust to moderate facial expression variations. Moreover, descriptors, three principle curvature vectors of these princi- it also achieves very competitive performance on the pose subset (over 98.6% except Yaw 90°) and the occlusion sub- B Huibin Li set (98.4%) of the Bosphorus database. Even in the case of [email protected] extreme pose variations like profiles, it also significantly out- Yinhang Tang performs the state-of-the-art approaches with a recognition [email protected] rate of 57.1%. The experiments carried out on the Gavab Xiang Sun databases further demonstrate the robustness of our method [email protected] to varies head pose variations. Jean-Marie Morvan [email protected] Keywords Principal curvature measures · 3D keypoint Liming Chen detection, description and matching · Expression, pose and [email protected] occlusion · Mesh-based 3D face recognition 1 Université de Lyon, CNRS, Ecole Centrale de Lyon, LIRIS, 69134 Lyon, France 1 Introduction 2 School of Mathematics and Statistics, Xi’an Jiaotong University, No.28, Xianning West Road, Xi’an, Shaanxi 710049, P.R. China Recently, with the rapid development of 3D imaging and 3 King Abdullah University of Science and Technologh, V.C.C. visualization techniques, 3D shape analysis and process- Research Center, Thuwal 23955-6900, Saudi Arabia ing (e.g. surface registration, 3D shape retrieval) have been 4 Université de Lyon, CNRS, Université Claude Bernard Lyon widely studied. As stated in [57], capturing 3D shape makes 1, ICJ UMR 5208, 69622 Villeurbanne, France an extremely wide array of new kinds of application possible. 123 J Math Imaging Vis For example, virtual reality (VR), augmented reality (AR), vatures (i.e. principal curvature measures) can be used for 3D biometrics, 3D human–machine interaction, 3D medi- 3D face recognition? How to estimate them on meshed 3D cal imaging, 3D remote sensing, to name just a few. Behind face scans, and are they effective and efficient for 3D face these applications, a fundamental and challenging problem description and recognition? To answer these questions, we is how to represent and characterize the geometric structures give the following contributions in this paper: of these discrete 3D shapes (surfaces or volumes). Up to now, many approaches [9,12,60,65] have been proposed to solve 1. Different from all the existing 3D face recognition this key issue. approaches, we are the first attempting to study the As a special application of 3D shape analysis and pro- effectiveness of the principal curvature measures for the cessing, the key issue of 3D face recognition has also practical 3D face recognition application. been widely addressed. Facial surface measurements, i.e. 2. We improved our previous work [78] and designed novel points [10], curves [63], stripes [5], regions [13,14], normals 3D facial keypoint detection and description schemes [39,77], curvatures [40,78], geodesic distance [11,42,43], based on principal curvature measures, and their corre- have been popularly used to represent and characterize 3D sponding principal curvature vectors. face shapes. Among them, surface curvatures, including 3. We conducted comprehensive experiments on three 3D Gaussian curvature, mean curvature, principal curvatures, as face databases, i.e., FRGC v2.0, Bosphorus and Gavab, well as shape index values, are the most widely used ones to evaluate the general face recognition performance and [16,20,28,39,40,42,43,67,76,78,84]. Thus, the representa- the robustness to pose and occlusion variations of the tion and characterization of 3D face surface are the most proposed approach. important techniques for an efficient 3D face recognition sys- tem, where curvatures play fundamental effect. Specifically, the proposed 3D face recognition approach However, 3D face scans are discrete surfaces and gener- consists of three modules. First of all, a group of discrim- ally represented by discontinuous point clouds, depth images inating feature points are detected automatically based on or piece-wise triangle meshes. This means that the classical principal curvature measures. Then, the histograms of three curvature estimation formulations for at least C2 smooth sur- principal curvature measures describe comprehensively the faces in the classical differential geometry are not suitable for local shape around these feature points. The descriptor is 3D face scans. Instead of working directly on face scans, a further endowed with rotation invariance by assigning canon- popular solution is to estimate the curvatures of a polynomial ical direction based on the corresponding principal curvature surface patch achieved by fitting it to the local 3D facial sur- vectors. The similarity measurement between face scans is face. Otherwise, pointwise curvatures can also be estimated finally defined as the sparse representation-based reconstruc- directly on triangle meshes. For example, the angle defect tion error. estimation [7] and the normal deviation estimation [85]are The proposed method was comprehensively evaluated on utilized to compute pointwise Gaussian curvature and mean three public 3D face databases, i.e. FRGC v2.0 [59], Bospho- curvature, respectively. However, the pointwise convergence rus [64], and Gavab [53]. On the FRGC v2.0 databases, guarantee is satisfied only for triangle meshes with special the effectiveness of each single curvature-based descriptor is structure [86]. validated, and the discriminative power of their score-level Fortunately, based on the normal cycle theory [17,58] and fusion demonstrates the strong informative complementarity geometry measure theory [18,54], Morvan et al. proposed among different principal curvature measures. Our approach the concepts of generalized curvatures, i.e. curvature mea- achieves rank-one recognition rates of 99.6% with neutral sures, for both smooth and discrete surfaces [17,54]. They probes and 95.7% with expressive probes, and a verification further proposed a surface meshing algorithm and proved that rate of 95.28% with 0.1% false accept rate (FAR). More- Gaussian curvature measure and mean curvature measure of over, when the experiments performed on the pose subsets a triangle mesh converge to the ones of its sampled C2 smooth of the Bosphorus and the Gavab databases, the proposed surface, if the mesh is well-shaped (i.e. with minimal angle approach achieves the best scores among the state-of-the- bound) and well-sized (i.e. with infinitesimal circumcircle) art ones showing its robustness to diverse pose changes. The [41]. Recently, they further generalized the second funda- outstanding recognition results of the descriptors are also mental form and asymptotic direction of smooth surface, and presented when different external occlusions are involved. proposed asymptotic cone and principal curvature measures The rest of this paper is organized as follows. Section 2 for general polyhedral surface [73]. The convergence of the gives a brief review of related works on 3D face recogni- principal curvature measures can also be guaranteed based tion and curvature estimation on discrete surfaces. Section 3 on the normal cycle theory [73] (see Sect. 3 for more details). introduces the theory of principal curvature measures of Motivated by this curvature estimation method for triangle smooth surface, triangle mesh, as well as the meshed 3D meshes, a natural idea is to see how these generalized cur- facial surface. 3D face recognition method based on prin- 123 J Math Imaging Vis cipal curvature measures and their corresponding principal [10,23,63], and expression deformation modelling based curvature vectors is introduced