IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, OCTOBER 2019 1 Towards a complete 3D morphable model of the human head Stylianos Ploumpis, Evangelos Ververas, Eimear O’ Sullivan, Stylianos Moschoglou, Haoyang Wang, Nick Pears, William A. P. Smith, Baris Gecer, and Stefanos Zafeiriou Abstract—Three-dimensional Morphable Models (3DMMs) are powerful statistical tools for representing the 3D shapes and textures of an object class. Here we present the most complete 3DMM of the human head to date that includes face, cranium, ears, eyes, teeth and tongue. To achieve this, we propose two methods for combining existing 3DMMs of different overlapping head parts: i. use a regressor to complete missing parts of one model using the other, ii. use the Gaussian Process framework to blend covariance matrices from multiple models. Thus we build a new combined face-and-head shape model that blends the variability and facial detail of an existing face model (the LSFM) with the full head modelling capability of an existing head model (the LYHM). Then we construct and fuse a highly-detailed ear model to extend the variation of the ear shape. Eye and eye region models are incorporated into the head model, along with basic models of the teeth, tongue and inner mouth cavity. The new model achieves state-of-the-art performance. We use our model to reconstruct full head representations from single, unconstrained images allowing us to parameterize craniofacial shape and texture, along with the ear shape, eye gaze and eye color. Index Terms—3DMM, Morphable Model combination, 3D reconstruction, craniofacial 3DMM. F 1 INTRODUCTION UE to their ability to infer and represent 3D surfaces, than one morphable model [7], [8], try to solve this problem with a D 3D Morphable Models (3DMMs) have many applications part-based approach where multiple separate models are fitted and in computer vision, computer graphics, biometrics, and medical then linearly blended into the final result. Our framework aims imaging [1], [2], [3], [4]. Many registered raw 3D images (‘scans’) at avoiding any discontinuities that might appear from part-based are required for correctly training a 3DMM, which comes at a very approaches by fusing all models into one single morphable model. large cost of manual labour for collecting and annotating such More specifically, although there have been many models of images with meta data. Sometimes, only the resulting 3DMMs the human face both in terms of identity [9], [10], [11] and ex- become available to the research community, and not the raw pression [12], [13], very few deal with the complete head anatomy 3D images. This is particularly true of 3D images of the human [14]. Building a high-quality, large-scale statistical model that de- face/head, due to increasingly stringent data protection regula- scribes the anatomy of the full human head paves directions across tions. Furthermore, even if 3DMMs have overlapping parts, their numerous disciplines. First, it will assist craniofacial clinicians in resolution and ability to express detailed shape variation may be diagnosis, surgical planning, and assessment. Second, generating quite different, and we may wish to capture the best properties of proportionally correct head models based on the geometry of the multiple 3DMMs within a single model. However, it is currently face will aid computer graphics designers to create realistic avatar- extremely difficult to combine and enrich existing 3DMMs with like representations. Third, ergonomic design of headwear, eye- different attributes that describe distinct parts of an object without wear, breathing apparatus and so on benefits from accurate models such raw data. Therefore, we present a general approach that of craniofacial shape variation across the population. Finally, a can be employed to combine 3DMMs from different parts of head model will give opportunities that aim at reconstructing a arXiv:1911.08008v2 [cs.CV] 19 Feb 2020 an object class into a single 3DMM. Due to their widespread full head representation from data-deficient sources, such as 2D use in the computer vision community, we fuse 3DMMs of the images. human face and the full human head as our exemplar. We add Our key contributions are: (i) a methodology that aims to detailed models of the ears, eyes and eye regions to our head fuse shape-based 3DMMs, using the human face, head and ear as model, along with a basic model of the oral cavity, tongue and an exemplar. In particular, we propose both a regression method teeth. Thus we create a large-scale, full-head morphable model based on latent shape parameters, and a covariance combination that has a more complete representation of shape variation than approach, utilized in a Gaussian process framework, (ii) a com- any other published to date. The technique is readily extensible bined large-scale statistical model of the human head in terms of to incorporate detailed models of the human body [5], [6], and ethnicity, age and gender that is significantly more accurate than indeed is applicable to any object class well-described by 3DMMs. any other existing head morphable model - we make this publicly- Recent works that aim at predicting the 3D representation of more available 1 for the benefit of the research community, including versions with and without eyes and teeth, and (iii) an application experiment in which we utilize the combined 3DMM to perform • The authors are with the Department of Computing, Imperial College London, South Kensington Campus, London SW7 2AZ, UK. full head reconstruction from unconstrained single images. • N. Pears and W. Smith are with the Department of Computer Science, The remainder of the paper is structured as follows. In Section University of York. E-mails: see https://ibug.doc.ic.ac.uk/people, https://cs.york.ac.uk/cvpr/ 1. Project url: https://github.com/steliosploumpis/Universal Head 3DMM IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, OCTOBER 2019 2 Fig. 1. A collection of arbitrary complete head reconstructions from unconstrained single images. Our work aims to combine the most important attributes of the human head (i.e. face, cranium, ears, eyes), in order to synthesize novel and realistic 3D head models from data deficient sources. 2 we review relevant related work. In Section 3 we elaborate on scans, a robust automated procedure was carried out including 3D the methodology of the face and head model combination and in landmark localization and error pruning of badly registered scans. Sections 4, 5 we describe the modeling of ears and eyes, which This work was the first to introduce bespoke models in terms of results in our complete head representation. In Section 7, we age, gender and ethnicity, and is the most information-rich 3DMM describe our head texture completion pipeline and in Section 8, of face shapes in neutral expression produced to date. we outline a series of quantitative and qualitative experiments. Applications such as 3D model-based reconstruction [22], [23] Finally, conclusions are presented in Section 9. and expression estimation [24], [25] in 2D images have greatly encouraged the advancement of statistical face models. With 2 RELATED WORK the advent of deep neural networks, several recent approaches A very recent survey [15] identified more complete statistical aimed to extend the traditional 3DMM by replacing the linear modelling of the human head as an important open challenge in models with non-linear approaches [26], [27] or by estimating the the development of 3DMMs. Motivated by this goal, we begin by coefficients of a 3DMM from a single image [28]. The main scope surveying existing attempts to model the face, the full craniofacial of this work lies in combining 3DMMs and building a united region, eyes and ears. An earlier version of the work in this paper representation of the most significant parts of the human head was originally presented in [16]. Here, we have extended the (i.e. face, cranium, ears and eyes), rather than creating alternatives model by additionally integrating detailed ear and eye models and for non-linear 3D face reconstruction. a full head texture model as well as including further experimental evaluation. 2.2 Head models In terms of 3DMMs associated with the human body, the main 2.1 Face models focus of the research literature has been on the reconstruction of The first 3DMM was proposed by Blanz and Vetter [17]. They the human face, but not other parts of the human head. The reason were the first to to recognize the generative capabilities of a for this is mainly due to the lack of 3D image datasets that describe 3DMM and they proposed a technique to capture the variations of the other parts of the human head. In recent years, a few works 3D faces. Only 200 scans were used to build the model (100 male such as [29] have tried to tackle this task, in which a total of 3; 800 and 100 female) where dense correspondences were computed head scans was utilized from the US and European CEASAR body based on optical flow that depends on an energy function that scan database [30] to build a statistical model of the entire head. describes both the shape and texture. The Basel Face Model The aim of this work focuses mainly on the temporal registration (BFM) is the most widely-used and well-known 3DMM, which of 3D scans rather than on the topology of the head area. The was built by Paysan et al. [18] and utilizes a better registration data consists of full body scans and the resolution in which the method than the original Blanz-Vetter 3DMM. They use a known head topology was recorded in is insufficient to depict correctly template mesh in which all the vertices have known positions the shape of each individual human head.