Sparse Shape Modelling for 3D Face Analysis Stephen Clement Doctor of Philosophy University of York Computer Science September 2014 Abstract This thesis describes a new method for localising anthropometric landmark points on 3D face scans. The points are localised by fitting a sparse shape model to a set of candidate landmarks. The candidates are found using a feature detector that is designed using a data driven methodology, this approach also informs the choice of landmarks for the shape model. The fitting procedure is developed to be robust to missing landmark data and spurious candidates. The feature detector and landmark choice is determined by the performance of different local surface descriptions on the face. A number of criteria are defined for a good landmark point and good feature detector. These inform a framework for measuring the performance of various surface descriptions and the choice of parameter values in the surface description generation. Two types of surface description are tested: curvature and spin images. These descriptions, in many ways, represent many aspects of the two most common approaches to local surface description. Using the data driven design process for surface description and landmark choice, a feature detector is developed using spin images. As spin images are a rich surface description, we are able to perform detection and candidate landmark labelling in a single step. A feature detector is developed based on linear discriminant analysis (LDA). This is compared to a simpler detector used in the landmark and surface description selection process. A sparse shape model is constructed using ground truth landmark data. This sparse shape model contains only the landmark point locations and relative positional variation. To localise landmarks, this model is fitted to the candidate landmarks using a RANSAC style algorithm and a novel model fitting algorithm. The results of landmark localisation show that the shape model approach is beneficial over template alignment approaches. Even with heavily contaminated candidate data, we are able to achieve good localisation for most landmarks. iii iv Contents Abstract iii List of Figures ix List of Tables xi Acknowledgements xiii Declaration xv 1 Introduction 1 1.1 Motivation . .1 1.2 Aims . .4 1.3 Contributions . .5 1.4 Terminology Disambiguation . .6 1.4.1 Imaging . .6 1.4.2 Landmark and Points . .6 1.4.3 Modelling and Localisation . .7 1.5 Thesis Structure . .7 2 Review of Literature 11 2.1 Face Analysis . 11 2.2 Holistic methods . 13 2.2.1 Global Surface Descriptions . 14 2.2.2 Shape Modelling . 15 2.3 Feature-Based Methods . 21 2.3.1 Landmarks . 22 2.4 Local Surface Descriptions . 24 2.4.1 Scalar Surface Descriptions . 25 v vi CONTENTS 2.4.2 Description Signatures . 28 2.5 Feature Matching . 36 2.5.1 Description Matching . 37 2.5.2 Landmark Model Fitting . 38 2.6 Conclusion . 43 3 Problem Analysis 45 3.1 Aims . 46 3.1.1 Problem Definition . 47 3.1.2 Landmark Detection . 49 3.1.3 Sparse Landmark Modelling . 49 3.1.4 Proposed System . 50 3.2 Datasets . 50 3.2.1 FRGC Dataset . 51 3.2.2 Basel Face Model Dataset . 52 3.3 Evaluation Criteria . 53 3.3.1 Limitations . 54 3.4 Conclusion . 55 4 Selecting Landmarks for use in a Sparse Shape Model 57 4.1 Aims . 58 4.2 Landmarks . 59 4.2.1 Types of Landmark . 59 4.2.2 Landmark Surface Descriptions . 60 4.2.3 Selecting Landmarks . 62 4.3 Descriptors . 63 4.3.1 Curvature . 64 4.3.2 Spin Images . 73 4.4 Conclusion . 82 5 Landmark Candidate Detection Using Spin Images 87 5.1 Detector Design . 88 5.1.1 Benefits of Using Spin Images . 90 5.2 Cross-Correlation Detector . 91 5.2.1 Learning . 91 5.2.2 Matching . 92 CONTENTS vii 5.3 Linear Discriminant Analysis (LDA) Detector . 94 5.3.1 Linear Discriminant Analysis . 94 5.3.2 Building the Detector . 95 5.4 Evaluation Methodology . 100 5.4.1 Methodology . 102 5.5 Results . 102 5.5.1 Expression Test . 104 5.6 Conclusion . 107 6 Localisation and Labelling of Landmarks Using a Sparse Shape Model 109 6.1 Related Work . 111 6.2 Building a Model . 112 6.2.1 Alignment . 113 6.2.2 Modelling . 116 6.3 Fitting the Model . 120 6.3.1 Fitting the Model with Missing Landmarks . 121 6.3.2 Searching . 122 6.3.3 Analytical Methods . 123 6.3.4 Fitting Alignment . 125 6.4 Testing the Model Fit . 126 6.4.1 Missing Points . 129 6.5 Using Landmark Candidates . 131 6.5.1 Candidate Space Reduction . 135 6.5.2 Optimal RANSAC . 139 6.6 RANSAC Fitting Results . 143 6.6.1 Input Data . 144 6.6.2 Performance Measures . 148 6.6.3 Results . 149 6.6.4 Reduced Model Fit . 151 6.6.5 Hard Failures . 153 6.7 Conclusion . 158 7 Conclusion and Further Work 161 7.1 Contributions . 162 7.1.1 Feature Detector and Landmark Labelling . 163 viii CONTENTS 7.1.2 Sparse Model Fitting . 163 7.2 Limitations and Further Work . 165 7.2.1 Landmark Selection . 165 7.2.2 Candidate Detection . 165 7.2.3 Sparse Shape Modelling and Landmark Localisation . 166 Bibliography 167 List of Figures 1.1 Texture swap example . .3 2.1 Face landmark types . 23 2.2 Spin image example . ..