THESIS Presented at Universite´ d’Auvergne For the degree of Doctor of the Universite´ d’Auvergne Speciality Computer Vision Defended by Florent BRUNET on 2010, November 30 Title Contributions to Parametric Image Registration and 3D Surface Reconstruction Jury President Laurent Sarry Co-Directors Adrien Bartoli Remy´ Malgouyres Nassir Navab Reviewers Lourdes Agapito Joachim Hornegger Etienne´ Memin´ 2 3 Abstract This thesis deals with the modelling and the estimation of parametric functions in Computer Vision. It focuses on three main topics: range surface fitting, image registration, and 3D reconstruction of smooth surfaces. In addition to these three main topics, we consider other transversal elements. In particular, we focus on the general problem caused by the hyperparameters in parametric model estimation, which includes regularization problems. All these topics are related by a single objective which is nowadays one of the most important goals in Computer Vision: the reconstruction of an arbitrary surface from images taken in an arbitrarily deforming environment. This thesis can be divided into four main parts. The first part deals with the basics. It includes background on optimization and on parameter estimation. The problems related to the hyperparameters are also explained and illustrated. The rest of the thesis is centred on our original contributions. The second part of this thesis deals with the problem of fitting a surface to range data. This problem consists in finding a parametric smooth surface that approximates accurately a sparse set of 3D points. We consider two main problems. First, we propose methods to automatically tune the hyperparameters such as a regularization weight. Second, we show how heteroskedastic noise may be handled. Heteroskedastic noise is an important problem since it is typical of range sensors, for instance Time-of-Flight cameras. The third part of this thesis is dedicated to the problem of image registration. We propose three contributions in this topic. First, we present a new warp (image deformation function) able to correctly model the effect of perspective projection. Second, we show how to solve an important problem that typically happens in direct image registration: the problem of the region of interest. Third, we propose a new framework to estimate in a reliable way the hyperparameters needed in feature-based image registration (threshold of an M-estimator, regularization weight, number of control points, etc). The last part of this thesis deals with the problem of reconstructing an inextensible surface from a monocular sequence of images. We also use the hypothesis that a reference shape is known. Using only the motion cue, the problem is ill-posed but, nonetheless, satisfying and plausible results can be obtained. We propose two new formulations to reconstruct the surface: the first one reconstruct a sparse set of points using a second order cone program, and the second one reconstruct a smooth parametric surface using a least-squares minimization problem. Resum´ e´ Cette these` traite de la modelisation´ et de l’estimation de fonctions parametriques´ en vison par ordinateur. Notre travail se concentre sur trois axes principaux : l’ajustement de surfaces sur donnees´ de profondeur, le recalage d’images et la reconstruction de surfaces tridimensionnelles a` partir d’images. En outre, nous abordons des sujets transversaux. En particulier, nous nous interessons´ aux problemes` poses´ par les hyperparametres` (ce qui inclue les problemes` de regularisation).´ Tous ces aspects convergent vers un seul et unique but ultime : la reconstruction de surface quelconques a` partir d’images prises en environnements deformables.´ Cette these` peut etreˆ divisees´ en quatre parties. La premiere` partie traite des el´ ements´ fondamentaux comme les conventions et les notations. Les bases de l’optimisation et de l’estimation des problemes` parametriques´ sont aussi expliquees.´ La notion d’hyperpa- rametre` est expliquee´ et illustree.´ Le reste de ce document est axe´ sur nos contributions originales. La deuxieme` partie de cette these` traite du probleme` de l’ajustement de surface sur des donnees´ de pro- fondeur. Ce probleme` consiste a` determiner´ une surface parametrique´ qui approxime de maniere` fidele` un 4 ensemble discret de points tridimensionnels. Nous etudions´ deux sous-problemes.` En premier lieu, nous pro- posons une methode´ permettant de choisir automatiquement les hyperparametres` controlantˆ l’importance de la regularisation.´ En second lieu, nous montrons comment un bruit het´ erosk´ edastique´ peut etreˆ ger´ e.´ La troisieme` partie de cette these` est dedi´ ee´ au recalage d’images. Nous proposons trois contributions dans ce domaine. Premierement,` nous presentons´ un nouveau modele` parametrique´ permettant de prendre en compte les effets d’une camera´ perspective. Deuxiemement,` nous montrons comment resoudre´ un probleme` important du recalage d’images par approche directe : le probleme` de la region´ d’inter´ et.ˆ Troisiemement,` nous propo- sons un nouveau cadre gen´ erique´ pour l’estimation automatique des hyperparametres` necessaires´ au recalage d’images par approche basee´ primitive (ce qui inclut les seuils de M-estimateurs, le poids de la regularisation´ ou le nombre de points de controlesˆ d’un modele` de deformation).´ Enfin, la quatrieme` partie de cette these` se concentre sur le probleme` de la reconstruction d’une surface a` partir d’une sequence´ monoculaire d’images. Nous faisons l’hypothese` que la surface a` reconstruire est inexten- sible et qu’une forme de ref´ erence´ est disponible. Si l’on n’utilise que l’information produite par le deplacement´ de la surface, ce probleme` est mal pose.´ Neanmoins,´ des resultats´ satisfaisants et plausibles peuvent etreˆ atteints. Nous proposons deux nouvelles formulations permettant de reconstruire la surface : la premiere` reconstruit un ensemble discret de points 3D par programmation conique du second ordre (second order cone programming); la seconde reconstruit une surface parametrique´ lisse en utilisant une minimisation de moindres carres.´ CONTENTS 5 Contents 1 Introduction 17 2 General Tools 23 2.1 Notation, First Definitions . 24 2.1.1 Basic Notations . 24 2.1.2 Functions . 24 2.1.3 Sets and Collections . 25 2.1.4 Matrices and Vectors . 25 2.1.5 Other Common Notation . 27 2.2 Basics on Continuous Optimization . 27 2.2.1 Generalities on Optimization . 28 2.2.2 Optimization Algorithms . 29 2.2.2.1 Iterative Optimization Algorithms . 30 2.2.2.2 Downhill Simplex . 30 2.2.2.3 Gradient descent . 32 2.2.2.4 Newton’s Method . 35 2.2.2.5 Gauss-Newton Algorithm . 37 2.2.2.6 The Normal Equations . 38 2.2.2.7 Levenberg-Marquardt . 39 2.2.2.8 Cholesky Factorization . 40 2.2.2.9 QR Factorization . 42 2.2.2.10 Singular Value Decomposition . 42 2.2.2.11 Iteratively Reweighed Least Squares . 44 2.2.2.12 Golden Section Search . 44 2.3 Parametric Models of Function . 45 2.3.1 Splines . 45 2.3.2 The B-spline Representation . 47 2.3.2.1 The B-Splines Functions . 48 2.3.2.2 Splines as Linear Combinations of B-Splines . 49 2.3.2.3 Uniform Cubic B-Splines . 52 2.3.2.4 Natural Splines . 55 2.3.2.5 B-Splines in Higher Dimensions . 56 2.3.3 Non Uniform Rational B-Splines (NURBS) ....................... 59 6 CONTENTS 2.3.3.1 Basics on NURBS ............................... 60 2.3.3.2 Properties . 62 2.3.4 Radial Basis Functions . 65 2.3.4.1 Generalities . 65 2.3.4.2 Basis Functions . 66 3 General Points on Parameter and Hyperparameter Estimation 69 3.1 Parameter Estimation . 70 3.1.1 General Points on Parameter Estimation . 70 3.1.1.1 Parametric Models of Function . 70 3.1.1.2 Data Measurements, Errors . 70 3.1.1.3 Probability Density Function (PDF) . 71 3.1.1.4 Maximum Likelihood Estimation . 71 3.1.1.5 Maximum A Posteriori Estimation . 72 3.1.1.6 Estimator . 72 3.1.2 Specific Techniques in Parameter Estimation . 73 3.1.2.1 Normal Distribution and Least-Squares . 73 3.1.2.2 Heavy-tailed Distributions and M-estimators . 75 3.1.2.3 Other robust estimators . 78 3.2 Hyperparameters . 80 3.2.1 Generalities . 81 3.2.1.1 A Practical Example . 81 3.2.1.2 Formal Definitions . 82 3.2.2 Automatic Computation of the Hyperparameters . 86 3.2.2.1 General Principle . 86 3.2.2.2 Cross-Validation . 86 3.2.2.3 Mallow’s CP .................................. 89 3.2.2.4 Akaike Information Criterion . 89 3.2.2.5 Bayesian Information Criterion . 91 3.2.2.6 Minimum Description Length . 91 3.2.2.7 Other Approaches . 92 4 Range Surface Fitting 95 4.1 First Definitions and Concepts . 96 4.1.1 First Definitions . 96 4.1.2 Acquisition of Range Data . 97 4.1.2.1 Generalities . 97 4.1.2.2 Passive Sensors . 97 4.1.2.3 Active Sensors . 99 4.1.3 An Introductory Example . 102 4.2 The L-Tangent Norm . 105 4.2.1 Supplementary Details on Range Surface Fitting . 105 4.2.1.1 Generalities . 105 4.2.1.2 The Bending Matrix . 107 4.2.2 The L-Curve Criterion . 112 CONTENTS 7 4.2.3 The L-Tangent Norm Criterion . 113 4.2.3.1 The Proposed Criterion . 115 4.2.3.2 Properties of the L-Tangent Norm . 115 4.2.4 Experimental Results . 116 4.2.4.1 Data . 116 4.2.4.2 Computation Timings . 117 4.2.4.3 Is the L-Tangent Norm an Approximation of Cross-Validation? . 120 4.2.4.4 Reconstructed Surfaces . 120 4.3 Range Surface Fitting with Heteroskedastic Noise . ..