Improved Algorithms for Incremental Self-Calibrated Reconstruction from Video
Total Page:16
File Type:pdf, Size:1020Kb
Improved Algorithms for Incremental Self-calibrated Reconstruction from Video por Rafael Lemuz López Tesis sometida como requisito parcial para obtener el grado de DOCTOR EN CIENCIAS EN LA ESPECIALIDAD DE CIENCIAS COMPUTACIONALES en el Instituto Nacional de Astrof´ısica, Optica´ y Electronica´ Abril 2008 Tonantzintla, Puebla Supervisada por: Dr. Miguel Octavio Arias Estrada, INAOE °c INAOE 2008 El autor otorga al INAOE el permiso de reproducir y distribuir copias en su totalidad o en partes de esta tesis ii iii Summary Self-calibrated 3D reconstruction algorithms deal with the problem of recov- ering the three-dimensional structure of the scene and the camera motion using 2D images. A distinctive property of self-calibrated reconstruction methods is that camera calibration (the estimation of the camera intrinsic parameters: focal length, principal point, and radial lens distortion; and extrinsic parameters: orien- tation and position) is computed using intrinsic geometric information contained in the projective images of real scenes. Algorithms to solve 3D reconstruction problems heavily relay in finding correct matches between salient features that correspond to the same scene elements in different images. Then, by using corre- spondence data, a projective estimate of 3D scene structure and camera motion is computed. Finally using geometric constraints the camera parameters and the projective model are upgrade to a metric one. This thesis proposes new algorithms to solve problems involved in self-calibrated reconstruction methods, including salient point detection, robust feature match- ing and projective reconstruction. An improved salient point detection algorithm is proposed, that ranks better interest points accordingly to the intuitive notion of corner points by computing directly the angular difference between dominant edges. A robust feature matching algorithm that merges spatial and appearance properties between putative match candidates that increase the number of cor- rect matches and discard false matches pairs is also proposed. In addition, a projective reconstruction algorithm is proposed that selects on-line the most con- tributing frames in the projective reconstruction process to overcome one of the intrinsic limitation of factorization like algorithms, to deal with the problem of key frame selection in the 3D self-calibrated pipeline. A full pipeline for a 3D reconstruction algorithm is developed with the proposed algorithms. Promising iv results are shown and contributions and limitations of this work are discussed. v vi Resumen Los algoritmos de reconstrucci´on3D auto-calibrada tratan con el problema de recuperar la informaci´on3D de una escena y el movimiento de la c´amara a partir de im´agenes. Una propiedad distintiva de los m´etodos de reconstrucci´on auto- calibrada es que los par´ametros intrinsecos de la c´amara: longitud focal, punto principal, e incluso la distorci´on radial; as´ıcomo los par´ametros extrinsecos: la orientaci´ony posici´onrelativa de la c´amara con respecto a la escena se calculan utilizando informaci´ongeom´etricaintrinsecamente contenida en las im´agenes de una escena real est´atica. Es decir, estos m´etodos no utilizan herramientas adi- cionales como motores de retroalimentaci´onpara el c´alculode la longitud focal o patrones de calibraci´onprefabricados. Sin embargo, el proceso de reconstrucci´on autocalibrada, depende fuertemente de tener identificados puntos de correspondencia entre regiones de imagenes que representan al mismo elemento de la escena capturados desde puntos de obser- vaci´ondiferentes. As´ı,utilizando unicamente puntos de correspondencia se obtiene una primera estimaci´on de la estructura de la escena y el movimento de la c´amara que no preserva distancias y ´angulos, llamada reconstrucci´onprojectiva. Poste- riormente haciendo algunas suposciones e imponiendo restricciones sobre algunos par´ametros de la c´amara el modelo proyectivo se lleva a un modelo euclideando que difiere de la representaci´on de la escena real por un factor de escala y la orientaci´onoriginal. En esta tesis se proponen nuevos algoritmos para el problema de reconstrucci´on autocalibrada, en particular para los problemas de: detecci´on de puntos de inter´es, b´usquedade correspondencias y reconstrucci´on proyectiva. Se propone un algoritmo para la detecci´onde puntos de inter´es,que ordena mejor los puntos detectados de acuerdo a la noci´onintuitiva de esquina calculando vii directamente la diferencia angular entre los bordes dominantes. Un nuevo algo- ritmo para la b´usquedade correspondencias que integra propiedades espaciales y de apariencia en una m´etrica de similaridad entre posibles puntos de corresopon- dencia. El nuevo algoritmo incrementa el n´umerode pares de correspondencia y al mismo tiempo disminuye los errores de empatamiento. Adem´as, se propone un al- goritmo de reconstrucci´onproyectiva que selecciona en tiempo de ejecuci´on las im- agenes que mas contribuyen durante el proceso de reconstrucci´onpara sobrepasar una de las limitaciones inerentes a los algoritmos de reconstrucci´on proyectiva basados en el m´etodo de factorizaci´on:la selecci´onde los frames m´asimportantes durante el proceso completo reconstruci´on auto-calibrada. Finalmente, se mues- tran resultados prometedores y se discuten las contribuciones y limitaciones de este trabajo. viii ix Acknowledgements There are many people who have provided guidance, and support throughout the years to whom I wish thanks. First my advisor, Miguel Octavio Arias Estrada who has guided me through these years and has taught me what it means to be a researcher. Secondly to Patrick Hebert, who pointed me, the significance of clear and precise communication of research results. I want to thank to the Professors Leopoldo Altamirano Robles, Olac Fuentes Chaves and Aurelio L´opez L´opez because they have a great impact in my academic and professional skills giving me the opportunity to interact with them during my stay at the INAOE. Then to Eliezer Jara for teaching me the way of systematic analysis in laboratory practices and share his invaluable experience in building prototypes for diverse computer vision applications which have an enormous impact in my professional formation. I also want to thank the interesting people I have met along the way whom I have the opportunity of interacting through informal discussions, and some provide support and encouragement, Blanca, Rita, Irene, Luis, Jorge, and Marco Aurelio. Specially I want to express my gratitude to Carlos Guillen for the hours invested in clarifying some mathematical concepts during the last year. And the guys of the LVSN lab at Laval university, in particular to Jean-Daniel Deschˆenes and Jean-Nicolas Ouellet for make so pleasant the visit to Quebec. Finally, I also want to recognize the facilities given by the technical staff of the INAOE in particular the people of the computer science department. This research was done with the financial support of the CONACYT scholar- ship grant 184921. x xi Dedicatory To my parents and brothers .... xii Contents 1 Introduction 1 1.1 Overview of 3D reconstruction from video . 4 1.1.1 Interest point detector . 6 1.1.2 Matching correspondence . 7 1.1.3 Projective reconstruction . 8 1.1.4 Self-Calibration . 9 1.1.5 Rectification . 10 1.1.6 Dense Stereo Reconstruction . 10 1.2 Objectives . 11 1.2.1 Main Objective . 11 1.2.2 Particular Objectives . 11 1.3 Contributions . 12 1.3.1 Robust feature matching . 12 1.3.2 Incremental 3D reconstruction by inter-frame selection . 12 1.4 Organization of the Thesis . 13 1.5 Conclusions . 13 2 Multiple View Geometry 15 2.1 Preliminaries . 15 2.1.1 Homogeneous Coordinates . 15 xiv CONTENTS 2.2 Camera Models . 16 2.2.1 Perspective model . 16 2.2.2 Orthographic Model . 19 2.2.3 Lens Distortion . 20 2.3 Multiple View Constraints . 20 2.3.1 Two view Geometry . 21 2.3.2 Fundamental Matrix estimation . 22 2.3.3 Planar Homography . 24 2.3.4 Homography estimation . 25 Number of Measurements . 26 2.3.5 Projective Reconstruction . 26 Merging Projective matrices using Epipolar Geometry . 26 The Factorization Method . 28 Non-linear Bundle Adjustment . 29 2.3.6 Incremental Projective Reconstruction . 30 2.4 3D Scene Reconstruction . 30 2.4.1 Camera Calibration . 30 2.4.2 Triangulation . 31 2.4.3 Survey of Camera Calibration . 32 Photogrammetric calibration . 32 Self-calibration . 33 2.4.4 Absolute Conic . 35 2.5 Stratified Self-calibration . 37 2.5.1 Affine Stratification . 38 2.6 RANSAC computation . 39 2.7 Conclusions . 40 CONTENTS xv 3 The Correspondence Problem 41 3.1 Introduction . 41 3.2 Feature Correspondence Overview . 42 3.3 Salient point detection . 43 3.3.1 Pioneer Feature Detectors . 44 First Derivative Methods . 44 Second derivative methods . 46 Local energy methods . 47 Detectors of junction regions . 47 3.3.2 Invariant Feature Detectors . 48 3.4 Salient point Descriptor . 49 3.4.1 SIFT descriptor . 50 3.5 Matching salient points . 51 3.6 Geometric Constraints for Matching . 51 3.7 The importance of Gaussian Integration Scale and Derivative filters 53 3.8 Cov-Harris: Improved Harris corner Detection . 55 3.8.1 Segmentation of Partial Derivatives . 55 3.8.2 Edge direction estimation by Covariance Matrix . 57 3.8.3 Ranking Corner Points by the Angular difference between dominant edges . 58 3.9 Discussion . 60 4 IC-SIFT: Robust Feature Matching Algorithm 63 4.1 Introduction . 63 4.2 Related Work . 64 4.2.1 Scale Invariant Feature Transform . 66 4.2.2 Iterative Closest Point ICP . 68 xvi CONTENTS 4.3 IC-SIFT: Iterative Closest SIFT . 71 4.3.1 Finding Initial Matching Pairs . 71 4.3.2 Matching SIFT features: adding a weighted distance factor 72 4.3.3 Differencing Registration Error . 73 4.4 Robust feature Matching Experimental Results . 76 4.5 Discussion . 83 5 A new Incremental Projective Factorization Algorithm 85 5.1 Introduction . 85 5.2 Related Work . 86 5.3 Projective Factorization .