UC San Diego UC San Diego Electronic Theses and Dissertations Title From pictures to 3D : global optimization for scene reconstruction Permalink https://escholarship.org/uc/item/8rs3b74c Author Chandraker, Manmohan Krishna Publication Date 2009 Peer reviewed|Thesis/dissertation eScholarship.org Powered by the California Digital Library University of California UNIVERSITY OF CALIFORNIA, SAN DIEGO From Pictures to 3D: Global Optimization for Scene Reconstruction A dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Philosophy in Computer Science by Manmohan Krishna Chandraker Committee in charge: Professor David Kriegman, Chair Professor Serge Belongie Professor Samuel Buss Professor Fredrik Kahl Professor Gert Lanckriet Professor Matthias Zwicker 2009 Copyright Manmohan Krishna Chandraker, 2009 All rights reserved. The dissertation of Manmohan Krishna Chandraker is approved and it is acceptable in quality and form for publication on microfilm and electronically: Chair University of California, San Diego 2009 iii DEDICATION To Papa, for his incomparable example. To Mom, for her innumerable sacrifices. To Didi, for her unbridled sisterly pride. iv EPIGRAPH Somewhere afield here something lies In Earth’s oblivious eyeless trust That moved a poet to prophecies - A pinch of unseen, unguarded dust. Thomas Hardy, “Shelley’s Skylark” v TABLE OF CONTENTS Signature Page............................................ iii Dedication.............................................. iv Epigraph...............................................v Table of Contents.......................................... vi List of Figures............................................ xi List of Tables............................................ xiii Acknowledgements........................................ xiv Vita................................................... xviii Abstract of the Dissertation................................... xx Chapter 1 Introduction......................................1 1.1 Multiview Geometry and Optimization.....................4 1.2 3D Reconstruction from 2D Images.......................7 1.2.1 The Projective Ambiguity.........................8 1.2.2 Projective Spaces and Projective Cameras.............. 10 1.2.3 Stratification of 3D Reconstruction................... 13 1.2.4 Autocalibration................................ 15 1.2.5 Feature Selection and Matching..................... 17 1.3 The Optimization Framework........................... 19 1.3.1 Global Optimization for 3D Reconstruction............. 20 1.3.2 Optimization for Robust SFM...................... 25 1.4 Contributions of the Dissertation......................... 26 1.5 How to Read This Dissertation.......................... 27 Chapter 2 Preliminaries: Projective Geometry....................... 30 2.1 Axiomatic Projective Geometry.......................... 30 2.2 Projective Geometry of 2D............................. 32 2.3 Projective Geometry of 3D............................. 36 2.3.1 Points and Planes.............................. 36 2.3.2 Lines...................................... 36 2.3.3 Quadrics.................................... 39 2.4 The Projective Camera................................ 39 2.5 The Plane at Infinity and Its Denizens...................... 42 2.5.1 The Absolute Conic............................. 44 vi 2.5.2 Image of the Absolute Conic....................... 45 2.5.3 The Absolute Dual Quadric........................ 46 Chapter 3 Preliminaries: Multiview Geometry....................... 50 3.1 Feature Selection and Matching.......................... 50 3.1.1 Corner detection............................... 51 3.1.2 Feature matching.............................. 52 3.1.3 Advanced feature descriptors....................... 53 3.2 Epipolar Geometry.................................. 54 3.3 Projective Reconstruction.............................. 57 3.3.1 Pairwise reconstruction.......................... 57 3.3.2 Factorization-based approaches..................... 58 3.4 Stratification...................................... 60 3.5 Chirality......................................... 62 3.5.1 Bounding the plane at infinity...................... 64 Chapter 4 Global Optimization................................. 65 4.1 Approaches to Global Optimization....................... 66 4.2 Convex Optimization................................. 69 4.2.1 Convex Sets.................................. 69 4.2.2 Convex Functions.............................. 71 4.2.3 Convex Optimization Problems..................... 72 4.2.4 Linear Matrix Inequalities......................... 74 4.3 Branch and Bound Theory............................. 76 4.3.1 Bounding................................... 79 4.3.2 Branching................................... 80 4.4 Global Optimization for Polynomials...................... 82 Chapter 5 Triangulation and Resectioning.......................... 89 5.1 Introduction....................................... 89 5.1.1 Related Work................................. 90 5.1.2 Outline..................................... 91 5.2 Problem Formulation................................. 92 5.3 Traditional Approaches............................... 94 5.3.1 Linear Solution................................ 94 5.3.2 Bundle Adjustment............................. 96 5.4 Fractional Programming............................... 97 5.4.1 Bounding................................... 98 5.5 Applications to Multiview Geometry....................... 100 5.5.1 Triangulation................................. 101 5.5.2 Camera Resectioning............................ 103 5.5.3 Projections from Pn to Pm ........................ 103 5.6 Multiview Fractional Programming....................... 104 vii 5.6.1 Bounds Propagation............................. 104 5.6.2 Initialization.................................. 106 5.6.3 Coordinate System Independence.................... 107 5.7 Experiments....................................... 108 5.7.1 Synthetic Data................................ 108 5.7.2 Real Data................................... 112 5.8 Discussions....................................... 115 Chapter 6 Stratified Autocalibration.............................. 117 6.1 Introduction....................................... 118 6.2 Background....................................... 121 6.2.1 The Infinite Homography Relation................... 121 6.2.2 Modulus Constraints............................ 124 6.2.3 Chirality Bounds on Plane at Infinity.................. 126 6.2.4 Need for Global Optimization...................... 126 6.3 Previous Work..................................... 128 6.4 The Branch and Bound Framework........................ 130 6.4.1 Constructing Convex Relaxations.................... 131 6.5 Global Estimation of Plane at Infinity...................... 132 6.5.1 Traditional Solution............................. 132 6.5.2 Problem Formulation............................ 133 6.5.3 Convex Relaxation............................. 133 6.5.4 Incorporating Bounds on the Plane at Infinity............ 134 6.6 Globally Optimal Metric Upgrade........................ 136 6.6.1 Traditional Solution............................. 136 6.6.2 Problem Formulation............................ 137 6.6.3 Convex Relaxation............................. 138 6.7 Experiments....................................... 141 6.8 Conclusions and Further Discussions...................... 150 Chapter 7 Direct Autocalibration................................ 153 7.1 Introduction....................................... 153 7.2 Background....................................... 156 7.2.1 Autocalibration Using the Absolute Dual Quadric......... 157 7.2.2 Chirality.................................... 158 7.3 Related Work...................................... 159 7.4 Problem Formulation................................. 161 7.4.1 Imposing rank degeneracy and positive semidefiniteness of Q∗ 161 7.4.2 Imposing chirality constraints......................1 162 7.4.3 Choice of objective function....................... 163 7.5 Experiments with synthetic data.......................... 165 7.6 Experiments with real data............................. 167 7.7 Conclusions....................................... 172 viii Chapter 8 Bilinear Programming................................ 174 8.1 Introduction....................................... 174 8.2 Related Work...................................... 176 8.3 Formulation....................................... 177 8.3.1 LP relaxation for the L1-norm case................... 177 8.3.2 SOCP relaxation for the L2-norm case................. 179 8.3.3 Additional notes for the L2 case..................... 180 8.4 Branching strategy.................................. 181 8.5 Experiments....................................... 184 8.5.1 Synthetic data................................. 184 8.5.2 Applications.................................. 188 8.6 Discussions....................................... 190 Chapter 9 Line SFM Using Stereo............................... 192 9.1 Introduction....................................... 192 9.2 Related Work...................................... 195 9.3 Structure and Motion Using Lines........................ 197 9.3.1 A Simple Solution?............................. 197 9.3.2 Geometry of the Problem......................... 198 9.3.3 Linear Solution................................ 198 9.3.4 Efficient Solutions for Orthonormality................. 200 9.3.5 Solution for Incremental Motion.................... 204 9.3.6 A Note on Number of Lines......................