Technische Universität Berlin THE ROBUST 3-D SCENEFLOW Generalized Video-based 3-D Analysis using Robust Camera and Scene Geometry Estimations Von der Fakult¨at IV - Elektrotechnik und Informatik der Technischen Universit¨at Berlin zur Verleihung des akademischen Grades Doktor der Ingenieurwissenschaften Dr.-Ing. genehmigte Dissertation vorgelegt von Jang Heon Kim aus S¨ud Korea Promotionsausschuss: Vorsitzender: Prof. Dr.-Ing. Klaus-Robert M¨uller Berichter: Prof. Dr.-Ing. Olaf Hellwich Berichter: Prof. Dr.-Ing. Thomas Sikora Tag der wissenschaftlichen Aussprache : 9. 9. 2008 Berlin 2009 D83 Technische Universität Berlin THE ROBUST 3-D SCENEFLOW Generalized Video-based 3-D Analysis using Robust Camera and Scene Geometry Estimations by Jang Heon Kim Master of Science in Electronic Engineering Yonsei University, Seoul, Korea, 2003 A dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Engineering in Electrical Engineering and Computer Science Faculty IV. Electrical Engineering and Computer Sciences Technische Universit¨at Berlin Berlin 2009 D83 Copyright c 2008 Jang Heon Kim All Rights Reserved. No part of the publication may be reproduced in any form by print, photoprint, microfilm or any other means without written permission from the publisher. The University reserves the right to make copies of the work for academic purposes within the University and to allow free access to the copy of the thesis retained by the Library. This page must form part of any copies made. In written agreements of sponsorship, the institute retains the ownership of the intellectual property in supported contributions. Technische Universit¨at Berlin Address: Strasse des 17. Juni 135, 10623 Berlin, Germany Phone: +49 (0)30 314-0, Fax: +49 (0)30 314-23222 http://www.tu-berlin.de Institut f¨ur Telekommunkationssysteme Address: Sekr. EN 1, Einsteinufer 17, 10587 Berlin, Germany Phone: +49 (0)30 314-25093, Fax: +49 (0)30 314-22514 http://www.nue.tu-berlin.de Declaration This dissertation is submitted to the Technische Universit¨at Berlin in partial fulfillment for the degree of Doctor of Philosophy. It is an account of work undertaken at the Faculty IV. Electrical Engineering and Computer Sciences between May 2004 and September 2008 under the supervision of Prof. Dr.-Ing. Thomas Sikora. I already read and understand all regulations for completion of the degree of Doctor of Philosophy. This dissertation is the result of my own work and includes nothing which is the outcome of work done in collaboration. This dissertation is not substantially the same as any I have submitted for a degree or diploma or other qualification at any other University. I further state that no part of my dissertation/thesis has already been, or is being concurrently submitted for any other degree, or other qualification. September, 2008 ——————————————————— Jang Heon Kim iii To my Parents inspiring me and dedicating their whole lives for my education. Contents Declaration iii List of Tables x List of Fingures xiii Acknowledgments xiv Abstract xv Zusammenfassung xvi 1 Introduction 1 1.1 DepthCues .................................................... 1 1.2 3-DVideo...................................................... 3 1.3 Multi-viewApproachesfor3-DMotionAnalysis........................ 6 1.4 ProposedMethod................................................ 9 1.5 BackgroundandContributions..................................... 12 2 Related Works 15 2.1 MultiviewVideoAnalysis ......................................... 15 2.2 Image-basedModelingandRendering................................ 17 2.3 CameraGeometryEstimation...................................... 17 2.4 SceneGeometryEstimation........................................ 20 3 Camera Geometry Estimation 25 3.1 Projective Geometry Estimation . 25 3.2 PinholeCameraModel............................................ 30 3.3 Complete3-DCameraModel ...................................... 33 3.4 Ray-spaceDefinitionusingOpticalFlow.............................. 36 3.5 ExperimentalResults............................................. 39 vii 4 Spatio-temporal Collineation of View Geometry 46 4.1 EpipolarGeometry............................................... 46 4.2 Fundamental Matrix . 48 4.3 Fundamental Matrix and Camera Projections . 49 4.4 Fundamental Matrix Estimation with Parallax . 50 4.5 EssentialMatrixEstimation ....................................... 52 4.6 Maximum and Minimum Bounds of Epipolar Range . 53 4.7 Epipolar Rectification . 54 4.8 TemporalEpipolarGeometry ...................................... 57 4.9 ExperimentalResults............................................. 58 5 Unified Representation of Robust Estimations 68 5.1 RobustnessoftheEstimator....................................... 69 5.2 RobustEstimationwithRandomSampling............................ 70 5.3 Unified Representation of Robust Fundamental Matrix Estimation . 71 5.4 AnisotropicRegularizationusingRobustEstimator..................... 75 5.5 Perceptual Maximum Variation Modeling for Over-diffusion Problem . 82 5.6 Robust Anisotropic Color Image Regularization with Perceptual Maximum VariationModeling .............................................. 87 5.7 ExperimentalResults............................................. 89 6 Robust 3-D Sceneflow Estimation 99 6.1 RobustAnisotropicDisparityEstimation............................. 99 6.2 Anisotropic Disparity Estimation with Perceptual Maximum Variation . 106 6.3 RobustHybridRecursiveOpticalFlowEstimation...................... 107 6.4 3-DSceneflowEstimation ......................................... 117 6.5 ExperimentalResults............................................. 119 7 Confocal Stereo with Pinhole Image Recovery 133 7.1 ProblemofPinholeCameraAssumption.............................. 133 7.2 Real-ApertureStereoCameraAnalysis............................... 134 7.3 ConfocalDisparityEstimation...................................... 135 7.4 ConfocalConstraintofDefocus..................................... 136 7.5 Anisotropic Disparity Estimation with Confocal Constraint and Recovery of PinholeImage .................................................. 138 7.6 ExperimentalResults............................................. 139 8 Applications 143 8.1 Image-andVideo-basedRendering.................................. 143 8.2 DepthSegmentationandImageComposition.......................... 148 viii 8.3 ImageandVideo-basedModeling................................... 152 9 Discussion 155 9.1 SceneflowMethodforRobust3-DVideoAnalysis....................... 155 9.2 Future directions . 157 10 Conclusion 159 A Appendix 160 A.1 Relationship between Collineations and Epipolar Geometry . 160 A.2 Triangulation................................................... 161 A.3 HomographyandCanonicalBasis................................... 162 A.4 Projective Transformations of a Plane . 162 A.5 8-PointAlgorithm............................................... 163 A.6 ComputationofRotationandTranslation............................. 164 A.7 Bundle Adjustment . 166 ix List of Tables 1.1 2-Dand3-Dmotionmodelsincomputervision ........................ 4 3.1 Measures that remain invariant under the transformations in the hierarchy of geometries..................................................... 28 5.1 Several kinds of ρ, ψ and w functions................................. 71 5.2 Graphs of ρ, ψ and w functions..................................... 72 5.3 Algorithmoftheperceptualcolormodeling ........................... 83 5.4 Comparison using Lena image with 4, 10% color impulsive noise . 93 6.1 Performancecomparisonusingpercentoferrorpixels.................... 127 6.2 Performancecomparisonfordiscretemotion........................... 128 x List of Figures 1.1 Humanvisualsystem............................................. 2 1.2 Disparitymap .................................................. 3 1.3 Spatio-temporal motion effects . 4 1.4 Relationship between SfM (Structure-from-Motion) and multi-stereo methods 7 1.5 Blockdiagramoftheproposedmethod............................... 10 1.6 3-Dsceneflowmethod ............................................ 12 2.1 Multiviewcameraconfigurations.................................... 16 2.2 Collineation in camera calibration pattern . 19 2.3 Stereocamerabaselines........................................... 21 2.4 Combinationofwideandnarrowbaselines............................ 22 3.1 Perspective projection and parallelism . 27 3.2 Pinholecameramodel............................................ 31 3.3 Cameracalibrationandskewfactor.................................. 32 3.4 Transformation between the camera and world coordinates . 34 3.5 Plenopticfunction............................................... 36 3.6 Space-timeray-space............................................. 37 3.7 Featuretrackinginaplanarsurface ................................. 40 3.8 Trajectory of moving features by camera motion . 41 3.9 Cameracalibrationresultsfor4frames............................... 41 3.10Relative3-Dpositionbetweencamerasandtrackedpoints................ 42 3.11Optical-ray..................................................... 43 3.12Virtualviewpointinray-space...................................... 44 3.13 Fourier spectral analysis of ray-space . 45 4.1 Epipolargeometry............................................... 47 4.2 Parallax in projection rays . 51 4.3 Epipolar rectification . 55 4.4 Minimum angle to avoid pixel loss in epipolar rectification . 56 4.5 Rectified images . 56 xi 4.6 Spatio-temporalepipolargeometry.................................