On Motion Estimation Problems in Computer Vision Jiaolong Yang A thesis submitted for the degree of Doctor of Philosophy of The Australian National University September 2016 c Jiaolong Yang 2016 Except where otherwise indicated, this thesis is my own original work. The con- tent is mostly based on the publications during my PhD study as listed below. Publications YANG, J.; LI, H.; AND JIA, Y., 2013. Go-ICP: Solving 3D Registration Efficiently • and Globally Optimally. In International Conference on Computer Vision (ICCV), 1457–1464. YANG, J.; DAI, Y.; LI, H.; GARDNER, H.; AND JIA, Y., 2013. Single-shot Extrinsic • Calibration of a Generically Configured RGB-D Camera Rig from Scene Con- straints. In International Symposium on Mixed and Augmented Reality (ISMAR), 181–188. YANG, J.; LI, H.; AND JIA, Y., 2014. Optimal Essential Matrix Estimation via • Inlier-Set Maximization. In European Conference on Computer Vision (ECCV), 111– 126. YANG, J. AND LI, H., 2015. Dense, Accurate Optical Flow Estimation with Piece- • wise Parametric Model. In IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 1019–1027. YANG, J.; LI, H.; DAI, Y.; AND TAN, R. T., 2016. Robust Optical Flow Estimation • of Double-Layer Images under Transparency or Reflection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1410–1419. YANG, J.; LI, H.; CAMPBELL, D.; AND JIA, Y., 2016. A Globally Optimal Solu- • tion to 3D ICP Point-Set Registration. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 38(11), 2241–2254. YANG, J.; REN, P.; CHEN, D.; WEN, F.; LI, H.; AND HUA, G., 2016. Neural Aggre- • gation Network for Video Face Recognition. In arXiv preprint, arXiv:1603.05474. Jiaolong Yang 15 September 2016 Dedicated to my parents and wife. Abstract Motion estimation is one of the fundamental problems in computer vision. It has broad applications in the fields of robot navigation, mixed and augmented reality, visual tracking, image and video processing, intelligent transportation systems and so on. Up until now, motion estimation is far from a solved problem, and it is still one of the active research topics in and beyond the computer vision community. This thesis is dedicated to both camera motion estimation – including motion estimation for 3D and 2D cameras – and dense image motion for color images. We push the limits of the state of the art in various aspects such as optimality, robustness, accuracy and flexibility. The main contributions are summarized as follows. First, a globally optimal 3D point cloud registration algorithm is proposed and applied to motion estimation of 3D imaging devices. Based on Branch-and-Bound (BnB) optimization, we optimally solve the registration problem defined in Itera- tive Closest Point (ICP). The registration error bounds are derived by exploiting the structure of the SE(3) geometry. Other techniques such as the nested BnB and the integration with ICP are also developed to achieve efficient registration. Experiments demonstrate that the proposed method is able to guarantee the optimality, and can be well applied in estimating the global or relative motion of 3D imaging devices such as 3D scanners or depth sensors. Second, a globally optimal inlier-set maximization algorithm is proposed for color camera motion estimation. We use BnB to seek for the optimal motion which gives rise to the maximal inlier set under a geometric error. An explicit, geometrically meaningful relative pose parameterization – a 5D direct product space of a solid 2D disk and a solid 3D ball – is proposed, and efficient, closed-form bounding functions of inlier set cardinality are derived to facilitate the 5D BnB search. Experiments on both synthetic data and real images confirm the efficacy of the proposed method. Third, a scene constraint based method for relative pose estimation between a 2D color camera and a 3D sensor is developed. We formulate the relative pose estimation as a 2D-3D registration problem minimizing the geometric errors from the known scene constraints. Our method takes only a single pair of color and depth images as input, and is correspondence-free. In addition, a new single-view 3D reconstruction algorithm is proposed for obtaining initial solutions. The experiments show that the method is both flexible and effective, producing accurate relative pose estimates and high-quality color-depth image registration results. Fourth, a highly-accurate optical flow estimation algorithm based on piecewise parametric motion model is proposed. It fits a flow field piecewise to a variety of parametric models where the domain of each piece (i.e., shape, position and size) and its model parameters are determined adaptively, while at the same time main- taining a global inter-piece flow continuity constraint. The energy function takes into vii viii account both the piecewise constant model assumption and the flow field continuity constraint, enabling the proposed algorithm to effectively handle both homogeneous motions and complex motions. The experiments on three public optical flow bench- marks show that the proposed algorithm achieves top-tier performances. At last, we propose a robust algorithm for optical flow estimation in the presence of transparency or reflection. It deals with a challenging, frequently encountered, yet not properly investigated problem in optical flow estimation: the input two frames contain one background layer of the scene and one distracting, possibly moving layer due to transparency or reflection. The proposed algorithm performs both optical flow estimation and image layer separation. It exploits a generalized double-layer brightness consistency constraint connecting these two tasks, and utilizes the priors for both of them. The experiments on synthetic and real images confirm its efficacy. Key Words: Camera Motion, Image Motion, Point Cloud Registration, Branch and Bound, Relative Pose Estimation, Optical Flow, Piecewise Parametric Model, Image Layer Separation. Contents Abstract vii 1 Introduction and Literature Overview 1 1.1 The Camera Motion Estimation Problem . .2 1.1.1 3D Camera Motion Estimation . .2 1.1.2 2D Color Camera Motion Estimation . .5 1.1.3 2D Color Camera and 3D Camera Relative Pose Estimation . 10 1.2 The Image Motion (Optical Flow) Estimation Problem . 12 1.3 Thesis Outline and Contributions . 16 2 Globally Optimal 3D Registration and 3D Camera Motion Estimation 19 2.1 Related Work . 20 2.2 Problem Formulation . 23 2.3 The Branch and Bound Algorithm . 24 2.3.1 Domain Parametrization . 25 2.4 Bounding Function Derivation . 26 2.4.1 Uncertainty Radius . 26 2.4.2 Bounding the L2 Error . 27 2.5 The Go-ICP Algorithm . 30 2.5.1 Nested BnBs . 30 2.5.2 Integration with the ICP Algorithm . 32 2.5.3 Outlier Handling with Trimming . 32 2.6 Experiments . 34 2.6.1 Optimality . 34 2.6.2 “Partial" to “Full" Registration and Camera Global Motion Es- timation . 37 2.6.3 “Partial” to “Partial” Registration and Camera Relative Motion Estimation . 44 2.7 Conclusion . 46 3 2D Camera Motion Estimation via Optimal Inlier-set Maximization 47 3.1 Related Work . 48 3.2 Essential Manifold Parametrization . 49 3.3 Optimization Criteria . 51 3.4 Branch and Bound over D2 B3 ....................... 52 p × p 3.4.1 Lower-bound Computation . 52 3.4.2 Upper-bound Computation via Relaxation . 53 ix x Contents 3.4.3 Efficient Bounding with Closed-form Feasibility Test . 54 3.4.4 The Main Algorithm . 56 3.5 Experiments . 57 3.5.1 Synthetic Scene Test: Normal Cases . 57 3.5.2 Synthetic Scene Test: Special Cases . 60 3.5.3 Real Image Test . 61 3.6 Conclusion . 63 4 2D Camera and 3D Camera Relative Pose Estimation from Scene Constraints 65 4.1 Related Work . 66 4.2 Color and Depth Camera Relative Pose Estimation from Scene Con- straints . 68 4.2.1 Problem Statement . 68 4.2.2 The Proposed Approach . 68 4.2.3 Inverse Projection Estimation . 69 4.2.4 Scene Constraints . 71 4.2.5 Geometric Error Minimization . 72 4.3 Initial Relative Pose Estimation . 73 4.3.1 Single View 3D Reconstruction . 73 4.3.2 Point Cloud Registration . 75 4.4 Experiments . 76 4.4.1 Tests on Synthetic Data . 76 4.4.2 Tests on a Real-world Scene . 79 4.5 Conclusion . 82 5 Piecewise Parametric Optical Flow Estimation 85 5.1 Related work . 87 5.2 Piecewise Parametric Flow Estimation . 88 5.2.1 Energy function . 88 5.2.2 Data term . 89 5.2.3 Flow continuity(inter-piece compatibility)term . 89 5.2.4 Potts model term . 90 5.2.5 MDL term . 91 5.3 Optimization . 91 5.3.1 Alternation . 91 5.3.2 Initialization . 93 5.4 Post-processing . 93 5.4.1 Occlusion handling . 93 5.4.2 Refinement . 93 5.5 Experiments . 94 5.5.1 Results on KITTI . 95 5.5.2 Results on Middlebury . 97 5.5.3 Results on MPI Sintel . 100 5.5.4 Running Time . 101 Contents xi 5.6 Conclusion . 101 6 Layerwise Optical Flow Estimation under Transparency or Reflection 103 6.1 Related Work . 105 6.2 Problem Setup . 106 6.2.1 Linear Additive Imaging Model . 106 6.2.2 Double Layer Brightness Constancy . 107 6.2.3 The Double Layer Optical Flow Problem . 107 6.3 Regularization . 108 6.3.1 Natural Image Prior: Sparse Gradient . 108 6.3.2 Optical Flow Priors: Spatial Smoothness . 109 6.4 Energy Minimization . 109 6.4.1 The Overall Objective Function . 109 6.4.2 Alternated Minimization . 110 6.5 Experiments . 114 6.5.1 Static Foreground Cases . 114 6.5.2 Dynamic Foreground Cases . 119 6.6 Conclusion . 121 7 Summary and Future Work 125 7.1 Summary and Contributions . 125 7.2 Future Work . ..