Surface Recovery: Fusion of Image and Point Cloud Siavash Hosseinyalamdary Alper Yilmaz The Ohio State University 2070 Neil avenue, Columbus, Ohio, USA 43210 pcvlab.engineering.osu.edu Abstract construct surfaces. Implicit (or volumetric) representation of a surface divides the three dimensional Euclidean space The point cloud of the laser scanner is a rich source of to voxels and the value of each voxel is defined based on an information for high level tasks in computer vision such as indicator function which describes the distance of the voxel traffic understanding. However, cost-effective laser scan- to the surface. The value of every voxel inside the surface ners provide noisy and low resolution point cloud and they has negative sign, the value of the voxels outside the surface are prone to systematic errors. In this paper, we propose is positive and the surface is represented as zero crossing two surface recovery approaches based on geometry and values of the indicator function. Unfortunately, this repre- brightness of the surface. The proposed approaches are sentation is not applicable to open surfaces and some mod- tested in realistic outdoor scenarios and the results show ifications should be applied to reconstruct open surfaces. that both approaches have superior performance over the- The least squares and partial differential equations (PDE) state-of-art methods. based approaches have also been developed to implicitly re- construct surfaces. The moving least squares(MLS) [22, 1] and Poisson surface reconstruction [16], has been used in 1. Introduction this paper for comparison, are particularly popular. Lim and Haron review different surface reconstruction techniques in Point cloud is a valuable source of information for scene more details [20]. Berger et al. also describe the advantages understanding. There are scientific endeavors to detect and of different methods to handle noise, sparsity, missing data classify objects in a scene using Kinect point cloud [17]. In and misalignment in the point cloud and they investigate addition, urban challenge has proven that the point cloud, different applications of surface reconstruction [3]. collected from laser scanner, is an essential source of in- formation for reliable traffic understanding resulted in au- tonomous driving [19]. The accurate and high resolution Beside point cloud, the surface can be recovered from laser scanners are expensive and may not be cost-effective image content, known as shape from shading. The semi- to be used in autonomous vehicles. In contrast, point cloud nal works of Horn and his colleagues have initiated shape generated from inexpensive laser scanners are noisy and low from shading approaches [13, 15]. This problem has been resolution and prone to systematic errors. Therefore, 3D extensively studied for the laboratory environments where scene recovery based on sparse and noisy point clouds has the different illuminations, light sources, object properties, attracted scientific attentions. Independently, scientists has and pose of the cameras are known [28, 7]. However, this also attempted to reconstruct the surface of objects based on problem has still remained unsolved for realistic scenarios the images for many years. In this paper, we combine these where the surface reflectance and light source properties are two sources of information to recover the surface of the ob- unknown. jects and remove the noise from the point cloud. Figure 1 shows the sparse point cloud collected by laser scanner overlaid on the image. This paper relates the surface curvature and the bright- Surface of an object can be implicitly or explicitly rep- ness changes on the surface and introduces a cost function resented in three dimensional Euclidean space. In explicit to minimize their difference. Consequently, a regularizer representation, a function is fitted to the sampled points is constructed based on this cost function and it is used to of the surface. The Delaunay triangulation is the earliest recover the surface. Due to the complexity of the lighting attempt to explicitly represent the surface using triangles. situation, the regularizer cannot obtain satisfactory results Moreover, splines and its derivatives [18] are applied to re- unless it is updated by the sampled points of the surface. 28 (0,0) (u,v) 0 X W Figure 1. This paper uses the image (left) and point cloud (right) n sources to reconstruct the surface. X' -1 -50 0 50 2. Methodology x Figure 2. The profile of the surface in local coordinate system is A local coordinate system is defined at every sampled presented in this figure. The sampled point X on the surface is the point to locally represent the surface of an object. We pro- origin of the local coordinate system and first and second axes of pose geometry and brightness based approaches to recover the local coordinate system are laid on the tangent space, ~u and ′ the surface at the neighborhood of the sampled points. The ~v. Every point on the surface, X , has two components, u and geometry based approach has been previously described in v on the tangent space and one component in normal direction, X′ X′ [14] .Finally, we introduce a global constraint to remove W . Point can be stated with =[u,v,W (u,v)] in the local the discontinuities between locally estimated surfaces and coordinate system. assure continuity of the surface. 2.1. Local coordinate system Euclidean coordinate system to the point X and it maps X~ , Every surface is a two dimensional manifold embedded Y~ , and Z~ of Euclidean coordinate system into ~u, ~v, and ~n into the three dimensional Euclidean space, S : R2 Λ of the local coordinate system. Figure 2 shows a profile of → ⊂ R3. Λ is a subset of R3 that encompasses the surface. If the surface where it is passed through point X and point X′. we assume that the surface of the object is smooth and con- The tangent space is defined at point X and point X′ can be sequently differentiable, it becomes a Riemannian manifold recovered if W (uX′ ,vX′ ) is estimated. and the tangent space, , can be defined. T In order to estimate W (uX′ ,vX′ ), point X and its nor- X Let’s assume that point is given on the surface and the mal vector are not sufficient and other sampled point of the X normal vector of the surface, ~n, is known at point . Let’s surface should be projected to the local coordinate system. define two orthonormal vectors ~u and ~v in the tangent space. Let’s assume, the sampled points of the surface in the neigh- Local coordinate system is defined in the way that its origin borhood of point X are Π = X1,..., Xn . The moving X is located at point and its bases are the ~u, ~v, and ~n. Let’s least squares (MLS) fits a polynomial{ to the} sampled points X define ΩX the neighborhood of the point on the surface in the local coordinate system to recover the surface. The X′ and assume there is a point ΩX on the surface at the disadvantage of MLS is that it assumes the neighborhood X ∈ X′ neighborhood of point . The coordinates of point is is the same in all directions. Therefore, it may violate the X′ ′ ′ ′ ⊤ ′ X′ =[uX ,vX ,wX ] . wX is the distance of the point boundaries of the surface. We propose an approach to de- from the tangent space and it can be written as a function fine the neighborhood based on geometry of the sampling of the point coordinates in tangent space, u and v such that points to preserve the boundaries. W (uX′ ,vX′ ) = wX′ . W is a scalar field which indicated Geometry of the surface at the local neighbor- the distance of every point from the surface to the tangent hood can be represented as the co-variance matrix space. In order to recover the surface, it suffices that the of the sampled points. The co-variance matrix, scalar field W is estimated in the neighborhood of point X. i i SXX = (X X)⊤(X X) represents a three- i The estimation of W is independent of the definition of two X ∈ΩX − − vector bases ~u and ~v. Here, we simplify the definition of W dimensionalP ellipsoid. The axes of the ellipsoid are eigen- by choosing the following transformation between the local vectors, (θ1,θ2,θ3) of the matrix SXX corresponding to the coordinate system, defined for the neighborhood of point eigenvalues, (λ1 λ2 λ3 0), of this matrix. The X, and Euclidean coordinate system, such that normal to the surface≥ at≥ point ≥X is corresponding to the eigenvector θ3, with the smallest eigenvalues, λ3. Two other X′ R R X′ = 1(α) 2(β) Euclidean. (1) eigenvectors, θ1,θ2, are corresponding to the two largest eigenvalues λ λ , indicate an ellipse in the tangent 1 ≥ 2 If n1,n2, and n3 are elements of ~n in 3D Euclidean space, space. The ellipse follows geometry of the surface in the it leads to α = arctan( n2 ) and β = arccos n2 + n2. − n3 2 3 tangent space. That is, it become a circle within the bound- R R ~ ~ 1 and 2 are rotations around X and Y axes ofp Euclidean aries of the surface and it elongates at the boundaries of the coordinate system. This transformation transfers origin of surface. Therefore, this definition of the neighborhood of 29 point X does not violate the boundaries of the surface and Solution of this equation is a uniform gaussian filter that consequently, it preserves the boundaries. Figure 3 shows provides a smooth surface. The disadvantage of this method our definition of the neighborhood of point X in the tangent is that it smoothes the corners and edges of the surface since space. it is uniform gaussian filter.