MODELLING THE WORLD IN REAL TIME Rui Liu, Michael Suppa, Gerd Hirzinger, Darius Burschka Institute of Robotics and Mechatronics, German Aerospace Center (DLR) Oberpfaffenhofen, 82234 Wessling, Germany. E-mail: [email protected] Abstract: This paper presents the automatic landscape modelingssystem of German Aerospace Center. It focuses on the real time 3D modeling process, which is performed at three scales: the first is large scale, in which the outdoor landscape point clouds generated by stereo-methods with a high resolution stereo-camera on a flying plane will be tiled and processed to a group of simplified meshes with compressed textures automatically; the second is medium scale, in which the indoor scans obtained with a laser range scanner are registered to each other without any marker, then the resulted point-clouds are meshed and simplified; and the third is small scale, in which a hand-guided multi-sensor device captures the detailed information of small objects, which will be merged into the whole 3D models. 1. Introduction With the rapid development of computer technology and widely spreading of internet, digital landscape becomes a high topic in scientific area recently. Man uses airplane, airship, even satellite to gain the colored 3D information outdoors and utilizes medium- and small-sized devices to capture the scene indoors. Of course, the obtained data amounts are very large. Therefore, how to handle these data, to simplify them, to transform them to 3D models fast and automatically for further applications, becomes a challenging task. This paper presents a real time automatic modeling process to generate 3D models at three layers (See Figure 1.). Input Modelling 2.5D Colored Texture Layer I Tiling Meshing Simplify Merger Point-cloud Mapping Mesh Automatic Fusion Laser Range Meshing Optimization Layer II Images Registration Online Hole Texture Point-cloud Meshing Filling Mapping Layer III & Pictures Figure 1: Three-layer real-time world modelling system The first layer is the at the largest scale, which converts the huge amount of 2.5D colored point clouds of landscape to simplified meshes mapped with compressed pictures. The input data are resulted from stereo matching methods [1] with resolution of 0.2 m. The hardware device on this layer is a HRSC (High Resolution Stereo Camera) [2] assembled on an airplane (See Figure 2a) with sensing distance of 2~50 m. The second layer is at medium scale with sensing distance of 2~50 m and resolution of 0.5~10 cm. It combines the automatic matching, meshing, mesh optimization in one work flow to generate 3D models for the interior of buildings. The 3D point clouds are obtained with an advanced laser range scanner – Z+F Imager [3] (See Figure 2b). And the last layer is at the smallest scale. With a multi-sensory 3D-Modeler [4, 5] (See Figure 2c) developed by robotics institute of Germany Aerospace Center, it digitalizes the small objects in distance from 5~200 cm and resolution of 1~5 mm. 3D-modeling and visualization of these data is performed parallel to the scanning process. HRSC a) HRSC on Airplane b) Z+F Imager c) DLR 3D-Modeler Figure 2: Hardware at three layers The main contribution of this paper is the exploration in the fast and automatic modeling of the world systematically. This paper focuses on our new development of real-time modelling procedures at the first two layers, for the modelling at the third layer is already published in [6, 7]. Our paper is organized as follows. In section 2, we summarize the existing researches in this area. Then, section 3 describes our main methods at three layers. Their respective experimented models are shown in this section too. After that, the remained problems and possible improvements in the future are addressed in section 4. For the more, the related acknowledgement is announced in section 5. And the end section is referenced materials. 2. Related Work For the automatic modelling and visualisation of landscape data at the first layer, the principle of LOD (Level of Detail) [8] is widely used. To enhance the 3D roaming velocity, “Buffer- Quadtree” algorithm is introduced by Li [9]. Recently, more and more efforts have been taken to find and reconstruct the man-made and natural structures automatically. For example, buildings, highways, trees etc. [10, 11]. For the modelling of laser range images at middle scale, Allen [12] presented an automated 3D modelling pipeline, which consists of scan registration, surface generation and texture mapping. A set of markers that the scanner can automatically recognize were used to register the scans together. And for the surface generation, they used VripPack [13], which utilises a cumulative weighted signed distance function to integrate groups of aligned range images and employs a modified Marching Cubes algorithm with a lookup table to generate the final surface. And other notable projects, e.g. modelling of Michelangelo’s David [14] and IBM’s Pieta project [15], presented also modelling pipelines for large statures, but involved manual pre-aligning of different views to solve the coarse matching problems. Recently, many researchers reported their improvements for fully automatic registration of laser range images. Liu and Hirzinger [16] introduced “Matching Tree” structure to reach a global maximum matching based on local m:n corresponding relationships in a closed form with polynomial runtime. Rabbini and Heuvel [17] used constrained search for finding correspondent objects. Kang and Zlatanova [18] used corner detector to extract feature points from reflectivity images and by construction of a triangle mesh to reduce the search space. For the modeling of objects at small scale, some smart hand-guided device were developed, e.g. David [19] of TU Braunschweig. 3. Main Method In this section, we mainly focus on the real time 3D-modellingsprozess at the software aspect, which is performed at three layers: large, medium and small scales. The common character of them is: the input data are 2.5D or 3D point clouds with color information and the outputs are simplified colored mesh. The details of these three methods, the differences between them and the related experiments are showed below. 3.1. Large Scale The input data of this process are generated with the High Resolution Stereo Camera (HRSC) fixed on an airplane flying over the land and optimised by the photogrammetric pre- processing: correction, refining and stereo-matching. The HRSC camera was originally developed by the Planetary Research Institute of German Aerospace Center for the exploration of the Mars surface. The airborne version HRSC-AX is currently used for capturing earth’s landscape and cities from flight altitudes between 1500m to 5000 m. The input of this 3D-modellingsprocess is the 2.5 D point cloud resulted from the photogrammetric pre-processing: correction, refining and stereo-matching. The input data format is double-Tiff-format: one Tiff-file contains the height-information of every pixel, and the other holds the RGB colour-information. One of these input files is normally several Gigabytes, which overrides the capabilities of fast all of the 3D modelling and visualisation tools. To reduce the data amount and prepare for the online visualisation of the coloured landscape model, we use a real time and fully automatic 3D-modellingsprocess. It is consist of six steps: 1. Tiling with overlap: The whole area will be divided in tiles with a proper size. To avoid great distort out of merging of meshes, we set a overlap between neighboured tiles. 2. Mesh generation: A triangle mesh will be generated in each tile by connection the shorter diagonal of every rectangle grid. 3. Mesh simplifying: An efficient mesh-simplifying will be applied in this step. Geometric characters become more outstanding, for example, edge becomes sharper. And the data amount can be reduced over 90% without distinguish distortion. A modified quadric error metric algorithm is used to contract vertex-pairs with minimal contraction error iteratively [20]. 4. Mesh cut: Exactly along the middle axis of the overlapping area, every mesh will be cut neatly in this step. That means, the projection of all resulted boundary edges from one cut action should be on a straight line. 5. Mesh merger: Due to the 3D property of the meshes, there are tiny gaps between the neighboured meshes. To make the adjacent meshes seamless, we apply an efficient merger algorithm by taking the following “stitch-to-point” and “stitch to edge” actions. 6. Texture mapping: The RGB information of each tile will be saved as compressed image format. And the mapping from texture to 3D mesh will be achieved by projection the 3D meshes to the texture coordinate system. The result is a group of simplified meshes with corresponding textures for an efficient online visualisation. Monastery Andechs Hall of Liberation Figure 3: Reconstructed landscapes at the first Layer On a computer with Intel Core2 Duo Processor E6600 (2.4GHz) and 2048 MB RAM, experiments have been done for the reconstruction of diverse regions in Bavaria, Germany, e.g. city of Kelheim with Befreiungshalle ("Hall of Liberation"), Monastery Andechs, Neuschwanstein Castle etc. The processing time is showed in Table 1. Area Size Resolution Input Size Output Runtime Region (km2) (m) n x m (pixels2) (Triangles) (min) Kehlheim 11.33 0.20 24480 x 11570 3,623,330 26,5 Andechs 0.81 0.15 6000 x 6000 463,333 3,6 Wiekirche 1.44 0.20 6000 x 6000 465,557 3,7 Table 1: Processing time of diverse regions at the first layer 3.2. Medium Scale On this layer, we use Z+F Imager to get laser range scans inside the building. The sensing distance is from 2m to 50 m with resolution from 5mm to 10 cm.