TOPOGRAPHY-PRESERVING, NON-LINEAR INPAINTING FOR AUTONOMOUS BARE EARTH DIGITAL ELEVATION MODEL (DEM) RECONSTRUCTION

Josef DeVaughn Allen, Software Engineer Anthony O’Neil Smith, Software Engineer Mark Rahmes, Software Engineer Harris Corporation Government Communications Systems Division Melbourne, Florida 32905 [email protected] [email protected] [email protected]

ABSTRACT

Harris describes a novel way to autonomously inpaint missing data into high resolution single reflective surfaces utilizing a variant of the Navier-Stokes equations. One product of this process is a high-resolution bare earth Digital Elevation Model (DEM) with the same resolution as the input data. Inpainting allows generation of high resolution bare earth DEMs in both high and low frequency terrain environments for urban 3-D modeling. Having this bare earth DEM accounts for a dramatic increase in accuracy in all other steps of the urban 3-D modeling process. The LiteSite™ toolkit has the capability to automatically extract buildings and vegetation from an urban scene. The resulting DEM from this step of the process acts as input to the inpainting process. The expected building and vegetation base heights can then be inpainted into the area of extraction where data is now missing. The inpainting process maintains building and vegetation base height consistency in the inpainted regions and performs high accuracy interpolation and edge propagation on DEMs. Inpainting is highly effective for data sets where the occurrence of missing data is common and undesired. Examples of such data include Shuttle Radar Topography Mission (SRTM), LIDAR, IFSAR, correlated DEMs from imagery, or any other single reflectance collection data set. This technology preserves height contours. A more accurate bare earth product allows for better automated building vector extraction and therefore reduces manual building vector editing. Automated texturing of 3D model products with aerial or satellite imagery is accomplished using Harris’ RealSiteTM Toolkit.

Keywords: Navier-Stokes’, partial differential equation, anisotropic diffusion, Inpainting

INTRODUCTION

This paper describes a novel way of automatically inpainting missing data into high resolution single reflective surfaces utilizing a variant of the heat and/or Navier-Stokes’ equations. More specifically, a general methodology is presented to fill in variable sized voids of a high resolution Digital Elevation Model (DEM) and LIDAR derived 3D site models. Evaluation and results of sample models are provided. The goal of this paper is to introduce an algorithm that jointly performs high accuracy interpolation and contour preservation around extracted features on DEMs. While inpainting has commonly been a method of interpolation applied to images we are unaware of any other work proposing to apply this idea to DEMs (Verdera et al. 2003). Inpainting is most commonly used as an image restoration technique whereby missing data is flowed in from the boundaries of the manually identified region where filling is desired in the image. The technique presented in this paper autonomously identifies and restores voided elevation data. Ideally we would like to replenish removed or missing culture from the input data. Harris Corporation’s current LiteSite™ algorithm can be run on input DEMs with data missing from collection and processing, but can also perform automated culture extraction and filling of the voided regions resulting from this extraction. In the following sections more explanation of the algorithm will be provided along with results from its application.

MAPPS/ASPRS 2006 Fall Conference November 6 – 10, 2006 * San Antonio, Texas CURRENT STATE

In the past many techniques have been used to perform void filling on digital elevation models (DEMs). Most interpolation techniques can leave visual artifacts in the center of the filled region and tend to blur edge contours. Preserving edge contours in DEMs is absolutely vital to generating accurate topography. The sinc function, also known as the “sampling function,” assumes the signal is band-limited; while this is ideal for communications and audio signals, it may not be the case in this paradigm. Polynomial interpolation techniques perform well in smooth regions but are computationally expensive. When using this type of technique boundary conditions can be tricky to handle and may not be exact. There is also a trade-off between the order of the polynomial and the data fitting accuracy. Splines are very popular and achieve higher reconstruction accuracy than polynomial techniques. The major draw back seen when attempting to fit this method to the data is that it is difficult to accurately solve a global spline over the entire DEM. This seems to point to a need for a method that can accurately propagate edge content without producing visual artifacts in a way that is flexible to the DEM paradigm while being relatively computationally inexpensive. We believe the algorithm described in this paper is extremely well-suited for this purpose.

INPAINTING METHODOLOGY

Inpainting is the process of filling in part of an image or video using information from the surrounding area. We extend the canonical paper by M. Bertalmio et al. (2000) such that it autonomously detects and fills in variable sized holes in Digital Elevation Models (DEM); it uses a variant of the heat equation to propagate the information from the boundary, ∂Ω, into the voided area Ω. Moreover, the numerical partial differential equation for a Digital Elevation Models will yield the non-linear solution:

Η n+1 = Η n + Δ Η n ∀ ∈ Ω (i, j) (i, j) t t (i, j), (i, j) (Equation 1)

Here Hn+1 represents the new updated DEM at iteration n+1, where n denotes the iteration. Current height, Hn, represents the DEM at iteration n. The crux of the technology centers on update to change in current DEM height, n Δ H t . The t represents the rate of improvement allowed per iteration. The improved image is given by:

⎛ ⎞ n = ⎜∂ n • N(i, j,n) ⎟ ∇ n Ht (i, j) ⎜ L (i, j) ⎟ | H (i, j) | (Equation 2) ⎝ | N(i, j,n) | ⎠

with H = H , where H is the initial DEM. Here the rate of change of the Laplacian, L, is propagated in the |∂Ω o o direction of minimum change. Visual representation is shown in Figure 1.

MAPPS/ASPRS 2006 Fall Conference November 6 – 10, 2006 * San Antonio, Texas Propagate height information from outside inpainting region ? along direction of iso-contour (lines of constant contour height values). ∂H Isophote direction = ∇L • N ∂t Obtain iso-contour direction, N, by taking 90o rotation of DEM gradient. ∇ L is discrete Laplacian. Full inpainting equation: nn+1 =+Δ∀∈Ω n Gradient direction H(, ij ) H (, ij ) tHijt (, ), (, ij )

Figure 1. Inpainting Propagation of Height Information. Another partial differential equation was presented by Bertalmıo et al (2001). A relationship between the aforementioned approach and fluid dynamics of incompressible fluids is found. This novel solution was similar to the derived equation for 2-D vorticity stream equation. The 2-D vorticity stream function is the derived equation is calculated by taking the cross product of the primitive Navier-Stokes equation:

G G ⎛ ∂v G G G⎞ ∂ω G G G ∇ × ⎜ + ()v • ∇ v = −∇p +υ∇ 2v ⎟ = + ()v • ∇ ω = υ∇ 2ω (Equation 3) ⎝ ∂t ⎠ ∂t

G G 2 with ∇2H = ω H = H , where H is the initial DEM andω = ∇ × v = ∇ Ψ . The stream function |∂Ω o o Ψ represents the DEM heights. Taking the perpendicular at the gradient to Ψ gives the minimum rate of change of ⊥ G the heights (i.e. ∇ Ψ = v ). We propagate this minimum rate of change via the non-linear advection term, G G ()v • ∇ ω . Observe that the advection term propagates the Laplacian of the heights. Equations 1& 2 are variants of the geometric heat equation whereas equation 3 is the derived equation of the primitive equation for momentum for fluids. In the LiteSite™ tool we take advantage of both basic methodologies and extend these ideas to autonomously detect and inpaint variable sized holes in a DEM. We also leverage characteristics of the DEMs and LIDAR in our implementation of the two transport equations. Furthermore, anisotropic diffusion is added to both inpainting algorithms such that the data becomes more continuous. The anisotropic diffusion equation is taken from Perona and Malik (1990):

MAPPS/ASPRS 2006 Fall Conference November 6 – 10, 2006 * San Antonio, Texas ∂I = div(g(| ∇I |)∇I) (Equation 4) ∂t

We have chosen to use our own various anisotropic diffusers that align with the data. The main products from these technologies are bare earth and occlusion removal for 3-D urban scenes. LiteSite™ toolkit current algorithms perform autonomous culture and vegetation extraction. These features are autonomously detected and voids are created and filled in the single reflective surface input DEM in place.

LITESITE PROCESS

The LiteSite™ algorithm currently reads a list of 3-D LIDAR or IFSAR points (usually several million). A competitive filter is used to take an unordered list of LIDAR points and generate an equally spaced Digital Elevation Model (DEM) at a given resolution. Alternatively, optical imagery may be correlated to create the DEM as shown in Figure 2. The buildings and trees are then extracted to a separate DEM from the ground and can then even be separated from each other. The remaining ground data then has all of its resulting missing data filled using our inpainting algorithm to complete a full bare earth DEM with edge contours propagated. Filtering is then performed to remove any noisy LIDAR returns. A line following algorithm followed by a multi-stage generalization algorithm applies 3-D chord points, to be used as polygon vertices, to the rooftop features. The sides of the buildings are extruded straight down to the void filled bare earth from the rooftops. The accuracy of this stage of the process is highly dependent on the accurate void filling of the inpainting algorithm. The vertices from the building roofs and sides as well as the ground surface are mapped into polygons and projected into geo-spatial coordinates. Textures may then be automatically applied to their corresponding polygons.

Figure 2. Lidar and Commercial Imagery Exploitation.

MAPPS/ASPRS 2006 Fall Conference November 6 – 10, 2006 * San Antonio, Texas AUTONOMOUS INPAINTING FOR BARE EARTH

The objective is to autonomously reconstruct the bare earth in places where buildings and trees have been removed or where data is missing from the collection and pre-processing stages while keeping continuous height contours. Inpainting accurately propagates information from extracted building boundaries as shown in Figures 3 through 6. The input DEM is iteratively evolved until a steady state is achieved. We can control the speed of propagation. However there is a tradeoff between accuracy and the speed depending on data resolution and number of iterations. This allows our technique to generate high resolution bare earth DEMs from high frequency terrain.

Figure 3. Input DEM Figure 4. 500 Inpaint Iterations

Figure 5. 900 Inpaint Iterations Figure 6. 4000 Inpaint Iterations

MAPPS/ASPRS 2006 Fall Conference November 6 – 10, 2006 * San Antonio, Texas The process is initiated by identifying the areas of interest where cultural and vegetation features have been extracted. It is important to maintain height contours in the interior of identified inpainted regions. The DEM is treated as an incompressible fluid. Figures 7 through 12 show key steps in LiteSite™ process. Figure 10 shows the bare earth product with inpainting. In the next section, beneficial impact of filling voids with inpainting vs. median filling is discussed.

Figure 7. Input DEMFigure 8. Culture and Figure 9. Bare Earth with Vegetation Removed Median Filling

Figure 10. Bare Earth Figure 11. Culture and Figure 12. 3D site Model with Inpainting Vegetation to be Modeled

The demonstration data set shown in Figures 7 through 12 is a suburban area of Louisville, KY; with area of 2.3 square kilometers and 1m post spacing LIDAR points. Source data provided courtesy of Photo Science for purposes of illustrating Harris automated processing results for automated terrain, building, and vegetation extraction.

EXPERIMENTAL RESULTS

Over several data sets experimental results Table 1. Improved on RMSE for Extracted Buildings showing improvement with inpainting for Root with Inpainting vs. Median Fill Mean Squared Error (RMSE) of automatically RMSE Triangle Count Editing Cost generated buildings over actual height data and overall triangle count reduction are shown in Average Improvement 15.1% 12.3% 31.9% Table 1. Additionally, one of the key advantages of obtaining a more accurate bare earth DEM is a significant cost savings in manual editing time to improve the visual quality of buildings. A method for evaluating an automated 3D site model extracted from LIDAR is discussed by Yates and Rahmes (2006). Using the LiteSite™ toolkit, the user has the option of choosing to automatically generate a fine or coarse model. Modeling toolkits typically have an operating quality curve as shown in Figure 13 for given site model. The closer the operating quality curve is to the bottom and left of chart, the better the automatic model quality. A higher

MAPPS/ASPRS 2006 Fall Conference November 6 – 10, 2006 * San Antonio, Texas quality auto-generated model requires less manual editing and therefore costs are reduced. The operating quality curve with inpainting, represented by the green line, results in an editing cost savings of 31.9% as compared with the operating quality curve with median filling, represented by the blue line. This cost savings visually corresponds to a shorter dotted line between “selected” and “edited” end points, while achieving same model RMSE quality. The minimum number of triangles for a simple block building is ten. Since the example model had 8941 buildings, a horizontal solid line is drawn at this value across the chart. For a given model, there are a reasonable minimum number of triangles. A desired goal of 1m building RMSE given a source data post spacing of 1m, as shown in the chart, is represented by a vertical dotted line. The green area shows the desired measured quality of the 3D model.

More Fine Quality Detail Curve with Quality Median Filling Curve with Inpainting

Selected Edited

Editing line for Inpainting More is 31.9% shorter compared Coarse with Median filling Generalization 8941 Buildings * 10T/bldg (minimum)

LIDAR Source Post Spacing (m) Figure 13. Building RMSE vs. Triangle Count

MAPPS/ASPRS 2006 Fall Conference November 6 – 10, 2006 * San Antonio, Texas EXAMPLES

The data set being demonstrated in Figures 14 through 17 is a suburban area of Memphis, TN. with area of 36 square kilometers and 1m post spacing LIDAR points. Source data provided courtesy of the following organizations for purposes of illustrating Harris automated processing results: 3001, Inc. LIDAR data set used for automated terrain, building, and vegetation extraction; and Digital Globe Satellite imagery pan sharpened and used for automated building and ground texturing.

Figure 14. LIDAR DEM Figure 15. Bare Earth Figure 16. Culture and Vegetation

Figure 17. Site Model Examples

CONCLUSIONS

This paper presented a novel way to leverage a variant of the heat equation and fluid mechanic techniques in partial differential equations to DEMs for void filling where the DEMs have been treated as an incompressible fluid. The impact of improved bare earth products with inpainting on 3D site models on RMSE and reduced triangle count has been shown. Inpainting maintains building and vegetation base height consistency of the filled regions and performs high accuracy interpolation and edge propagation on DEMs. This method can be applied on many different types of data sets, some of which are listed in the abstract, and for many different reasons including, but are no limited to, filling in for bare earth following automatic extraction of culture and vegetation and filling in missing data produced by the data collection.

ACKNOWLEDGEMENTS

We thank Dr. Gnana Bhaskar Tenali of Florida Institute of Technology. Dr. Tenali has been invaluable as a technical consultant. Additionally, we would like to thank several engineers from Harris Corporation: Adrian Peter, Harlan Yates, Patrick Kelley, Dr. Emile Ganthier, Dr. Eric Spellman, and Dr. Douglas Carlson for their tireless input.

MAPPS/ASPRS 2006 Fall Conference November 6 – 10, 2006 * San Antonio, Texas REFERENCES

Bertalmıo, M., Bertozzi, A.L., Sapiro, G., “Navier-Stokes’ (2001) Fluid Dynamics, Image and Video Inpainting” Proceedings of the International Conference on Computer Vision and Pattern Recognition, IEEE, Dec. 2001, Kauai, HI volume I, pp. I-355-I362. Bertalmio, M., Sapiro, G., Ballester, C., and Casellas, V. (2000) “Image Inpainting” Computer Graphics, SIGGRAPH 2000, pp. 414-424, July 2000. Perona, P., Malik, J., (1990) “Scale-Space and Edge Detection Using Anisotropic Diffusion”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 12, No. 7 July 1990. Verdera, J., Caselles, V., Bertalmio, M., and Sapiro, G., (2003) “Inpainting surface holes,” IEEE International Conference on Image Processing, ICIP 2003, Barcelona, Spain, pp. 903-906, Sept. 2003. Yates, J. H., Rahmes, M. (2006) “Evaluation of Automated LIDAR 3D Model Generation”, International Lidar Mapping Forum, Feb 2006.

MAPPS/ASPRS 2006 Fall Conference November 6 – 10, 2006 * San Antonio, Texas