Eurographics Symposium on Geometry Processing (2005) M. Desbrun, H. Pottmann (Editors) Example-Based 3D Scan Completion Mark Pauly1,2 Niloy J. Mitra1 Joachim Giesen2 Markus Gross2 Leonidas J. Guibas1 1 Computer Graphics Laboratory, Stanford University, Stanford CA 94305, USA 2 Department of Computer Science, ETH Zurich, CH-8092 Zurich, Switzerland Abstract We present a novel approach for obtaining a complete and consistent 3D model representation from incomplete surface scans, using a database of 3D shapes to provide geometric priors for regions of missing data. Our method retrieves suitable context models from the database, warps the retrieved models to conform with the input data, and consistently blends the warped models to obtain the final consolidated 3D shape. We define a shape matching penalty function and corresponding optimization scheme for computing the non-rigid alignment of the context models with the input data. This allows a quantitative evaluation and comparison of the quality of the shape extrapolation provided by each model. Our algorithms are explicitly designed to accommodate uncertain data and can thus be applied directly to raw scanner output. We show on a variety of real data sets how consistent models can be obtained from highly incomplete input. The information gained during the shape completion process can be utilized for future scans, thus continuously simplifying the creation of complex 3D models. 1. Introduction in the perceived object. In digital 3D shape acquisition, we are faced with a similar problem: Most optical acquisition 3D shape acquisition has become a major source for the devices will produce incomplete and noisy data due to oc- generation of complex digital 3D models. Numerous scan- clusions and physical limitations of the scanner. How can ning systems have been developed in recent years, includ- we obtain a complete and consistent representation of the ing low-cost optical scanners that allow 3D data acquisi- 3D shape from this acquired data? One approach is to apply tion on a large scale. Obtaining a complete and consistent low-level geometric operations, such as noise and outlier re- 3D model representation from acquired surface samples is moval filters [Tau95], [JDD03], [FDCO03], [WPH∗04]and still a tedious process, however, that can easily take mul- hole-filling techniques based on smooth extrapolations, e.g., tiple hours, even for an experienced user. Significant man- [DMGL02], [VCBS03], [Lie03], [CDD∗04]. These methods ual assistance is often required for tasks such as scan path are successful in repairing small deficiencies in the data, but planning, data cleaning, hole filling, alignment, and model have difficulties with complex holes or when large parts of extraction. Knowledge about the acquired shape gained in the object are missing. In such cases, trying to infer the cor- this process is typically not utilized in subsequent scans, rect shape by only looking at the acquired sample points where the same time consuming procedure has to be re- quickly becomes infeasible. A common way to address this peated all over again. Our goal is to simplify the model cre- ill-posed problem is to use an explicit prior in the form of a ation process by exploiting previous experience on shapes carefully designed template model. The prior is aligned with stored in a 3D model database. This allows the generation the acquired data and holes are filled by transferring geo- of clean and complete 3D shape models even from highly metric information from the warped template. The cost of incomplete scan data, reducing the complexity of the ac- designing the template model is quickly amortized when a quisition process significantly. The main idea is to mimic whole set of similar objects is digitized, as has been demon- the experienced-based human approach to shape perception strated successfully with human heads [BV99], [KHYS02], and understanding. Humans have the intuitive capability to and bodies [ACP03]. We extend this idea to arbitrary shapes quickly grasp a perceived 3D shape, even though only a frac- by replacing a single, tailor-made template model with an tion of the actual geometric data is available to the eye. This entire database of 3D objects. This allows shape completion is possible because we make extensive use of prior knowl- by combining geometric information from different context edge about shapes, acquired over years of experience. When models. To successfully implement such a system, we need seeing an object, we immediately put it into context with to address the following issues: How can we extract models other similar shapes that we have previously observed and from the database that provide a meaningful shape continu- transfer information from those shapes to fill missing parts c The Eurographics Association 2005. M. Pauly et al. / Example-Based 3D Scan Completion Data Non-rigid Blending Classification Alignment raw input scored aligned consolidated point cloud point cloud models model 3D Model Database Segmentation Evaluation Acquisition Retrieval physical model candidate segmented model database models models Figure 1: High-level overview of our context-based shape completion pipeline. ation in regions of missing data? How can we compute and a system that preserves high-frequency detail by replicating consistently evaluate local shape deformations that align the local patches within the acquired 3D data set, as an extension context models with the acquired data? How can we select of the 2D method proposed in [DCOY03]. Other approaches among multiple database models the ones that provide the for shape completion are the triangulation-based method most adequate shape completion in different regions of the proposed by [Lie03] and the system based on finite elements scan? And finally, how can we blend geometric information presented in [CDD∗04]. The underlying assumption in all from different models to obtain a complete and consistent of these methods is that an appropriate shape continuation representation of the acquired object? can be inferred from the acquired sample points only, us- ing generic smoothness or self-similarity priors for missing Contributions. We define a shape similarity metric and parts of the model. Since the surface reconstruction prob- corresponding non-rigid alignment method that allows to lem is inherently ill-posed, the use of explicit template pri- consistently evaluate the quality of fit of different context ors has been proposed by various authors. [RA99]presented models. We present an automatic segmentation method that a method to recognize and fit a parametric spline surface to locally selects the best matching geometry, and a blending acquired surface data. Template-based hole filling has also scheme that allows to combine contributions from different been used in [BV99], [KHYS02], [BMVS04]and[ACP03], models, while preserving appropriate continuity constraints. where input data and morphable template model were rep- Our algorithms are designed to work with uncertain data and resented as triangle meshes. These methods are well-suited can thus be applied directly to highly incomplete raw scan- for object classes with well-defined shape variability, where ner output. We show how these methods can be integrated to a single template model can be adjusted to fit the entire ac- yield a complete context-based shape completion pipeline. quired data set. Our approach differs in that we are not as- suming a priori knowledge of the underlying shape space, 1.1. Related Work but try to infer automatically how to combine different con- Surface reconstruction from point samples has become an text models that are retrieved from a database for the specific active area of research over the last few years. Early methods input data produced by the scan. A central component of based on signed distance field interpolation have been pre- our system is a method for computing non-rigid alignments ∗ sented by [HDD 92]and[CL96]. Voronoi-based approaches of database models with the acquired input data. [ACP03] with provable guarantees where introduced by [ABK98]and and [SP04] presented alignment algorithms similar to ours [DG03], who extended their work in [DG04] to handle noisy that use an optimization framework to compute a smooth input data. While these techniques can handle small holes warping function. We extend this scheme to allow a quan- and moderately under-sampled regions in the input data, they titative comparison of the quality of the alignment across are less suited for data sets with large holes that often oc- different models, which has not been a concern in previous cur in 3D data acquisition. To address this problem, vari- methods. An interesting alternative has been proposed by ous methods for model repair based on smooth extrapolation [ASK∗04], who introduced a probabilistic scheme for unsu- ∗ have been proposed. [CBC 01] use radial basis functions pervised registration of non-rigid shapes. Our system bears to extract manifold surfaces from incomplete point clouds. some resemblance to the shape modeling system proposed [DMGL02] presented an algorithm that applies volumetric by Funkhouser et al. [FKS∗04], where the user can create diffusion to a signed distance field representation to effec- new models by cutting and pasting rigid parts of existing tively handle topologically complex holes. An extension of shapes retrieved from a shape database. The focus of their this idea has been proposed by [VCBS03], who use partial work is on creative design and interaction, while we concen- differential equations to evolve the distance field, similar to trate on model repair and shape completion of surface scans. inpainting techniques used for images. [SACO04]presented c The Eurographics Association 2005. M. Pauly et al. / Example-Based 3D Scan Completion high λ c i * c i σ c i low λ Figure 2: Left: Acquisition setup with Cyberware Desktop Figure 3: Quality of fit estimate ci and local uniformity es- σ 3D Scanner 15 and physical model, right: Raw point cloud timate ci are combined to yield the final confidence ci.