Compression and Rendering of Textured Point Clouds Via Sparse Coding
Total Page:16
File Type:pdf, Size:1020Kb
High-Performance Graphics (2021) N. Binder and T. Ritschel (Editors) Compression and Rendering of Textured Point Clouds via Sparse Coding Kersten Schuster, Philip Trettner, Patric Schmitz, Julian Schakib, Leif Kobbelt Visual Computing Institute, RWTH Aachen University, Germany colored splats 1 atom per splat 0–8 atoms per splat Figure 1: Our proposed method compresses textured point clouds via sparse coding. We optimize a dictionary of atoms (each 32×32 pixels) and represent each splat texture as an average color plus a weighted sum of these atoms. With colored splats only, all high frequency content is lost (left). When adding a single atom, the quality is already improved significantly (center). Our approach can adapt the number of atoms per splat depending on texture content and the desired target quality. Abstract Splat-based rendering techniques produce highly realistic renderings from 3D scan data without prior mesh generation. Map- ping high-resolution photographs to the splat primitives enables detailed reproduction of surface appearance. However, in many cases these massive datasets do not fit into GPU memory. In this paper, we present a compression and rendering method that is designed for large textured point cloud datasets. Our goal is to achieve compression ratios that outperform generic texture compression algorithms, while still retaining the ability to efficiently render without prior decompression. To achieve this, we resample the input textures by projecting them onto the splats and create a fixed-size representation that can be approximated by a sparse dictionary coding scheme. Each splat has a variable number of codeword indices and associated weights, which define the final texture as a linear combination during rendering. For further reduction of the memory footprint, we compress geomet- ric attributes by careful clustering and quantization of local neighborhoods. Our approach reduces the memory requirements of textured point clouds by one order of magnitude, while retaining the possibility to efficiently render the compressed data. CCS Concepts • Computing methodologies → Point-based models; Texturing; Image compression; © 2021 The Author(s) Eurographics Proceedings © 2021 The Eurographics Association. K. Schuster, P. Trettner, P. Schmitz, J. Schakib, L. Kobbelt / Compression and Rendering of Textured Point Clouds via Sparse Coding 1. Introduction propose a compression scheme for the geometric attributes. Splat positions are clustered and encoded as local coordinates, and then Digital 3D reconstructions capture the geometry and appearance of quantized together with normals and cliplines into a representation real-world objects and environments based on non-invasive imag- that is suitable for uploading onto the GPU. ing technology. They enable faithful replications of locations and buildings, which can be explored interactively everywhere and at The resulting method permits random memory accesses for the any time. Such reconstructions are increasingly important in areas compressed texture and geometric data, which is crucial for effi- such as cultural heritage preservation, education in history and ar- cient evaluation during rendering. Our approach can furthermore chitecture, or virtual touring and telepresence applications. be adapted to different memory and GPU performance budgets, by choosing a variable dictionary size and number of atoms per High-precision terrestrial laser range scanners, camera-based splat. We achieve compression ratios that outperform generic tex- depth sensors, or image-based 3D reconstruction by structure- ture compression algorithms, while not suffering from visible block from-motion offer a number of mature technologies to capture real artifacts. We achieve PSNR scores comparable to input textures that environments with ever-increasing resolution and detail. The result- have been downsampled once, but the perceived visual quality is ing datasets typically contain massive amounts of point cloud data superior at a much lower memory footprint. With our method, we combined with a large number of photographs. For high-precision are able to compress a dataset of 71 million splats, textured with laser range scanners, a reconstructed scene can easily consist of 441 high-resolution photographs, to about 1 GB of GPU memory. hundreds of high-resolution images and billions of 3D points. Our contributions to the efficient rendering of large textured One way to visualize such datasets is to render the point cloud point clouds can be summarized as follows: data directly, without intermediate surface mesh extraction. For that, points are expanded to small disks, so-called splats, within • Compression of splat textures via sparse dictionary encoding their local tangent plane. This is sufficient for visual reproduc- • Compression of splat geometry via clustering and quantization tion, but requires less manual preprocessing work. Automatically • Efficient direct rendering of the compressed per-splat data generating topologically accurate meshes from 3D scan data is of- ten problematic due to insufficient sampling density [KBH06], and 2. Related Work does not improve the visual quality. Point-based rendering tech- niques don’t suffer from this limitation and can yield highly real- 2.1. Point Cloud Compression istic renderings. High-quality splat rendering techniques use pho- Captured point clouds tend to be quite memory-intensive and thus, tographs for perspectively correct texturing, and perform shading their compression is well-researched. based on a smoothly varying normal field for precise surface ap- proximation [BHZK05]. Several existing methods use octrees to efficiently encode po- sitions, and simultaneously provide culling and level-of-detail While high-quality reproductions are possible this way, stor- [SK06]. De-duplicating similar subtrees leads to sparse voxel ing many high-resolution photographs in GPU memory for tex- DAGs [KSA13; DKB*16; VMG17]. Instead of the uniform sub- ture mapping quickly becomes a limiting factor. Existing meth- division of octrees, DE QUEIROZ and CHOU use a k-d tree and ods solve the problem by page-based streaming of splat textures predictive coding to better adapt to the data [dC16]. FAN et al. [SBMK20]. However, for their vast amount of data (1.2 terabyte use hierarchical clustering for compression and level-of-detail and on their dataset), they require high-end systems with large storage achieve less than one bit per normal-equipped point at good quality bandwidths and cause a continuously high CPU load for on-the-fly [FHP13]. decompression. This can be prohibitive for many potential applica- tions and target systems. Traditional image compression schemes can also sometimes be employed. One main source of point clouds is LiDAR sensor data, Our goal in this paper is to enable high-quality reproduction of which is effectively a cylindrical heightmap and can thus directly textured point cloud data without the bandwidth and CPU overhead benefit from traditional image compression [CTMT16]. In a more induced by streaming approaches. Instead, we devise an efficient general setting, HOUSHIAR and NÜCHTER create panorama im- compression scheme that drastically reduces the memory footprint ages of the point cloud and compress those [HN15]. Strong com- of such datasets. This enables high-quality textured point cloud ren- pression is especially required when continuous streams of point dering on a wider range of target systems, and removes the contin- clouds are captured. THANOU et al. use a graph-based compression uous system load to enable new types of applications. approach that is suited for scans of moving people [TCF16]. TU et To this end, we propose a texture compression method that is tar- al. focus on scenarios where the scanner itself moves and employ geted specifically at textured splat rendering. By using fixed-size SLAM based predictions for higher compression [TTMT17]. snippets of input textures projected onto each splat as the basis The strong predictive capabilities of neural networks can be used of our representation, we leverage the advantages of block-based for compression as well. TU et al. compress LiDAR data using re- compression without introducing the typical compression artifacts. current networks [TTCT19], and WANG et al. use variational au- As neighboring splats are blended like described in [BHZK05], po- toencoders for scanned objects [WZM*19]. tentially visible block boundaries are concealed. A sparse coding technique finds an optimized representation of the texture content, For rendering, the point clouds must fit into GPU memory and that is reconstructed during rendering by weighted blending of a are decoded on-the-fly. From the previous works, only the sparse small number of atoms from a learned dictionary. We furthermore voxel DAGs are suited for direct rendering. Other approaches that © 2021 The Author(s) Eurographics Proceedings © 2021 The Eurographics Association. K. Schuster, P. Trettner, P. Schmitz, J. Schakib, L. Kobbelt / Compression and Rendering of Textured Point Clouds via Sparse Coding support this try to find regions of the point cloud that can be approx- images divided into fixed-sized square patches. They show that k- imated by a regular heightmap. These heightmaps can be stored and SVD performs significantly better in terms of the root-mean-square rendered as progressive quadtrees, the individual entries encoded error (RMSE) for a given memory budget. via vector quantization [SMK07; SMK08]. Due to its importance, there have been several