Graph-Based Point Cloud Compression Electrical And
Total Page:16
File Type:pdf, Size:1020Kb
Graph-based Point Cloud Compression Paulo Jorge Robert de Oliveira Rente Thesis to obtain the Master of Science Degree in Electrical and Computer Engineering Supervisors: Prof. Fernando Manuel Bernardo Pereira Prof. João Miguel Duarte Ascenso Prof. Catarina Isabel Carvalheiro Brites Examination Committee: Chairperson: Prof. José Eduardo Charters Ribeiro da Cunha Sanguino Supervisor: Prof. Fernando Manuel Bernardo Pereira Member of the Committee: Prof. Sérgio Manuel Maciel Faria November, 2017 ii Acknowledgments First of all, I would like to thank the closest persons in my life: my family and girlfriend; for their endless support, patient and motivation I am eternally grateful. Without them, it would be impossible to achieve this academic degree. I would also like to show my deepest gratitude to my supervisors: Professor Fernando Pereira; Professor João Ascenso; and Professor Catarina Brites. Their orientation, availability and support throughout this process were a key component for its success. Last but not least, I would like to thank all my friends. iii iv Resumo Os modelos tri-dimensionais (3D) de representação visual, como light fields e point clouds, estão a tornar-se cada vez mais populares devido à sua capacidade para representar o mundo real de uma forma mais completa e imersiva. Neste sentido, os modelos 3D abrem caminho para cenários aplicacionais novos e mais avançados. O modelo point cloud permite representar eficazmente a superfície de objetos/cenas através de pontos 3D com atributos associados. Tendo em conta as suas características, é já bastante utilizado em diferentes contextos, desde veículos autónomos a realidade aumentada. Recentemente, novos sensores de imagem vieram facilitar a aquisição de point clouds mais ricas e densas, constituídas por milhões de pontos. A enorme quantidade de dados associada a este tipo de informação torna o seu armazenamento e transmissão particularmente difíceis. Assim sendo, é necessário desenvolver soluções de codificação eficientes, que possibilitem experiências de multimédia imersivas e ofereçam uma maior qualidade de experiência ao utilizador. O objetivo desta Dissertação é o desenho, implementação e avaliação de uma solução de compressão de informação geométrica de point clouds estáticas. Deste modo, este trabalho começa por introduzir alguns conceitos base, aplicações e estruturas gerais acerca de point clouds e revê soluções de codificação para point clouds disponíveis na literatura. A solução proposta usa um método baseado em octrees na camada base e a transformada de grafos na camada superior, onde um resíduo entre camadas é codificado. A análise de resultados mostra um aumento de desempenho comparativamente com o estado da arte, especialmente para baixos e médios débitos. Palavras Chave: aquisição de point clouds, compressão de point clouds, rendering de point clouds, análise de point clouds, transformada de grafos. v vi Abstract Recently, three-dimensional (3D) visual representation models such as light fields and point clouds are becoming popular due to their capability to represent the real world in a more complete and immersive way, paving the road for new and more advanced application scenarios. The point cloud representation model is able to efficiently represent the surface of objects/scenes by means of 3D points and associated attributes and is already widely used, from autonomous cars to augmented reality. Emerging advanced imaging sensors have made easier to perform richer and denser point cloud acquisition, notably with millions of points. However, the massive amount of data associated with this type of information makes its storage and transmission rather difficult and in some cases even impossible. Hence, there is a strong demand to develop efficient point cloud coding solutions which allow more powerful multimedia experiences and offer better quality of experience (QoE) to the user. The objective of this Thesis is the design, implementation and assessment of a solution that compresses the geometry information of static point clouds. Considering this context, this Thesis introduces some basic concepts, applications and overall frameworks about the point cloud representation model and reviews relevant point cloud coding solutions available in the literature. The proposed coding solution uses an octree-based approach for a base layer and a graph-based transform approach for an enhancement layer where an Inter-layer residual is coded. The performance assessment shows very significant compression gains regarding the state-of-the- art, especially for the most relevant lower and medium rates. Keywords: point cloud acquisition, point cloud compression, point cloud rendering, point cloud assessment, graph transform. vii viii Table of Contents Acknowledgments ......................................................................................................................... iii Resumo ......................................................................................................................................... v Abstract ........................................................................................................................................ vii Table of Contents .......................................................................................................................... ix List of Figures ................................................................................................................................ xi List of Tables ................................................................................................................................ xv List of Acronyms ......................................................................................................................... xvii Chapter 1 ....................................................................................................................................... 1 1. Introduction ................................................................................................................................ 1 1.1. Context and Motivation ....................................................................................................... 1 1.2. Objectives ........................................................................................................................... 3 1.3. Thesis Structure ................................................................................................................. 4 Chapter 2 ....................................................................................................................................... 5 2. State-of-the-Art on 3D Visual Representation and Main Point Cloud Coding Solutions ........... 5 2.1. Basic Concepts on 3D Representation .............................................................................. 5 2.2. Applications and Requirements for 3D Representations Models....................................... 9 2.3. Point Cloud System Architecture and Walkthrough ......................................................... 12 2.4. Point Cloud Acquisition Systems ..................................................................................... 14 2.4.1. Indirect Acquisition .................................................................................................... 14 2.4.2. Point Cloud Direct Acquisition ................................................................................... 15 2.4.3. Interesting Acquisition System Example ................................................................... 17 2.5. Rendering of Point Clouds ............................................................................................... 18 2.6. Point Cloud Objective Quality Metrics .............................................................................. 22 2.7. Point Cloud Coding: Reviewing Main Solutions ............................................................... 27 2.7.1. Octree-based Coding: the Point Cloud Library Coding Solution ............................... 29 2.7.2. Point Cloud Attribute Compression with Graph Transform ....................................... 34 2.7.3. Graph Transform Attribute Compression for Sparse Point Clouds ........................... 39 2.7.4. Patch-based Attribute and Geometry Compression ................................................. 42 Chapter 3 ..................................................................................................................................... 47 3. Proposing a Graph Transform-Based Point Cloud Geometry Compression Solution ............ 47 3.1. Architecture and Walkthrough .......................................................................................... 47 3.2. Main Tools Detailed Description ...................................................................................... 50 3.2.1. BL Octree Creation and Coding ................................................................................ 50 3.2.2. BL Upsampling .......................................................................................................... 51 ix 3.2.3. EL Clustering ............................................................................................................. 53 3.2.4. U-BL Clusters Creation ............................................................................................. 55 3.2.5. Inter-layer Prediction ................................................................................................. 57 3.2.6. Geometry Residuals Computation ...........................................................................