Adaptive Plane Projection for Video-Based Point Cloud Coding
Total Page:16
File Type:pdf, Size:1020Kb
Adaptive Plane Projection for Video-based Point Cloud Coding Eurico Manuel Rodrigues Lopes Thesis to obtain the Master of Science Degree in Electrical and Computer Engineering Supervisors: Prof. Dr. Fernando Manuel Bernardo Pereira Prof. Dr. João Miguel Duarte Ascenso Examination Committee Chairperson: Prof. Dr. José Eduardo Charters Ribeiro da Cunha Sanguino Supervisor: Prof. Dr. João Miguel Duarte Ascenso Member of the Committee: Prof. Dr. Nuno Miguel Morais Rodrigues November 2018 ii Declaration I declare that this document is an original work of my own authorship and that it fulfills all the requirements of the Code of Conduct and Good Practices of the Universidade de Lisboa. iii iv Acknowledgments I would like to thank all my family and friends that have supported me throughout this degree with a special mention to my parents, for all the financial and emotional support they have provided me during these 5 years. I’d also like to thank my girlfriend for her love, and for her support and motivation throughout the most difficult challenges we have both surpassed together. I would also like to thank my supervisors for their guidance, orientation and lessons that I take for life: Prof. Fernando Pereira, Prof. João Ascenso and Prof. Catarina Brites. Lastly, I’d like to thank to my colleagues at Instituto de Telecomunicações, namely Afonso Costa and Antoine Dricot, for the many discussions we have held, and for the help they have given me during the development of the Thesis. Without these people, achieving this degree would never be possible. v vi Resumo A recente evolução dos sistemas de aquisição 3D e dos métodos de processamento deste tipo de informação levaram ao desenvolvimento de novos modelos de representação 3D, nomeadamente point clouds, holographic imaging e light fields. Uma point cloud é um conjunto de pontos com uma localização 3D associada, e opcionalmente alguns atributos, tais como cor, refletência e normais. Este modelo de representação viabiliza um novo conjunto de aplicações, desde a telepresença 3D a mapas 3D de larga escala. Contudo, a elevada imersão e o realismo acarretam consigo altos débitos binários, e por isso, soluções de codificação eficientes são essenciais. Como consequência, o MPEG lançou um projeto de normalização de soluções de codificação eficiente de point clouds. Este projeto deu origem a duas soluções de codificação: o Video-based Point Cloud Coding (V-PCC) para conteúdo dinâmico e o Geometry-based Point Cloud Coding (G-PCC) para conteúdo estático. O objetivo desta Dissertação é o desenho, implementação e avaliação de uma solução de codificação que seguindo a estrutura do MPEG V-PCC seja capaz de melhorar o seu desempenho na codificação de point clouds estáticas (modo intra). Assim, este trabalho começa por abordar o estado- da-arte na área das point clouds, com especial foco nos conceitos mais relevantes, bem como nas soluções de codificação mais relevantes na literatura, incluindo as duas soluções de codificação do MPEG. A solução de codificação proposta segue a estrutura do MPEG V-PCC mas inova ao adaptar os planos de projeção às características específicas de cada point cloud. A análise de resultados demonstra um melhor desempenho de geometria relativamente ao MPEG V-PCC, especialmente para médias e altas qualidades. Palavras-chave: Modelos de representação visual, sistemas de aquisição 3D, point clouds , soluções de codificação, point clouds estáticas. vii viii Abstract In recent years, the evolution of 3D capture systems and processing methods have unleashed the breakthrough of new 3D representation models, namely point clouds, holographic imaging and light fields. A point cloud is a set of points with an associated 3D location and, optionally, some attributes, such as colour, reflectance, and normals. This visual representation model enables a more complete, realistic and immersive representation of the visual world, which allows a whole new set of applications, from 3D telepresence to large-scale 3D maps. However, the immersion and realism of the point cloud representation model comes at the cost of a very high bitrate, and thus efficient coding solutions are essential. As a consequence, MPEG has launched a standardization project on efficient point cloud coding solutions. This project has given rise to two coding solutions, notably Video-based Point Cloud Coding (V-PCC) to encode dynamic content and Geometry-based Point Cloud Coding (G-PCC) to encode static content. This Thesis targets the design, implementation and assessment of a static point cloud coding solution following the MPEG V-PCC framework with increased compression performance. To achieve this ultimate goal, this Thesis starts by addressing the state-of-art on point clouds, with a special focus on the main relevant concepts and the most relevant coding solutions in the literature, including the two MPEG coding solutions. The coding solution follows MPEG V-PCC and innovates by adapting the projection planes to the specific characteristics of each point cloud. The assessment results show a solid improvement on the geometry performance with respect to the MPEG V-PCC, especially for medium to high rates. Keywords: Visual representation models, 3D acquisition systems, point clouds, coding solutions, static point clouds. ix x Table of Contents 1. Introduction .............................................................................................................................. 1 1.1. Context and Motivation .......................................................................................................... 1 1.2. Objectives and Structure of the Thesis ................................................................................. 2 2. Point Cloud Coding: Basics and Main Coding Solutions ......................................................... 4 2.1. Main Concepts ....................................................................................................................... 4 2.2. End-to-End Point Cloud Processing Architecture ................................................................. 7 2.3. Point Cloud Use Cases ....................................................................................................... 14 2.4. Point Cloud Quality Assessment ......................................................................................... 17 2.5. Main Point Cloud Coding Solutions ..................................................................................... 21 2.5.1. Point Cloud Library Solution ............................................................................................. 21 2.5.2. JPEG-Based Point Cloud Colour Coding ......................................................................... 22 2.5.3. Graph-based Transform Coding Solution ......................................................................... 26 2.5.4. Graph-based Compression of Dynamic Point Clouds ...................................................... 30 2.5.5. Dynamic Point Cloud Compression with Coding-Mode Decision ..................................... 33 2.6. MPEG Geometry-based Point Cloud Coding ...................................................................... 37 3. MPEG Video-based Point Cloud Coding ............................................................................... 43 3.1. Architecture and Walkthrough ............................................................................................. 43 3.2. Patch Generation Process................................................................................................... 46 4. Proposing Adaptive Plane Projections for V-PCC ................................................................. 53 4.1. Objectives, Benefits and Potential Complications ............................................................... 54 4.2. Architecture and Walkthrough ............................................................................................. 56 4.3. Main Tools Description ........................................................................................................ 58 4.3.1. K-Means Clustering .......................................................................................................... 59 4.3.2. Adaptive Plane Selection and Local Coordinate System Transformation ........................ 61 4.3.3. Adaptive Resolution Control ............................................................................................. 66 4.4. Additional Tools ................................................................................................................... 67 4.4.1. Recolouring Pre and Post-Processing .............................................................................. 68 4.4.2. Duplicates Removal .......................................................................................................... 69 5. AP2V-PCC Performance Assessment ................................................................................... 72 xi 5.1. Test Material ........................................................................................................................ 72 5.2. Test Conditions .................................................................................................................... 73 5.3. Objective Evaluation Metrics ............................................................................................... 74 5.4. AP2V-PCC Parameters Optimization .................................................................................. 75 5.4.1. Number of Clusters (K) Optimization