Point Cloud Subjective Evaluation Methodology Based on Reconstructed Surfaces

Point cloud subjective evaluation methodology based on reconstructed surfaces Evangelos Alexioua, Antonio M. G. Pinheirob, Carlos Duartec, Dragan Matkovićd, Emil Dumićd, Luis A. da Silva Cruzc, Lovorka Gotal Dmitrovićd, Marco V. Bernardob Manuela Pereirab and Touradj Ebrahimia aMultimedia Signal Processing Group, Ecole´ Polytechnique Fédéralede Lausanne, Switzerland; bInstitute of Telecommunications, University of Beira Interior, Portugal; cDepartment of Electrical and Computer Engineering, University of Coimbra and Institute of Telecommunications, Portugal; dDepartment of Electrical Engineering, University North, Croatia ABSTRACT Point clouds have been gaining importance as a solution to the problem of efficient representation of 3D geometric and visual information. They are commonly represented by large amounts of data, and compression schemes are important for their manipulation transmission and storing. However, the selection of appropriate compression schemes requires effective quality evaluation. In this work a subjective quality evaluation of point clouds using a surface representation is analyzed. Using a set of point cloud data objects encoded with the popular octree pruning method with different qualities, a subjective evaluation was designed. The point cloud geometry was presented to observers in the form of a movie showing the 3D Poisson reconstructed surface without textural information with the point of view changing in time. Subjective evaluations were performed in three different laboratories. Scores obtained from each test were correlated and no statistical differences were observed. Scores were also correlated with previous subjective tests and a good correlation was obtained when compared with mesh rendering in 2D monitors. Moreover, the results were correlated with state of the art point cloud objective metrics revealing poor correlation. Likewise, the correlation with a subjective test using a different representation of the point cloud data also showed poor correlation. These results suggest the need for more reliable objective quality metrics and further studies on adequate point cloud data representations. Keywords: Subjective Quality Assessment, Point Cloud, Quality Metrics 1. INTRODUCTION The significant growth of interest in adopting 3D imaging modalities in modern information technologies and communication systems, indicate the need for advanced content representations. Among the alternatives, point clouds have been gaining a large interest from the technological market, as a result of the efficiency and simplicity they offer in capturing, storing and rendering of 3D objects. Considering the current availability of several existing solutions for acquisition and display, a wide range of applications and use cases can benefit from this type of data representation. In particular, a well-documented list has already been identified, and detailed in the JPEG Pleno Point Clouds { Use Cases and Requirements.1 In this document, the following major classes have been defined: 1) Rendering of content for virtual, augmented and mixed reality technologies; 2) 3D content creation; 3) Medical applications; 4) Construction and manufacturing; 5) Consumer and retail; 6) Cultural heritage; 7) Remote Sensing, GIS; 8) Autonomous vehicles, drones; and 9) Surveillance. Point clouds are typically represented by huge amounts of data. Thus, efficient and reliable compression solutions are essential. However, the identification of best approaches, or even their improvement, is limited by the lack of adequate solutions for evaluation of the visual quality of a point cloud content. The latter, is currently one of the major challenges and open problems in point cloud imaging. Independently of the type of Further author information: (Send correspondence to Antonio M. G. Pinheiro.) Antonio M. G. Pinheiro: E-mail: [email protected] data representation, the visual quality of a content is typically evaluated through either subjective or objective quality assessments. Objective quality assessment is performed through computer algorithms that are designed to estimate signal distortions. Subjective quality assessment involves the participation of subjects in experiments in which distorted objects are visualized and their quality is rated. Such experiments are expensive in terms of both cost and time. Thus, the objective quality metrics are frequently used instead, especially, for real-time quality assessment, which is typically part of advanced compression schemes. However, subjective evaluations are necessary to provide the ground truth, against which the objective scores are calibrated and benchmarked. In this paper we report the results of point cloud subjective evaluation experiments, that took place in three different test laboratories. The point clouds were compressed by octree pruning2 using the Point Cloud Library (PCL) v1.8.0.3 The processed point cloud stimuli were then presented and assessed by the subjects after 3D rendering using Screened Poisson surface reconstruction,4 and displayed as a 3D sequence showing the object from changing point of views. Three different 3D monitors were used according to each testing laboratory availability. The dataset proposed in a previous work5 was employed to facilitate a comparison between previous work, on our aim to identify statistical differences that may occur by different point cloud rendering and representation approaches. 2. RELATED WORK Recently, a significant amount of work has been reported in the literature for subjective and objective quality assessment of point clouds. In particular, several objective quality metrics have been proposed and their correlation with subjective quality scores has already been investigated. However, in most such studies, point clouds were displayed as sets of points without applying any surface reconstruction algorithm before the final rendering; notice that the latter reflects a rather common way to consume 3D contents nowadays. In particular, Zhang et al.6 performed subjective assessment of raw point clouds degraded by geometry and color, after applying uniform noise. However, this type of degradation is not realistic neither for color nor for geometry artifacts. Mekuria et al.7 proposed a generic and real-time time-varying point cloud codec for 3D immersive video. The subjective quality of the codec performance was evaluated in a realistic 3D tele-immersive system in a virtual 3D room scenario where users are represented and interact as 3D avatars (synthetic content) and/or 3D PC (naturalistic content). Different aspects of the quality were tested, including overall quality of the 3D human rendition, the quality of the colors, the perceived realism of the point cloud, the motion quality, the near/far quality, and the level of immersiveness with the observer comparing the avatar and the point cloud representations. Mekuria et al.8 provides bitrate results and objective scores for their proposed point cloud codec,7 which was assessed in the framework of the recent activities of the MPEG standardization committee. The selected data set consists of colored objects and different encoding strategies were considered, providing bitrate results and objective scores. For the subjective evaluations, a passive approach was adopted by obtaining a video from a point cloud renderer using a predefined path for the virtual camera. Raw point clouds were visualized, with each point being displayed as a cube of a fixed size. Although the point size was allowed to be configured, a fixed value was set per content. However, in these studies, no correlation between the subjective and objective scores is reported. In,9 point cloud denoising algorithms were subjectively evaluated and the test contents were visualized after applying the Screened Poisson surface reconstruction.4 A passive assessment was adopted and 2D video sequences were formed, after capturing the resulting mesh objects from different viewpoints by vertical and hor- izontal rotation. However, the impact of visualizing reconstructed meshes instead of raw point clouds was not investigated. In,10 subjective quality assessment of colored point clouds was conducted, subject to simple octree and graph-based encoding algorithms. To render the point cloud data, primitive cubes were employed, whose size was adjusted based on the local neighborhood. The resulting test contents were captured from different viewing angles with the virtual camera following a spiral path. The subjects visualized animated 2D video sequences to provide their scores. In2 and,11 an interactive approach to subjectively assess geometry-only point clouds in a typical desktop setup was proposed, using the Double-Stimulus Impairment Scale (DSIS) and the Absolute Category Rating (ACR) evaluation methodologies, respectively. In the latter study, the correlation between these two tests was also investigated. The contents under evaluation were degraded after applying Gaussian noise, and octree-based compression. The test contents were presented to the subjects as sets of points. In,12 a subjective methodology for point clouds using head mounted displays in an augmented reality scenario was proposed. The subjects visualized raw point clouds and their interaction with the 3D models was performed by physical movements in the virtual reality world environment. More recently,5 a subjective evaluation in five different independent laboratories with visualization of surface reconstructed point clouds using the Screened Poisson surface reconstruction algorithm.4 The point

Load more