IEEE Robotics and Automation Letters (RAL) paper presented at the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) October 25-29, 2020, Las Vegas, NV, USA (Virtual) A Novel Coding Architecture for LiDAR Point Cloud Sequence Xuebin Sun1*, Sukai Wang2*, Miaohui Wang3, Zheng Wang4 and Ming Liu2, Senior Member, IEEE Abstract— In this paper, we propose a novel coding architec- the point cloud data. However, these methods are unsuitable ture for LiDAR point cloud sequences based on clustering and for unmanned vehicles. Traditional image or video encoding prediction neural networks. LiDAR point clouds are structured, algorithms, such as JPEG2000 , JPEG-LS [3], and HEVC which provides an opportunity to convert the 3D data to a 2D array, represented as range images. Thus, we cast the [4], were designed mostly for encoding integer pixel values, LiDAR point clouds compression as a range images coding and using them to encode floating-point LiDAR data will problem. Inspired by the high efficiency video coding (HEVC) cause significant distortion. Furthermore, the range image is algorithm, we design a novel coding architecture for the point characterized by sharp edges and homogeneous regions with cloud sequence. The scans are divided into two categories: nearly constant values, which is quite different from textured intra-frames and inter-frames. For intra-frames, a cluster-based intra-prediction technique is utilized to remove the spatial video. Thus, coding the range image with traditional tools redundancy. For inter-frames, we design a prediction network such as the block-based discrete cosine transform (DCT) model using convolutional LSTM cells, which is capable of followed by coarse quantization can result in significant predicting future inter-frames according to the encoded intra- coding errors at sharp edges, causing a safety hazard in frames. Thus, the temporal redundancy can be removed. autonomous driving. Experiments on the KITTI data set show that the proposed method achieves an impressive compression ratio, with 4.10% In this research, we address the LiDAR point cloud at millimeter precision. Compared with octree, Google Draco sequence compression problem. Learning from the HEVC and MPEG TMC13 methods, our scheme also yields better architecture, we propose a novel coding architecture for performance in compression ratio. LiDAR point cloud sequences, which mainly consists of I. INTRODUCTION intra-prediction and inter-prediction technologies. For-intra- frames, we utilize a cluster-based intra-prediction method Advances in autonomous driving technology have widened to remove the spatial redundancy. There are great structural the use of 3D data acquisition techniques. LiDAR is almost similarities between adjacent point clouds. For inter-frames, indispensable for outdoor mobile robots, and plays a funda- we train a prediction neural network, which is capable of mental role in many autonomous driving applications such generating the future inter-frames using the encoded intra- as localization , path planning [1], and obstacle detection [2], frames. The intra- and inter-residual data is quantified and etc. The enormous volume of LiDAR point cloud data could coded using lossless coding schemes. Experiments on the be an important bottleneck for transmission and storage. KITTI dataset demonstrate our method yields an impressive Therefore, it is highly desirable to develop an efficient coding performance. algorithm to satisfy the requirement of autonomous driving. In our previous paper [5], an efficient compression algo- Octree methods have been widely researched for point rithm for a single scan is developed based on clustering. cloud compression. The main idea of octree-based coding Based on this previous technique, we propose a novel coding methods is to recursively subdivide the current data ac- architecture for LiDAR point cloud sequences in this work. cording to the range of coordinates from top to bottom, The contributions of the paper are summarized as follows. and gradually form an octree adaptive structure. Octree method can hardly compress LiDAR data into very small • Learning from the HEVC algorithm, we develop a novel volumes with low information loss. The vehicle-mounted coding architecture for LiDAR point cloud sequences. LiDAR data is structured, which provides a chance to convert • For inter-frames, we design a prediction network model them into a 2D panorama range image. Some researchers using convolutional LSTM cells. The network model is focus on using image-based coding methods to compress capable of predicting future inter-frames according to the encoded intra-frames. * The first two authors contributed equally to this work. • The coding scheme is specially designed for LiDAR 1 Xuebin Sun is now with the Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong. He point cloud sequences for autonomous driving. Com- contributed to this work during his time at HKUST. (email: sunxue- pared with octree, Google Draco and MPEG TMC13 [email protected]; [email protected]) methods, our method yields better performance. 2 Sukai Wang and Ming Liu are with the Department of Electronic & Computer Engineering, Hong Kong University of Science and Technology, Hong Kong. (email: [email protected]; [email protected]) II. RELATED WORK 3 Miaohui Wang is with the College of Electrical and Information Engi- neering, Shenzhen University, China. (email: [email protected]) The compression of 3D point cloud data has been widely 4 Zheng Wang is with the Department of Mechanical Engineering, researched in literature. According to the types of point cloud the University of Hong Kong, Hong Kong, also with the Department of Mechanical and Energy Engineering, Southern University of Science and data, compression algorithms can be roughly classified into Technology, China. (email: [email protected]) four categories. Copyright ©2020 IEEE Structured single point cloud compression: Some re- cloud compression methods. In [15], Tu et al. develop a searchers have focused on developing compression methods recurrent neural network with residual blocks for LiDAR for structured LiDAR point cloud data. Houshiar et al. point cloud streams compression. Their network structure [6] propose an image-based 3D point cloud compression is like a coding and decoding process. The original point method. They map the 3D points onto three panorama cloud data is encoded into low-dimensional features, which images, and use an image coding method to compress the is treated as encoded bit stream. The decoding process is to images. Similar to their approach, Ahn et al. [7] introduce decode these low-dimensional features to the original point an adaptive range image coding algorithm for the geometry cloud data. In [16], Tu et al. present a real-time point cloud compression of large-scale 3D point clouds. They explore compression scheme for 3D LiDAR sensor U-Net. Firstly, a prediction method to predict the radial distance of each some frames are choosen as key frames (I-frame), then they pixel using previously encoded neighbors, and only encode use the U-net to interpolate the remaining LiDAR frames the resulting prediction residuals. In contrast, Zanuttigh et al. (P-frames) between the key frames. [8] focus on efficient compression of depth maps of RGB- Unstructured point cloud sequence compression: D data. They develop a segmentation method to identify Saranya et al. [17] propose a real-time compression strategy the edges and main objects of a depth map. After that, on various point cloud streams. They perform an octree- an efficient prediction process is performed according to based spatial decomposition to remove the spatial redun- the segmentation result, and the residual data between the dancy. Additionally, by encoding the structural differences predicted and real depth map is calculated. Finally, the of adjacent point clouds, the temporal redundancy can be few prediction residuals are encoded by conventional image removed. Thanou et al. [18] present a graph-based compres- compression methods. sion for dynamic 3D point cloud sequences. In their method, Unstructured singe point cloud compression: Elseberg the time-varying geometry of the point cloud sequence is et al. [9] propose an efficient octree data structure to store represented by a set of graphs, where 3D points and color and compress 3D data without loss of precision. Experimen- attributes are considered as signals. Their method is based tal results demonstrate their method is useful for an exchange on exploiting the temporal correlation between consecutive file format, fast point cloud visualization, sped-up 3D scan point clouds and removing the redundancy. Mekuria et al. matching, and shape detection algorithms. Golla et al. [10] [19] introduce a generic and real-time time-varying point present a real-time compression algorithm for point cloud cloud coding approach for 3D immersive video. They code data based on local 2D parameterizations of surface point intra-frames with an octree data structure. Besides this, they cloud data. They use standard image coding techniques to divide the octree voxel space into macroblocks and develop compress the local details. Zhang et al. [11] introduce a an inter-prediction method. clustering- and DCT-based color point cloud compression Generally, the aforementioned approaches can signifi- method. In their method, they use the mean-shift technique cantly reduce the size of point cloud data, and are capable