DOCSLIB.ORG

Dictionary-Based Compression Algorithms in Mobile Packet Core

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Master of Science in Computer Science October 2019 Dictionary-based Compression Algorithms in Mobile Packet Core Lakshmi Venkata Sai Sri Tikkireddy Faculty of Computing, Blekinge Institute of Technology, 371 79 Karlskrona, Sweden This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Computer Science. The thesis is equivalent to 20 weeks of full-time studies. Contact Information: Author(s): Lakshmi Venkata Sai Sri Tikkireddy E-mail: [email protected] External advisor: 1. Erik Vargas ([email protected]) 2. Nils Ljungberg ([email protected]) University advisor: Siamak Khatibi Department of Telecommunications Faculty of Telecommunications Internet : www.bth.se Blekinge Institute of Technology Phone : +46 455 38 50 00 SE-371 79 Karlskrona, Sweden Fax : +46 455 38 50 57 ii ABSTRACT Context: With the rapid growth in technology, the amount of data to be transmitted and stored is increasing. The efficiency of information retrieval and storage has become a major drawback, thereby the concept of data compression has come into the picture. Data compression is a technique that effectively reduces the size of the data to save storage and speed up the transmission of the data from one place to another. Data compression is present in various formats and mainly categorized into lossy compression and lossless compression where lossless compression is often used to compress the data. In Ericsson, SGSN-MME is using one of the data compression technique namely Deflate, to compress each user data independently. Due to the compression ratio between compress and decompress speed, the deflate algorithm is not optimal for the SGSN-MME’s use case. To mitigate this problem, the deflate algorithm has to be replaced with a better compression algorithm. Objectives: This research is performed on several dictionary-based lossless data compression algorithms to find a suitable compression algorithm for the SGSN-MME's use case. To achieve this goal, we need to find out the type of data that is required for creating a dataset for the compression algorithms. After the dataset creation, various dictionary-based algorithms are examined to find the suitable dictionary-based algorithm for the use case and compare the performance of the found algorithm when using/not using a pre-defined dictionary. Methods: In this research, an experiment is performed to evaluate the performance of different dictionary-based algorithms based on compression ratio and time for compression. For this experiment, the data is provided by Ericsson AB, Gothenburg. The dataset consists of the user data from SGSN- MME. The selected dictionary-based algorithms namely LZ4, Brotli, Zstandard are evaluated based on their performance compared to Deflate based on the compression factors such as compression ratio and compression speed. Results: On observation and analysis of the experiment, Zstandard with dictionary was better in performance when compared with the compression factors such as compression ratio and compression speed. Conclusions: This research is concluded by identifying a suitable dictionary-based algorithm. The conclusion of the research is decided by showing the identified algorithm performs better than remaining selected algorithms LZ4, Brotli and Deflate. Keywords: Data Compression, Lossless compression, LZ4, Brotli, Zstandard, Deflate, Dictionary- based compression. iii ACKNOWLEDGMENT Firstly, I would like to thank my supervisor Prof. Siamak Khatibi, Department of Telecommunications and Prof. Emilia Mendes, Head of the Department of Computer Science and Engineering, for their unmatched guidance and support without which I could not have completed this study successfully. I sincerely thank them for believing in me and encouraging me all through this study. I would also like to thank my external supervisors Erik Vargas and Nils Ljungberg at Ericsson AB, Gothenburg. I thank them for the never-ending support and incredible motivation for this thesis. I am grateful to my parents for their unconditional love and support. Lastly, I would like to thank my near and dear friends for traveling alongside me in all my endeavors. iv LIST OF FIGURES FIGURE 1: THE PROCESS OF DATA COMPRESSION. ................................................................................. 10 FIGURE 2: VARIOUS COMPRESSION TECHNIQUES................................................................................... 11 FIGURE 3:TYPES OF LOSSLESS COMPRESSION ALGORITHMS. ............................................................... 13 FIGURE 4: FLOW CHART OF THE LZ4 ALGORITHM. ................................................................................ 15 FIGURE 5: ZSTD VS ZLIB (DEFLATE) [40] ............................................................................................... 18 FIGURE 6: CITRIX OVERVIEW................................................................................................................. 20 FIGURE 7: COMPRESSION OF ZSTANDARD (NORMAL) ............................................................................ 25 FIGURE 8: COMPRESSION OF ZSTANDARD USING THE DICTIONARY...................................................... 25 FIGURE 9: COMPARISON OF THE COMPRESSED SIZES ............................................................................. 26 FIGURE 10: ZSTD VS ZSTD_DICT IN TERMS OF COMPRESSION RATIO (AVERAGE) ................................. 27 FIGURE 11: ZSTD VS ZSTD_DICT IN TERMS OF COMPRESSION SPEED (AVERAGE) ................................. 28 FIGURE 12: COMPRESSION SPEED VS RATIO .......................................................................................... 28 FIGURE 13: ZSTD VS ZSTD_DICT IN TERMS OF SPACE SAVINGS (AVERAGE) .......................................... 29 v LIST OF TABLES TABLE 1: RATIO VS SPEED COMPARISON [42] ....................................................................................... 17 TABLE 2: COMPARING THE RATIOS OF THE ALGORITHM RESULTS. ....................................................... 27 TABLE 3: RANKS OF THE TEST CASE RATIOS FOR THE ALGORITHMS. .................................................... 30 TABLE 4: AVERAGE RANKS OF THE ALGORITHMS ................................................................................. 30 vi ABBREVIATIONS 1) SGSN Service GPRS Support Node 2) GPRS General Packet Radio Service 3) MME Mobility Management Entity 4) UE User Equipment 5) GSNWS GSN Work Space 6) ETS Erlang Term Storage 7) OTP Erlang Open Telecom Platform 8) Zstd Zstandard 9) Zstd_dict Zstandard using the developed dictionary 10) FSE Finite State Entropy 11) ANS Asymmetric Numeral Systems 12) tANS Tabled Variant of ANS vii TABLE OF CONTENTS Abstract ................................................................................................................................................... iii Acknowledgment ..................................................................................................................................... iv List of figures............................................................................................................................................ v List of tables ............................................................................................................................................ vi Abbreviations ......................................................................................................................................... vii Table of Contents .................................................................................................................................. viii 1 Introduction ................................................................................................................................... 10 1.1 Problem statement: ............................................................................................................... 11 2 Related Work ................................................................................................................................. 13 2.1 Entropy-Based Encoding ...................................................................................................... 14 2.1.1 Huffman Coding ............................................................................................................... 14 2.1.2 Arithmetic Coding ............................................................................................................ 14 2.2 Dictionary Based Encoding .................................................................................................. 14 2.2.1 LZ4 ................................................................................................................................... 15 2.2.2 Brotli................................................................................................................................. 16 2.2.3 Deflate .............................................................................................................................. 16 2.2.4 Zstandard .......................................................................................................................... 17 3 Method ........................................................................................................................................... 19 3.1 Experiment Workspace ......................................................................................................... 20 3.1.1 Why
Recommended publications
  • Data Compression: Dictionary-Based Coding 2 / 37 Dictionary-Based Coding Dictionary-Based Coding

    Data Compression: Dictionary-Based Coding 2 / 37 Dictionary-Based Coding Dictionary-Based Coding

    Dictionary-based Coding already coded not yet coded search buffer look-ahead buffer cursor (N symbols) (L symbols) We know the past but cannot control it. We control the future but... Last Lecture Last Lecture: Predictive Lossless Coding Predictive Lossless Coding Simple and effective way to exploit dependencies between neighboring symbols / samples Optimal predictor: Conditional mean (requires storage of large tables) Affine and Linear Prediction Simple structure, low-complex implementation possible Optimal prediction parameters are given by solution of Yule-Walker equations Works very well for real signals (e.g., audio, images, ...) Efficient Lossless Coding for Real-World Signals Affine/linear prediction (often: block-adaptive choice of prediction parameters) Entropy coding of prediction errors (e.g., arithmetic coding) Using marginal pmf often already yields good results Can be improved by using conditional pmfs (with simple conditions) Heiko Schwarz (Freie Universität Berlin) — Data Compression: Dictionary-based Coding 2 / 37 Dictionary-based Coding Dictionary-Based Coding Coding of Text Files Very high amount of dependencies Affine prediction does not work (requires linear dependencies) Higher-order conditional coding should work well, but is way to complex (memory) Alternative: Do not code single characters, but words or phrases Example: English Texts Oxford English Dictionary lists less than 230 000 words (including obsolete words) On average, a word contains about 6 characters Average codeword length per character would be limited by 1
  • Package 'Brotli'

    Package 'Brotli'

    Package ‘brotli’ May 13, 2018 Type Package Title A Compression Format Optimized for the Web Version 1.2 Description A lossless compressed data format that uses a combination of the LZ77 algorithm and Huffman coding. Brotli is similar in speed to deflate (gzip) but offers more dense compression. License MIT + file LICENSE URL https://tools.ietf.org/html/rfc7932 (spec) https://github.com/google/brotli#readme (upstream) http://github.com/jeroen/brotli#read (devel) BugReports http://github.com/jeroen/brotli/issues VignetteBuilder knitr, R.rsp Suggests spelling, knitr, R.rsp, microbenchmark, rmarkdown, ggplot2 RoxygenNote 6.0.1 Language en-US NeedsCompilation yes Author Jeroen Ooms [aut, cre] (<https://orcid.org/0000-0002-4035-0289>), Google, Inc [aut, cph] (Brotli C++ library) Maintainer Jeroen Ooms <[email protected]> Repository CRAN Date/Publication 2018-05-13 20:31:43 UTC R topics documented: brotli . .2 Index 4 1 2 brotli brotli Brotli Compression Description Brotli is a compression algorithm optimized for the web, in particular small text documents. Usage brotli_compress(buf, quality = 11, window = 22) brotli_decompress(buf) Arguments buf raw vector with data to compress/decompress quality value between 0 and 11 window log of window size Details Brotli decompression is at least as fast as for gzip while significantly improving the compression ratio. The price we pay is that compression is much slower than gzip. Brotli is therefore most effective for serving static content such as fonts and html pages. For binary (non-text) data, the compression ratio of Brotli usually does not beat bz2 or xz (lzma), however decompression for these algorithms is too slow for browsers in e.g.
  • A Survey Paper on Different Speech Compression Techniques

    A Survey Paper on Different Speech Compression Techniques

    Vol-2 Issue-5 2016 IJARIIE-ISSN (O)-2395-4396 A Survey Paper on Different Speech Compression Techniques Kanawade Pramila.R1, Prof. Gundal Shital.S2 1 M.E. Electronics, Department of Electronics Engineering, Amrutvahini College of Engineering, Sangamner, Maharashtra, India. 2 HOD in Electronics Department, Department of Electronics Engineering , Amrutvahini College of Engineering, Sangamner, Maharashtra, India. ABSTRACT This paper describes the different types of speech compression techniques. Speech compression can be divided into two main types such as lossless and lossy compression. This survey paper has been written with the help of different types of Waveform-based speech compression, Parametric-based speech compression, Hybrid based speech compression etc. Compression is nothing but reducing size of data with considering memory size. Speech compression means voiced signal compress for different application such as high quality database of speech signals, multimedia applications, music database and internet applications. Today speech compression is very useful in our life. The main purpose or aim of speech compression is to compress any type of audio that is transfer over the communication channel, because of the limited channel bandwidth and data storage capacity and low bit rate. The use of lossless and lossy techniques for speech compression means that reduced the numbers of bits in the original information. By the use of lossless data compression there is no loss in the original information but while using lossy data compression technique some numbers of bits are loss. Keyword: - Bit rate, Compression, Waveform-based speech compression, Parametric-based speech compression, Hybrid based speech compression. 1. INTRODUCTION -1 Speech compression is use in the encoding system.
  • ACS – the Archival Cytometry Standard

    ACS – the Archival Cytometry Standard

    http://flowcyt.sf.net/acs/latest.pdf ACS – the Archival Cytometry Standard Archival Cytometry Standard ACS International Society for Advancement of Cytometry Candidate Recommendation DRAFT Document Status The Archival Cytometry Standard (ACS) has undergone several revisions since its initial development in June 2007. The current proposal is an ISAC Candidate Recommendation Draft. It is assumed, however not guaranteed, that significant features and design aspects will remain unchanged for the final version of the Recommendation. This specification has been formally tested to comply with the W3C XML schema version 1.0 specification but no position is taken with respect to whether a particular software implementing this specification performs according to medical or other valid regulations. The work may be used under the terms of the Creative Commons Attribution-ShareAlike 3.0 Unported license. You are free to share (copy, distribute and transmit), and adapt the work under the conditions specified at http://creativecommons.org/licenses/by-sa/3.0/legalcode. Disclaimer of Liability The International Society for Advancement of Cytometry (ISAC) disclaims liability for any injury, harm, or other damage of any nature whatsoever, to persons or property, whether direct, indirect, consequential or compensatory, directly or indirectly resulting from publication, use of, or reliance on this Specification, and users of this Specification, as a condition of use, forever release ISAC from such liability and waive all claims against ISAC that may in any manner arise out of such liability. ISAC further disclaims all warranties, whether express, implied or statutory, and makes no assurances as to the accuracy or completeness of any information published in the Specification.
  • Arxiv:2004.10531V1 [Cs.OH] 8 Apr 2020

    Arxiv:2004.10531V1 [Cs.OH] 8 Apr 2020

    ROOT I/O compression improvements for HEP analysis Oksana Shadura1;∗ Brian Paul Bockelman2;∗∗ Philippe Canal3;∗∗∗ Danilo Piparo4;∗∗∗∗ and Zhe Zhang1;y 1University of Nebraska-Lincoln, 1400 R St, Lincoln, NE 68588, United States 2Morgridge Institute for Research, 330 N Orchard St, Madison, WI 53715, United States 3Fermilab, Kirk Road and Pine St, Batavia, IL 60510, United States 4CERN, Meyrin 1211, Geneve, Switzerland Abstract. We overview recent changes in the ROOT I/O system, increasing per- formance and enhancing it and improving its interaction with other data analy- sis ecosystems. Both the newly introduced compression algorithms, the much faster bulk I/O data path, and a few additional techniques have the potential to significantly to improve experiment’s software performance. The need for efficient lossless data compression has grown significantly as the amount of HEP data collected, transmitted, and stored has dramatically in- creased during the LHC era. While compression reduces storage space and, potentially, I/O bandwidth usage, it should not be applied blindly: there are sig- nificant trade-offs between the increased CPU cost for reading and writing files and the reduce storage space. 1 Introduction In the past years LHC experiments are commissioned and now manages about an exabyte of storage for analysis purposes, approximately half of which is used for archival purposes, and half is used for traditional disk storage. Meanwhile for HL-LHC storage requirements per year are expected to be increased by factor 10 [1]. arXiv:2004.10531v1 [cs.OH] 8 Apr 2020 Looking at these predictions, we would like to state that storage will remain one of the major cost drivers and at the same time the bottlenecks for HEP computing.
  • Annual Report 2016

    Annual Report 2016

    ANNUAL REPORT 2016 PUNJABI UNIVERSITY, PATIALA © Punjabi University, Patiala (Established under Punjab Act No. 35 of 1961) Editor Dr. Shivani Thakar Asst. Professor (English) Department of Distance Education, Punjabi University, Patiala Laser Type Setting : Kakkar Computer, N.K. Road, Patiala Published by Dr. Manjit Singh Nijjar, Registrar, Punjabi University, Patiala and Printed at Kakkar Computer, Patiala :{Bhtof;Nh X[Bh nk;k wjbk ñ Ò uT[gd/ Ò ftfdnk thukoh sK goT[gekoh Ò iK gzu ok;h sK shoE tk;h Ò ñ Ò x[zxo{ tki? i/ wB[ bkr? Ò sT[ iw[ ejk eo/ w' f;T[ nkr? Ò ñ Ò ojkT[.. nk; fBok;h sT[ ;zfBnk;h Ò iK is[ i'rh sK ekfJnk G'rh Ò ò Ò dfJnk fdrzpo[ d/j phukoh Ò nkfg wo? ntok Bj wkoh Ò ó Ò J/e[ s{ j'fo t/; pj[s/o/.. BkBe[ ikD? u'i B s/o/ Ò ô Ò òõ Ò (;qh r[o{ rqzE ;kfjp, gzBk óôù) English Translation of University Dhuni True learning induces in the mind service of mankind. One subduing the five passions has truly taken abode at holy bathing-spots (1) The mind attuned to the infinite is the true singing of ankle-bells in ritual dances. With this how dare Yama intimidate me in the hereafter ? (Pause 1) One renouncing desire is the true Sanayasi. From continence comes true joy of living in the body (2) One contemplating to subdue the flesh is the truly Compassionate Jain ascetic. Such a one subduing the self, forbears harming others. (3) Thou Lord, art one and Sole.
  • Fast Cosine Transform to Increase Speed-Up and Efficiency of Karhunen-Loève Transform for Lossy Image Compression

    Fast Cosine Transform to Increase Speed-Up and Efficiency of Karhunen-Loève Transform for Lossy Image Compression

    Fast Cosine Transform to increase speed-up and efficiency of Karhunen-Loève Transform for lossy image compression Mario Mastriani, and Juliana Gambini Several authors have tried to combine the DCT with the Abstract —In this work, we present a comparison between two KLT but with questionable success [1], with particular interest techniques of image compression. In the first case, the image is to multispectral imagery [30, 32, 34]. divided in blocks which are collected according to zig-zag scan. In In all cases, the KLT is used to decorrelate in the spectral the second one, we apply the Fast Cosine Transform to the image, domain. All images are first decomposed into blocks, and each and then the transformed image is divided in blocks which are collected according to zig-zag scan too. Later, in both cases, the block uses its own KLT instead of one single matrix for the Karhunen-Loève transform is applied to mentioned blocks. On the whole image. In this paper, we use the KLT for a decorrelation other hand, we present three new metrics based on eigenvalues for a between sub-blocks resulting of the applications of a DCT better comparative evaluation of the techniques. Simulations show with zig-zag scan, that is to say, in the spectral domain. that the combined version is the best, with minor Mean Absolute We introduce in this paper an appropriate sequence, Error (MAE) and Mean Squared Error (MSE), higher Peak Signal to decorrelating first the data in the spatial domain using the DCT Noise Ratio (PSNR) and better image quality.
  • Arithmetic Coding

    Arithmetic Coding

    Arithmetic Coding Arithmetic coding is the most efficient method to code symbols according to the probability of their occurrence. The average code length corresponds exactly to the possible minimum given by information theory. Deviations which are caused by the bit-resolution of binary code trees do not exist. In contrast to a binary Huffman code tree the arithmetic coding offers a clearly better compression rate. Its implementation is more complex on the other hand. In arithmetic coding, a message is encoded as a real number in an interval from one to zero. Arithmetic coding typically has a better compression ratio than Huffman coding, as it produces a single symbol rather than several separate codewords. Arithmetic coding differs from other forms of entropy encoding such as Huffman coding in that rather than separating the input into component symbols and replacing each with a code, arithmetic coding encodes the entire message into a single number, a fraction n where (0.0 ≤ n < 1.0) Arithmetic coding is a lossless coding technique. There are a few disadvantages of arithmetic coding. One is that the whole codeword must be received to start decoding the symbols, and if there is a corrupt bit in the codeword, the entire message could become corrupt. Another is that there is a limit to the precision of the number which can be encoded, thus limiting the number of symbols to encode within a codeword. There also exist many patents upon arithmetic coding, so the use of some of the algorithms also call upon royalty fees. Arithmetic coding is part of the JPEG data format.
  • Encryption Introduction to Using 7-Zip

    Encryption Introduction to Using 7-Zip

    IT Services Training Guide Encryption Introduction to using 7-Zip It Services Training Team The University of Manchester email: [email protected] www.itservices.manchester.ac.uk/trainingcourses/coursesforstaff Version: 5.3 Training Guide Introduction to Using 7-Zip Page 2 IT Services Training Introduction to Using 7-Zip Table of Contents Contents Introduction ......................................................................................................................... 4 Compress/encrypt individual files ....................................................................................... 5 Email compressed/encrypted files ....................................................................................... 8 Decrypt an encrypted file ..................................................................................................... 9 Create a self-extracting encrypted file .............................................................................. 10 Decrypt/un-zip a file .......................................................................................................... 14 APPENDIX A Downloading and installing 7-Zip ................................................................. 15 Help and Further Reference ............................................................................................... 18 Page 3 Training Guide Introduction to Using 7-Zip Introduction 7-Zip is an application that allows you to: Compress a file – for example a file that is 5MB can be compressed to 3MB Secure the
  • Pack, Encrypt, Authenticate Document Revision: 2021 05 02

    Pack, Encrypt, Authenticate Document Revision: 2021 05 02

    PEA Pack, Encrypt, Authenticate Document revision: 2021 05 02 Author: Giorgio Tani Translation: Giorgio Tani This document refers to: PEA file format specification version 1 revision 3 (1.3); PEA file format specification version 2.0; PEA 1.01 executable implementation; Present documentation is released under GNU GFDL License. PEA executable implementation is released under GNU LGPL License; please note that all units provided by the Author are released under LGPL, while Wolfgang Ehrhardt’s crypto library units used in PEA are released under zlib/libpng License. PEA file format and PCOMPRESS specifications are hereby released under PUBLIC DOMAIN: the Author neither has, nor is aware of, any patents or pending patents relevant to this technology and do not intend to apply for any patents covering it. As far as the Author knows, PEA file format in all of it’s parts is free and unencumbered for all uses. Pea is on PeaZip project official site: https://peazip.github.io , https://peazip.org , and https://peazip.sourceforge.io For more information about the licenses: GNU GFDL License, see http://www.gnu.org/licenses/fdl.txt GNU LGPL License, see http://www.gnu.org/licenses/lgpl.txt 1 Content: Section 1: PEA file format ..3 Description ..3 PEA 1.3 file format details ..5 Differences between 1.3 and older revisions ..5 PEA 2.0 file format details ..7 PEA file format’s and implementation’s limitations ..8 PCOMPRESS compression scheme ..9 Algorithms used in PEA format ..9 PEA security model .10 Cryptanalysis of PEA format .12 Data recovery from
  • CALIFORNIA STATE UNIVERSITY, NORTHRIDGE LOSSLESS COMPRESSION of SATELLITE TELEMETRY DATA for a NARROW-BAND DOWNLINK a Graduate P

    CALIFORNIA STATE UNIVERSITY, NORTHRIDGE LOSSLESS COMPRESSION of SATELLITE TELEMETRY DATA for a NARROW-BAND DOWNLINK a Graduate P

    CALIFORNIA STATE UNIVERSITY, NORTHRIDGE LOSSLESS COMPRESSION OF SATELLITE TELEMETRY DATA FOR A NARROW-BAND DOWNLINK A graduate project submitted in partial fulfillment of the requirements For the degree of Master of Science in Electrical Engineering By Gor Beglaryan May 2014 Copyright Copyright (c) 2014, Gor Beglaryan Permission to use, copy, modify, and/or distribute the software developed for this project for any purpose with or without fee is hereby granted. THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. Copyright by Gor Beglaryan ii Signature Page The graduate project of Gor Beglaryan is approved: __________________________________________ __________________ Prof. James A Flynn Date __________________________________________ __________________ Dr. Deborah K Van Alphen Date __________________________________________ __________________ Dr. Sharlene Katz, Chair Date California State University, Northridge iii Contents Copyright .......................................................................................................................................... ii Signature Page ...............................................................................................................................
  • A Novel Coding Architecture for Multi-Line Lidar Point Clouds

    A Novel Coding Architecture for Multi-Line Lidar Point Clouds

    This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS 1 A Novel Coding Architecture for Multi-Line LiDAR Point Clouds Based on Clustering and Convolutional LSTM Network Xuebin Sun , Sukai Wang , Graduate Student Member, IEEE, and Ming Liu , Senior Member, IEEE Abstract— Light detection and ranging (LiDAR) plays an preservation of historical relics, 3D sensing for smart city, indispensable role in autonomous driving technologies, such as well as autonomous driving. Especially for autonomous as localization, map building, navigation and object avoidance. driving systems, LiDAR sensors play an indispensable role However, due to the vast amount of data, transmission and storage could become an important bottleneck. In this article, in a large number of key techniques, such as simultaneous we propose a novel compression architecture for multi-line localization and mapping (SLAM) [1], path planning [2], LiDAR point cloud sequences based on clustering and convolu- obstacle avoidance [3], and navigation. A point cloud consists tional long short-term memory (LSTM) networks. LiDAR point of a set of individual 3D points, in accordance with one or clouds are structured, which provides an opportunity to convert more attributes (color, reflectance, surface normal, etc). For the 3D data to 2D array, represented as range images. Thus, we cast the 3D point clouds compression as a range image instance, the Velodyne HDL-64E LiDAR sensor generates a sequence compression problem. Inspired by the high efficiency point cloud of up to 2.2 billion points per second, with a video coding (HEVC) algorithm, we design a novel compression range of up to 120 m.