Corpus for Comparing Compression Methods and an Extension of a Excom Library
Total Page：16
File Type：pdf, Size：1020Kb
Czech Technical University in Prague Faculty of Electrical Engineering Department of Computer Science and Engineering Master's Thesis Corpus for comparing compression methods and an extension of a ExCom library Bc. Jakub Rezn´ıˇcekˇ Supervisor: doc. Ing. Jan Holub, Ph.D. Study Programme: Electrical Engineering and Information Technology Field of Study: Computer Science and Engineering May 2010 iv Acknowledgements I would especially like to thank my supervisor, doc. Ing. Jan Holub, Ph.D., for the idea and for his invaluable contributions to this thesis. I would also like to thank my family and my beloved girlfriend Lucie for their support and encouragement, and Bc. OndˇrejSkaliˇcka for his quality comments. v vi Declaration I hereby declare that I have completed this thesis independently and that I have listed all the literature and publications used. I have no objection to usage of this work in compliance with the act x 60 under Act No. 121/2000, Coll. (copyright law), and with the rights connected with the copyright act including the changes in the act. In Litv´ınov on May 14, 2010 ............................................................. vii viii Abstract This thesis deals with the development of a new corpus whose purpose is to evaluate the lossless compression algorithms. We also propose a design of a methodology to maintain its timeliness. The methodology is conceived as a list of steps which are necessary to update a corpus. The efficiency and validity of the corpus, called the Prague Corpus, is verified by thorough ex- periments using the various compression techniques. This includes the representatives of the statistical and dictionary based methods which were implemented and consequently adapted for the universal library of compression algorithms called ExCom. The second part of the thesis deals with an efficient implementation of all methods which has been researched and described in detail. Abstrakt Tato pr´acese zab´yv´av´yvojem nov´ekorpusu, jehoˇz´uˇcelemje porovn´an´ıbezeztr´atov´ych kom- presn´ıch algoritm˚u. Souˇcasnˇepˇredkl´ad´amen´avrhmetodiky pro zachov´an´ıjeho aktu´alnosti. Tato metodika je koncipov´anajako seznam krok˚u,kter´ejsou pro aktualizaci nezbytn´e. Uˇcinnost´ a platnost korpusu, kter´ybyl pojmenov´anjako Praˇzsk´yKorpus, je ovˇeˇrenapomoc´ıpodrobn´ych experiment˚uza vyuˇzit´ır˚uzn´ych kompresn´ıch technik. Mezi tyto techniky patˇr´ız´astupci stati- stick´ych a slovn´ıkov´ych metod, kter´ebyly implementov´any a n´aslednˇeupraveny pro univerz´aln´ı knihovnu kompresn´ıch algoritm˚uExCom. Druh´aˇc´astpr´ace je vˇenov´anaefektivn´ıimplementaci vˇsech metod, kter´ebyly detailnˇeprozkoum´any a pops´any. ix x Contents List of Figures xv List of Tables xvii 1 Introduction1 1.1 History........................................1 1.2 Motivation......................................1 1.3 Main goals......................................2 1.4 ExCom library....................................2 1.4.1 Purpose....................................2 1.4.2 Description..................................3 1.5 Thesis organisation..................................3 2 Data compression5 2.1 Introduction......................................5 2.2 Basic concepts and definitions............................5 2.3 Entropy and redundancy...............................7 2.4 A classification of methods.............................8 2.4.1 Statistical methods..............................8 2.4.2 Dictionary-based methods..........................9 2.4.3 Other methods................................9 2.5 A classification of adaptivity............................ 10 2.6 Compression methods implemented in this work.................. 10 2.6.1 Shannon-Fano coding............................ 11 2.6.2 Static Huffman coding............................ 13 2.6.2.1 The sibling property........................ 15 2.6.2.2 Comparison of Static Huffman coding and Shannon-Fano ap- proach............................... 15 2.6.3 Dynamic Huffman coding.......................... 17 2.6.3.1 FGK algorithm........................... 18 2.6.3.2 Vitter's method.......................... 18 2.6.4 LZ77...................................... 18 2.6.4.1 Decompression algorithm..................... 22 2.6.5 LZSS..................................... 22 2.6.5.1 Techniques to improve the speed performance......... 24 2.6.6 LZ78...................................... 25 2.6.7 LZW...................................... 28 2.7 Summary....................................... 31 3 Implementation 33 xi 3.1 Implementation details................................ 33 3.2 Supporting classes.................................. 33 3.2.1 Universal codes................................ 33 3.2.2 Sliding window................................ 34 3.3 Statistical methods.................................. 35 3.3.1 Shannon-fano coding............................. 35 3.3.2 Static Huffman coding............................ 36 3.3.3 Dymamic Huffman coding.......................... 37 3.4 Dictionary methods................................. 37 3.4.1 LZ77 & LZSS................................. 37 3.4.2 LZ78 & LZW................................. 38 3.4.2.1 Dictionary search using a hash function............. 38 3.4.2.2 Dictionary search using a binary tree.............. 39 3.5 Additional notes................................... 41 4 A new corpus 43 4.1 General purpose................................... 43 4.2 Existing corpora................................... 43 4.2.1 Calgary Corpus................................ 43 4.2.2 Canterbury Corpus.............................. 44 4.2.3 Silesia Corpus................................. 46 4.2.4 Other corpora................................. 46 4.3 Data file classes analysis............................... 47 4.3.1 Text and binary files............................. 47 4.3.2 Commonly used data files.......................... 47 4.4 The design of methodology............................. 48 4.4.1 Obtaining data and further preprocessing................. 49 4.4.2 Post-processing of results.......................... 49 4.5 Corpus files...................................... 50 4.6 Summary....................................... 51 5 Experimental measurements 53 5.1 Experimental details................................. 53 5.2 Performance experiments.............................. 54 5.2.1 Canterbury Corpus.............................. 54 5.2.2 Prague Corpus................................ 57 5.3 Parameter experiments................................ 61 5.4 Existing methods................................... 62 5.5 Summary....................................... 64 6 Conclusion 67 6.1 Future work...................................... 67 References 69 A The Corpus methodology 73 B The Corpus report 77 C Prague corpus files 79 D Public domain data sources 85 xii E Detailed results of experiments 87 F User manual 95 G List of abbreviations 97 H Contents of DVD 99 xiii xiv List of Figures 2.1 Static model...................................... 10 2.2 Semi-adaptive model................................. 10 2.3 Adaptive model.................................... 11 2.4 Shannon-Fano algorithm in action......................... 12 2.5 Pseudocode of Shannon-Fano algorithm...................... 13 2.6 The particular steps of building Huffman tree................... 14 2.7 The sibling property................................. 16 2.8 Pseudocode of Huffman algorithm......................... 17 2.9 Pseudocode of adaptive Huffman coding...................... 19 2.10 Example of LZ77 Sliding window.......................... 21 2.11 Pseudocode of LZ77 encoding process....................... 23 2.12 Pseudocode of LZSS encoding process....................... 24 2.13 Pseudocode of LZ78 encoding process....................... 27 2.14 The first four steps of building the LZ78 dictionary................ 28 2.15 The final LZ78 dictionary tree............................ 29 2.16 Pseudocode of LZW compression method..................... 30 2.17 The LZW dictionary after its initialization..................... 31 2.18 The final form of LZW dictionary tree with an additional table......... 32 3.1 Preorder traversal of the Huffman tree....................... 36 3.2 The usage of sBufferIndices............................. 37 3.3 Example of LZW dictionary............................. 40 3.4 Illustration of tree nodes with the corresponding relations............ 40 4.1 Scatter plot for CPP source codes......................... 50 5.1 Compression and decompression times of the statistical methods........ 55 5.2 Compression times of the dictionary methods................... 56 5.3 Decompression times of the dictionary methods.................. 57 5.4 Compression times of all methods on the files from the Prague Corpus (set A). 59 5.5 Compression ratios of the all implemented methods on the files from the Prague Corpus (set A).................................... 60 5.6 Compression ratio of LZ78 depeneding on the dictionary size.......... 62 5.7 Compression time and ratio depending on the logarithm of the LZSS Search buffer size....................................... 63 5.8 Compression times of the context methods on the files from the Prague Corpus (set B)......................................... 65 E.1 Compression ratio of the dictionary methods on the files from the Canterbury files 87 E.2 Compression times of the statistical methods on the files from the Prague Corpus (set B)......................................... 90 xv E.3 Compression and decompression times of the dictionary methods on the files from the Prague Corpus (set