UCC Library and UCC researchers have made this item openly available. Please let us know how this has helped you. Thanks! Title Content-aware compression for big textual data analysis Author(s) Dong, Dapeng Publication date 2016 Original citation Dong, D. 2016. Content-aware compression for big textual data analysis. PhD Thesis, University College Cork. Type of publication Doctoral thesis Rights © 2016, Dapeng Dong. http://creativecommons.org/licenses/by-nc-nd/3.0/ Embargo information No embargo required Item downloaded http://hdl.handle.net/10468/2697 from Downloaded on 2021-10-10T20:32:08Z Content-aware Compression for Big Textual Data Analysis Dapeng Dong MSC Thesis submitted for the degree of Doctor of Philosophy ¡ NATIONAL UNIVERSITY OF IRELAND, CORK FACULTY OF SCIENCE DEPARTMENT OF COMPUTER SCIENCE May 2016 Head of Department: Prof. Cormac J. Sreenan Supervisors: Dr. John Herbert Prof. Cormac J. Sreenan Contents Contents List of Figures . iv List of Tables . vii Notation . viii Acknowledgements . .x Abstract . xi 1 Introduction 1 1.1 The State of Big Data . .1 1.2 Big Data Development . .2 1.3 Our Approach . .4 1.4 Thesis Structure . .8 1.5 General Convention . .9 1.6 Publications . 10 2 Background Research 11 2.1 Big Data Organization in Hadoop . 11 2.2 Conventional Compression Schemes . 15 2.2.1 Transformation . 15 2.2.2 Modeling . 17 2.2.3 Encoding . 22 2.3 Random Access Compression Scheme . 23 2.3.1 Context-free Compression . 23 2.3.2 Self-synchronizing Codes . 25 2.3.3 Indexing . 26 2.4 Emerging Techniques for Big Data Compression . 29 2.4.1 Probabilistic Data Structures . 31 2.4.2 Data De-Duplication . 32 2.4.3 Domain-specific Compression . 32 2.5 Summary . 34 3 Content-aware Partial Compression (CaPC) 36 3.1 Compression Scheme . 37 3.1.1 Compression Model . 39 3.1.2 Coding Scheme . 40 3.1.3 Sampling Strategy . 42 3.1.4 Algorithm Efficiency . 43 3.2 Evaluation . 44 3.2.1 Performance . 46 3.2.2 Further Compression . 48 3.2.3 Summary . 50 3.2.4 CaPC Characteristics . 50 3.3 Conclusion . 51 4 Approximated Huffman Compression (AHC) 53 4.1 Compression Scheme . 55 4.1.1 Compression Model . 56 4.1.2 Huffman Tree Upgrading . 57 4.1.3 Algorithm Efficiency . 59 Content-aware Compression for Big Textual Data i Dapeng Dong Analysis Contents 4.1.4 AHC Block Packaging . 65 4.1.5 AHC Hadoop Extension . 68 4.2 Evaluation . 68 4.2.1 Performance . 69 4.2.2 Effectiveness of Sampling . 71 4.2.3 Upgrading Effect . 72 4.2.4 Further Compression . 73 4.2.5 AHC Characteristics . 74 4.3 Conclusion . 76 5 Record-aware Compression (RaC) 78 5.1 Compression Scheme . 81 5.2 Evaluation . 85 5.2.1 Performance . 88 5.2.2 Compression Speed and Ratio . 91 5.2.3 Further Compression . 95 5.3 Conclusion . 96 6 Record-aware Partial Compression (RaPC) 97 6.1 Compression Scheme . 99 6.1.1 RaPC Layer-1 Compression Scheme . 99 6.1.2 RaPC Layer-2 Compression Scheme . 101 6.2 RaPC on Hadoop . 107 6.3 Evaluation . 109 6.3.1 Performance . 109 6.3.2 RaPC-L2 Block Size Effect . 122 6.3.3 Further Compression . 125 6.3.4 RaPC Characteristics . 126 6.3.4.1 Compression Ratio . 126 6.3.4.2 Compression Speed . 128 6.4 Conclusion . 130 7 Conclusion 132 7.1 Contributions . 133 7.2 Future Directions . 136 7.3 Epilogue . 137 A Experimental Environment A1 B AHC Codeword Example B1 C MapReduce with RaPC Demonstrated C1 D ASCII Table D1 E Efficient Cloud Server Consolidation Supported by Online Data Analysis Services E1 E.1 System Architecture . E2 E.2 Problem Formulation . E3 E.2.1 SLA Violation Minimization . E5 E.3 Evaluation . E7 Content-aware Compression for Big Textual Data ii Dapeng Dong Analysis Contents E.3.1 Algorithm Performance . E8 E.3.2 RDSS System Performance . E11 E.4 Conclusion . E14 Content-aware Compression for Big Textual Data iii Dapeng Dong Analysis List of Figures List of Figures 1.1 An illustration of compression as employed by Hadoop and MapReduce.4 1.2 An illustration of the conceptual architecture. .6 2.1 Data organization in HDFS and Map processing of MapReduce. 12 2.2 Data organization in HDFS and Reduce processing of MapReduce. 14 2.3 An example of Burrows-Wheeler transformation. 16 2.4 An example of Burrows-Wheeler inverse transformation. 16 2.5 An example of adaptive arithmetic coding step-by-step. 19 2.6 An example of adaptive arithmetic model evolution. 19 2.7 Classification of context-free compression schemes. 24 2.8 An example of Signature File indexing. 27 2.9 An example of using Rank and Select for random access in Wavelet Tree. 28 3.1 Content-aware Partial Compression architecture overview . 38 3.2 Examples of using CaPC in a MapReduce program. 38 3.3 An example of CaPC compressed data comparing to the original data. 39 3.4 An illustration of CaPC data sampling strategy. 43 3.5 Statistics on HDFS I/O and network I/O for 5-Gram and WordCount jobs with original data and CaPC compressed data, respectively. 47 3.6 Statistics on Disk I/O and CPU utilization (DataNodes) for 5-Gram and WordCount jobs with original data and CaPC compressed data, respec- tively. 49 3.7 Applying compression on original data and CaPC compressed data. 50 4.1 An example of a Huffman code table generated based on a semi-static compression model through a full-scan. 58 4.2 An example of Huffman code table generated based on a semi-static compression model through sampling. 59 4.3 An example of the upgraded Huffman code table generated based on a semi-static compression model through sampling. 60 4.4 An illustration of AHC lookup-table with Huffman tree. 63 4.5 An example of AHC lookup-table. 64 4.6 An illustration of broken codes in AHC. 66 4.7 AHC block-packaging scheme. 67 4.8 Comparison of AHC compression speed using sampling methods with the default sampling rate and full-scan methods. 71 4.9 Sampling rate effects on compression ratio. 72 4.10 Huffman upgrading effects on compression ratio. 72 4.11 Evaluation of the impacts on applying a further compression to the AHC compressed data. 73 4.12 AHC compression speed comparing to Hadoop supported algorithms with the default compression level configured for Bzip, Gzip and LZO. 74 4.13 AHC decompression speed comparing to Hadoop supported algorithms with the default compression level configured for Bzip, Gzip and LZO. 75 4.14 AHC compression ratio comparing to Hadoop supported algorithms with the default compression level configured for Bzip, Gzip and LZO.... 76 Content-aware Compression for Big Textual Data iv Dapeng Dong Analysis List of Figures 5.1 An illustration of the issues with none content-aware compression in HDFS. 80 5.2 An illustration of the default work-flow of MapReduce. 82 5.3 Three strategies designed for using RaC compressed data with MapRe- duce programs. 83 5.4 RaC block-packaging scheme. 85 5.5 Aggregated CPU time and total time consumed by both Map and Reduce tasks. 89 5.6 Averaged time allocation for MapReduce Map, Shuffle, Merge, and Re- duce tasks. 90 5.7 RaC compression speed comparing to Hadoop supported algorithms with highest compression level.