DOCSLIB.ORG

Chapter 5 Record-Aware Compression

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Chapter 5 Record-Aware Compression UCC Library and UCC researchers have made this item openly available. Please let us know how this has helped you. Thanks! Title Content-aware compression for big textual data analysis Author(s) Dong, Dapeng Publication date 2016 Original citation Dong, D. 2016. Content-aware compression for big textual data analysis. PhD Thesis, University College Cork. Type of publication Doctoral thesis Rights © 2016, Dapeng Dong. http://creativecommons.org/licenses/by-nc-nd/3.0/ Embargo information No embargo required Item downloaded http://hdl.handle.net/10468/2697 from Downloaded on 2021-10-10T20:32:08Z Content-aware Compression for Big Textual Data Analysis Dapeng Dong MSC Thesis submitted for the degree of Doctor of Philosophy ¡ NATIONAL UNIVERSITY OF IRELAND, CORK FACULTY OF SCIENCE DEPARTMENT OF COMPUTER SCIENCE May 2016 Head of Department: Prof. Cormac J. Sreenan Supervisors: Dr. John Herbert Prof. Cormac J. Sreenan Contents Contents List of Figures . iv List of Tables . vii Notation . viii Acknowledgements . .x Abstract . xi 1 Introduction 1 1.1 The State of Big Data . .1 1.2 Big Data Development . .2 1.3 Our Approach . .4 1.4 Thesis Structure . .8 1.5 General Convention . .9 1.6 Publications . 10 2 Background Research 11 2.1 Big Data Organization in Hadoop . 11 2.2 Conventional Compression Schemes . 15 2.2.1 Transformation . 15 2.2.2 Modeling . 17 2.2.3 Encoding . 22 2.3 Random Access Compression Scheme . 23 2.3.1 Context-free Compression . 23 2.3.2 Self-synchronizing Codes . 25 2.3.3 Indexing . 26 2.4 Emerging Techniques for Big Data Compression . 29 2.4.1 Probabilistic Data Structures . 31 2.4.2 Data De-Duplication . 32 2.4.3 Domain-specific Compression . 32 2.5 Summary . 34 3 Content-aware Partial Compression (CaPC) 36 3.1 Compression Scheme . 37 3.1.1 Compression Model . 39 3.1.2 Coding Scheme . 40 3.1.3 Sampling Strategy . 42 3.1.4 Algorithm Efficiency . 43 3.2 Evaluation . 44 3.2.1 Performance . 46 3.2.2 Further Compression . 48 3.2.3 Summary . 50 3.2.4 CaPC Characteristics . 50 3.3 Conclusion . 51 4 Approximated Huffman Compression (AHC) 53 4.1 Compression Scheme . 55 4.1.1 Compression Model . 56 4.1.2 Huffman Tree Upgrading . 57 4.1.3 Algorithm Efficiency . 59 Content-aware Compression for Big Textual Data i Dapeng Dong Analysis Contents 4.1.4 AHC Block Packaging . 65 4.1.5 AHC Hadoop Extension . 68 4.2 Evaluation . 68 4.2.1 Performance . 69 4.2.2 Effectiveness of Sampling . 71 4.2.3 Upgrading Effect . 72 4.2.4 Further Compression . 73 4.2.5 AHC Characteristics . 74 4.3 Conclusion . 76 5 Record-aware Compression (RaC) 78 5.1 Compression Scheme . 81 5.2 Evaluation . 85 5.2.1 Performance . 88 5.2.2 Compression Speed and Ratio . 91 5.2.3 Further Compression . 95 5.3 Conclusion . 96 6 Record-aware Partial Compression (RaPC) 97 6.1 Compression Scheme . 99 6.1.1 RaPC Layer-1 Compression Scheme . 99 6.1.2 RaPC Layer-2 Compression Scheme . 101 6.2 RaPC on Hadoop . 107 6.3 Evaluation . 109 6.3.1 Performance . 109 6.3.2 RaPC-L2 Block Size Effect . 122 6.3.3 Further Compression . 125 6.3.4 RaPC Characteristics . 126 6.3.4.1 Compression Ratio . 126 6.3.4.2 Compression Speed . 128 6.4 Conclusion . 130 7 Conclusion 132 7.1 Contributions . 133 7.2 Future Directions . 136 7.3 Epilogue . 137 A Experimental Environment A1 B AHC Codeword Example B1 C MapReduce with RaPC Demonstrated C1 D ASCII Table D1 E Efficient Cloud Server Consolidation Supported by Online Data Analysis Services E1 E.1 System Architecture . E2 E.2 Problem Formulation . E3 E.2.1 SLA Violation Minimization . E5 E.3 Evaluation . E7 Content-aware Compression for Big Textual Data ii Dapeng Dong Analysis Contents E.3.1 Algorithm Performance . E8 E.3.2 RDSS System Performance . E11 E.4 Conclusion . E14 Content-aware Compression for Big Textual Data iii Dapeng Dong Analysis List of Figures List of Figures 1.1 An illustration of compression as employed by Hadoop and MapReduce.4 1.2 An illustration of the conceptual architecture. .6 2.1 Data organization in HDFS and Map processing of MapReduce. 12 2.2 Data organization in HDFS and Reduce processing of MapReduce. 14 2.3 An example of Burrows-Wheeler transformation. 16 2.4 An example of Burrows-Wheeler inverse transformation. 16 2.5 An example of adaptive arithmetic coding step-by-step. 19 2.6 An example of adaptive arithmetic model evolution. 19 2.7 Classification of context-free compression schemes. 24 2.8 An example of Signature File indexing. 27 2.9 An example of using Rank and Select for random access in Wavelet Tree. 28 3.1 Content-aware Partial Compression architecture overview . 38 3.2 Examples of using CaPC in a MapReduce program. 38 3.3 An example of CaPC compressed data comparing to the original data. 39 3.4 An illustration of CaPC data sampling strategy. 43 3.5 Statistics on HDFS I/O and network I/O for 5-Gram and WordCount jobs with original data and CaPC compressed data, respectively. 47 3.6 Statistics on Disk I/O and CPU utilization (DataNodes) for 5-Gram and WordCount jobs with original data and CaPC compressed data, respec- tively. 49 3.7 Applying compression on original data and CaPC compressed data. 50 4.1 An example of a Huffman code table generated based on a semi-static compression model through a full-scan. 58 4.2 An example of Huffman code table generated based on a semi-static compression model through sampling. 59 4.3 An example of the upgraded Huffman code table generated based on a semi-static compression model through sampling. 60 4.4 An illustration of AHC lookup-table with Huffman tree. 63 4.5 An example of AHC lookup-table. 64 4.6 An illustration of broken codes in AHC. 66 4.7 AHC block-packaging scheme. 67 4.8 Comparison of AHC compression speed using sampling methods with the default sampling rate and full-scan methods. 71 4.9 Sampling rate effects on compression ratio. 72 4.10 Huffman upgrading effects on compression ratio. 72 4.11 Evaluation of the impacts on applying a further compression to the AHC compressed data. 73 4.12 AHC compression speed comparing to Hadoop supported algorithms with the default compression level configured for Bzip, Gzip and LZO. 74 4.13 AHC decompression speed comparing to Hadoop supported algorithms with the default compression level configured for Bzip, Gzip and LZO. 75 4.14 AHC compression ratio comparing to Hadoop supported algorithms with the default compression level configured for Bzip, Gzip and LZO.... 76 Content-aware Compression for Big Textual Data iv Dapeng Dong Analysis List of Figures 5.1 An illustration of the issues with none content-aware compression in HDFS. 80 5.2 An illustration of the default work-flow of MapReduce. 82 5.3 Three strategies designed for using RaC compressed data with MapRe- duce programs. 83 5.4 RaC block-packaging scheme. 85 5.5 Aggregated CPU time and total time consumed by both Map and Reduce tasks. 89 5.6 Averaged time allocation for MapReduce Map, Shuffle, Merge, and Re- duce tasks. 90 5.7 RaC compression speed comparing to Hadoop supported algorithms with highest compression level.
Load more

Recommended publications
  • Data Preparation & Descriptive Statistics
    Data Preparation & Descriptive Statistics (ver. 2.4) Oscar Torres-Reyna Data Consultant [email protected] PU/DSS/OTR http://dss.princeton.edu/training/ Basic definitions… For statistical analysis we think of data as a collection of different pieces of information or facts. These pieces of information are called variables. A variable is an identifiable piece of data containing one or more values. Those values can take the form of a number or text (which could be converted into number) In the table below variables var1 thru var5 are a collection of seven values, ‘id’ is the identifier for each observation. This dataset has information for seven cases (in this case people, but could also be states, countries, etc) grouped into five variables. id var1 var2 var3 var4 var5 1 7.3 32.27 0.1 Yes Male 2 8.28 40.68 0.56 No Female 3 3.35 5.62 0.55 Yes Female 4 4.08 62.8 0.83 Yes Male 5 9.09 22.76 0.26 No Female 6 8.15 90.85 0.23 Yes Female 7 7.59 54.94 0.42 Yes Male PU/DSS/OTR Data structure… For data analysis your data should have variables as columns and observations as rows. The first row should have the column headings. Make sure your dataset has at least one identifier (for example, individual id, family id, etc.) id var1 var2 var3 var4 var5 First row should have the variable names 1 7.3 32.27 0.1 Yes Male 2 8.28 40.68 0.56 No Female Cross-sectional data 3 3.35 5.62 0.55 Yes Female 4 4.08 62.8 0.83 Yes Male 5 9.09 22.76 0.26 No Female 6 8.15 90.85 0.23 Yes Female 7 7.59 54.94 0.42 Yes Male id year var1 var2 var3 1 2000 7 74.03 0.55 Group 1 1 2001 2 4.6 0.44 At least one identifier 1 2002 2 25.56 0.77 2 2000 7 59.52 0.05 Cross-sectional time series data Group 2 2 2001 2 16.95 0.94 or panel data 2 2002 9 1.2 0.08 3 2000 9 85.85 0.5 Group 3 3 2001 3 98.85 0.32 3 2002 3 69.2 0.76 PU/DSS/OTR NOTE: See: http://www.statistics.com/resources/glossary/c/crossdat.php Data format (ASCII)… ASCII (American Standard Code for Information Interchange).
    [Show full text]
  • Full Document
    R&D Centre for Mobile Applications (RDC) FEE, Dept of Telecommunications Engineering Czech Technical University in Prague RDC Technical Report TR-13-4 Internship report Evaluation of Compressibility of the Output of the Information-Concealing Algorithm Julien Mamelli, [email protected] 2nd year student at the Ecole´ des Mines d'Al`es (N^ımes,France) Internship supervisor: Luk´aˇsKencl, [email protected] August 2013 Abstract Compression is a key element to exchange files over the Internet. By generating re- dundancies, the concealing algorithm proposed by Kencl and Loebl [?], appears at first glance to be particularly designed to be combined with a compression scheme [?]. Is the output of the concealing algorithm actually compressible? We have tried 16 compression techniques on 1 120 files, and the result is that we have not found a solution which could advantageously use repetitions of the concealing method. Acknowledgments I would like to express my gratitude to my supervisor, Dr Luk´aˇsKencl, for his guidance and expertise throughout the course of this work. I would like to thank Prof. Robert Beˇst´akand Mr Pierre Runtz, for giving me the opportunity to carry out my internship at the Czech Technical University in Prague. I would also like to thank all the members of the Research and Development Center for Mobile Applications as well as my colleagues for the assistance they have given me during this period. 1 Contents 1 Introduction 3 2 Related Work 4 2.1 Information concealing method . 4 2.2 Archive formats . 5 2.3 Compression algorithms . 5 2.3.1 Lempel-Ziv algorithm .
    [Show full text]
  • Pack, Encrypt, Authenticate Document Revision: 2021 05 02
    PEA Pack, Encrypt, Authenticate Document revision: 2021 05 02 Author: Giorgio Tani Translation: Giorgio Tani This document refers to: PEA file format specification version 1 revision 3 (1.3); PEA file format specification version 2.0; PEA 1.01 executable implementation; Present documentation is released under GNU GFDL License. PEA executable implementation is released under GNU LGPL License; please note that all units provided by the Author are released under LGPL, while Wolfgang Ehrhardt’s crypto library units used in PEA are released under zlib/libpng License. PEA file format and PCOMPRESS specifications are hereby released under PUBLIC DOMAIN: the Author neither has, nor is aware of, any patents or pending patents relevant to this technology and do not intend to apply for any patents covering it. As far as the Author knows, PEA file format in all of it’s parts is free and unencumbered for all uses. Pea is on PeaZip project official site: https://peazip.github.io , https://peazip.org , and https://peazip.sourceforge.io For more information about the licenses: GNU GFDL License, see http://www.gnu.org/licenses/fdl.txt GNU LGPL License, see http://www.gnu.org/licenses/lgpl.txt 1 Content: Section 1: PEA file format ..3 Description ..3 PEA 1.3 file format details ..5 Differences between 1.3 and older revisions ..5 PEA 2.0 file format details ..7 PEA file format’s and implementation’s limitations ..8 PCOMPRESS compression scheme ..9 Algorithms used in PEA format ..9 PEA security model .10 Cryptanalysis of PEA format .12 Data recovery from
    [Show full text]
  • Steganography and Vulnerabilities in Popular Archives Formats.| Nyxengine Nyx.Reversinglabs.Com
    Hiding in the Familiar: Steganography and Vulnerabilities in Popular Archives Formats.| NyxEngine nyx.reversinglabs.com Contents Introduction to NyxEngine ............................................................................................................................ 3 Introduction to ZIP file format ...................................................................................................................... 4 Introduction to steganography in ZIP archives ............................................................................................. 5 Steganography and file malformation security impacts ............................................................................... 8 References and tools .................................................................................................................................... 9 2 Introduction to NyxEngine Steganography1 is the art and science of writing hidden messages in such a way that no one, apart from the sender and intended recipient, suspects the existence of the message, a form of security through obscurity. When it comes to digital steganography no stone should be left unturned in the search for viable hidden data. Although digital steganography is commonly used to hide data inside multimedia files, a similar approach can be used to hide data in archives as well. Steganography imposes the following data hiding rule: Data must be hidden in such a fashion that the user has no clue about the hidden message or file's existence. This can be achieved by
    [Show full text]
  • Winzip 12 Reviewer's Guide
    Introducing WinZip® 12 WinZip® is the most trusted way to work with compressed files. No other compression utility is as easy to use or offers the comprehensive and productivity-enhancing approach that has made WinZip the gold standard for file-compression tools. With the new WinZip 12, you can quickly and securely zip and unzip files to conserve storage space, speed up e-mail transmission, and reduce download times. State-of-the-art file compression, strong AES encryption, compatibility with more compression formats, and new intuitive photo compression, make WinZip 12 the complete compression and archiving solution. Building on the favorite features of a worldwide base of several million users, WinZip 12 adds new features for image compression and management, support for new compression methods, improved compression performance, support for additional archive formats, and more. Users can work smarter, faster, and safer with WinZip 12. Who will benefit from WinZip® 12? The simple answer is anyone who uses a PC. Any PC user can benefit from the compression and encryption features in WinZip to protect data, save space, and reduce the time to transfer files on the Internet. There are, however, some PC users to whom WinZip is an even more valuable and essential tool. Digital photo enthusiasts: As the average file size of their digital photos increases, people are looking for ways to preserve storage space on their PCs. They have lots of photos, so they are always seeking better ways to manage them. Sharing their photos is also important, so they strive to simplify the process and reduce the time of e-mailing large numbers of images.
    [Show full text]
  • The Ark Handbook
    The Ark Handbook Matt Johnston Henrique Pinto Ragnar Thomsen The Ark Handbook 2 Contents 1 Introduction 5 2 Using Ark 6 2.1 Opening Archives . .6 2.1.1 Archive Operations . .6 2.1.2 Archive Comments . .6 2.2 Working with Files . .7 2.2.1 Editing Files . .7 2.3 Extracting Files . .7 2.3.1 The Extract dialog . .8 2.4 Creating Archives and Adding Files . .8 2.4.1 Compression . .9 2.4.2 Password Protection . .9 2.4.3 Multi-volume Archive . 10 3 Using Ark in the Filemanager 11 4 Advanced Batch Mode 12 5 Credits and License 13 Abstract Ark is an archive manager by KDE. The Ark Handbook Chapter 1 Introduction Ark is a program for viewing, extracting, creating and modifying archives. Ark can handle vari- ous archive formats such as tar, gzip, bzip2, zip, rar, 7zip, xz, rpm, cab, deb, xar and AppImage (support for certain archive formats depends on the appropriate command-line programs being installed). In order to successfully use Ark, you need KDE Frameworks 5. The library libarchive version 3.1 or above is needed to handle most archive types, including tar, compressed tar, rpm, deb and cab archives. To handle other file formats, you need the appropriate command line programs, such as zipinfo, zip, unzip, rar, unrar, 7z, lsar, unar and lrzip. 5 The Ark Handbook Chapter 2 Using Ark 2.1 Opening Archives To open an archive in Ark, choose Open... (Ctrl+O) from the Archive menu. You can also open archive files by dragging and dropping from Dolphin.
    [Show full text]
  • How to 'Zip and Unzip' Files
    How to 'zip and unzip' files The Windows 10 operating system provides a very easy way to zip-up any file (or folder) you want by using a utility program called 7-zip (Seven Zip). The program is integrated in the context menu which you get when you right-click on anything selected. Here are the basic steps you need to take in order to: Zip a file and create an archive 1. Open a folder in your machine and select any file (by clicking on it once). I selected a single file called 'how-to send an email.docx' 2. Now right click the mouse to have the context menu appear 3. In the context menu you should be able to see some commands like the capture below 4. Since we want to zip up the file you need to select one of the bottom two options a. 'Add to archive' will actually open up a dialog of the 7-zip application and will give you the opportunity to customise the archive. b. 'Add to how-to send an email.zip' is actually the quickest way possible to create an archive. The software uses the name of the file and selects a default compression scheme (.zip) so that you can, with two clicks, create a zip archive containing the original file. 5. Now you can use the 'how-to send an email.zip' file and send it as a 'smaller' email attachment. Now consider that you have just received (via email) the 'how-to send an email.zip' file and you need to get (the correct phrase is extract) the file it contains.
    [Show full text]
  • Rapc) 97 6.1 Compression Scheme
    UCC Library and UCC researchers have made this item openly available. Please let us know how this has helped you. Thanks! Title Content-aware compression for big textual data analysis Author(s) Dong, Dapeng Publication date 2016 Original citation Dong, D. 2016. Content-aware compression for big textual data analysis. PhD Thesis, University College Cork. Type of publication Doctoral thesis Rights © 2016, Dapeng Dong. http://creativecommons.org/licenses/by-nc-nd/3.0/ Embargo information No embargo required Item downloaded http://hdl.handle.net/10468/2697 from Downloaded on 2021-10-11T16:19:07Z Content-aware Compression for Big Textual Data Analysis Dapeng Dong MSC Thesis submitted for the degree of Doctor of Philosophy ¡ NATIONAL UNIVERSITY OF IRELAND, CORK FACULTY OF SCIENCE DEPARTMENT OF COMPUTER SCIENCE May 2016 Head of Department: Prof. Cormac J. Sreenan Supervisors: Dr. John Herbert Prof. Cormac J. Sreenan Contents Contents List of Figures . iv List of Tables . vii Notation . viii Acknowledgements . .x Abstract . xi 1 Introduction 1 1.1 The State of Big Data . .1 1.2 Big Data Development . .2 1.3 Our Approach . .4 1.4 Thesis Structure . .8 1.5 General Convention . .9 1.6 Publications . 10 2 Background Research 11 2.1 Big Data Organization in Hadoop . 11 2.2 Conventional Compression Schemes . 15 2.2.1 Transformation . 15 2.2.2 Modeling . 17 2.2.3 Encoding . 22 2.3 Random Access Compression Scheme . 23 2.3.1 Context-free Compression . 23 2.3.2 Self-synchronizing Codes . 25 2.3.3 Indexing . 26 2.4 Emerging Techniques for Big Data Compression .
    [Show full text]
  • Deduplicating Compressed Contents in Cloud Storage Environment
    Deduplicating Compressed Contents in Cloud Storage Environment Zhichao Yan, Hong Jiang Yujuan Tan* Hao Luo University of Texas Arlington Chongqing University University of Nebraska Lincoln [email protected] [email protected] [email protected] [email protected] Corresponding Author Abstract Data compression and deduplication are two common approaches to increasing storage efficiency in the cloud environment. Both users and cloud service providers have economic incentives to compress their data before storing it in the cloud. However, our analysis indicates that compressed packages of different data and differ- ently compressed packages of the same data are usual- ly fundamentally different from one another even when they share a large amount of redundant data. Existing data deduplication systems cannot detect redundant data among them. We propose the X-Ray Dedup approach to extract from these packages the unique metadata, such as the “checksum” and “file length” information, and use it as the compressed file’s content signature to help detect and remove file level data redundancy. X-Ray Dedup is shown by our evaluations to be capable of breaking in the boundaries of compressed packages and significantly Figure 1: A user scenario on cloud storage environment reducing compressed packages’ size requirements, thus further optimizing storage space in the cloud. will generate different compressed data of the same con- tents that render fingerprint-based redundancy identifi- cation difficult. Third, very similar but different digital 1 Introduction contents (e.g., files or data streams), which would other- wise present excellent deduplication opportunities, will Due to the information explosion [1, 3], data reduc- become fundamentally distinct compressed packages af- tion technologies such as compression and deduplica- ter applying even the same compression algorithm.
    [Show full text]
  • Installing Your Cinesamples Product - Cinebells
    Installing Your Cinesamples Product - CineBells Step 1. In your Downloads folder, or in the location you have told your browser to send your downloads, you will find a file called“CineBells.zip”along with the five .rar files shown to the left. First Unzip the zip file to create your main CineBells folder. Then you will need a utility that can extract the .rar files, like RarMarchine or UnRarX for Mac, or WinRar for Windows. No matter which you use, you will only need to select and unarchive the first file (part1). The utility will automatically create a Step 2. CineBells_Samples folder, and decompress all five .rar files into it in one step. Please DO NOT use Stuffit for this - it will not work correctly. Make sure to check that all five rars downloaded completely - notice the sizes to the left. After the rars are extracted, go into the CineBells_Samples folder, and drag the Samples Step 3. folder inside it into the main CineBells folder that was created from the zip file. Next drag your final CineBells folder to your sample hard drive. It should look like the picture below afterwards. If this is your first Cinesamples product, you may want to create a Cinesamples folder beforehand. The “CineBells_Samples”folder should now be empty and can be deleted. Step 4. Next, open the full version of Kontakt 4 (at least 4.2.3) or Kontakt 5, select the Files tab, and navigate to your CineBells folder. Double click or drag nki files into the main window to load the different instruments.
    [Show full text]
  • Performance Analysis of Generic Compression Algorithm Tools Andrew Keating, Jonathon Marolf Embry-Riddle Aeronautical University
    Performance Analysis of Generic Compression Algorithm Tools Andrew Keating, Jonathon Marolf Embry-Riddle Aeronautical University Introduction Methods Results Conclusions Compression serves a very important purpose in conserving data backup space and improving the speed of digital data Archival Tool F F crit Environment • For each file type and size Best Compression Ratio - When looking at just compression RAR: 5mb 1719488 2.627441 communications. To take full advantage of compression, the Gentoo Linux Workstation with kernel version 2.6.31-r6. (i.e. text 5mb) an excel ratio 7zip wins for best audio compression overall. Info-zip and workbook was created that RAR: 25mb 119782.1 2.627441 best compression utility available for the given situation System contains an Intel Pentium 4 CPU rated at 2.5 GHZ gzip win for best compression of random data because of their should be used. Because of the inherent tradeoffs between would calculate the mean, RAR: 125mb 184026.4 2.627441 and contains 1GB of main memory rated at DDR 226. median, mode, and Info-zip: 5mb 3525.32 2.627441 small header files. 7zip wins for best compression of text data compression ratio and compression/decompression time, standard deviation for Info-zip: 25mb 3460.035 2.627441 and video data, just barely beating RAR within our confidence compression and identifying the ‘best’ utility can be troublesome. Tools Info-zip: 125mb 4588.768 2.627441 interval. Overall, for compression 7zip delivers the best decompression times as Each compression utility is meant to compress data, but Gzip: 5mb 1767.889 2.627441 7zip: version 9.04 of ‘p7zip’ well as the compression performance.
    [Show full text]
  • RAR Version 5.00
    RAR version 5.00 - Technical information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ THE ARCHIVE FORMAT DESCRIBED BELOW IS ONLY VALID FOR VERSIONS SINCE 1.50 ========================================================================== RAR archive file format ========================================================================== Archive file consists of variable length blocks. The order of these blocks may vary, but the first block must be a marker block followed by an archive header block. Each block begins with the following fields: HEAD_CRC 2 bytes CRC of total block or block part HEAD_TYPE 1 byte Block type HEAD_FLAGS 2 bytes Block flags HEAD_SIZE 2 bytes Block size ADD_SIZE 4 bytes Optional field - added block size Field ADD_SIZE present only if (HEAD_FLAGS & 0x8000) != 0 Total block size is HEAD_SIZE if (HEAD_FLAGS & 0x8000) == 0 and HEAD_SIZE+ADD_SIZE if the field ADD_SIZE is present - when (HEAD_FLAGS & 0x8000) != 0. In each block the followings bits in HEAD_FLAGS have the same meaning: 0x4000 - if set, older RAR versions will ignore the block and remove it when the archive is updated. if clear, the block is copied to the new archive file when the archive is updated; 0x8000 - if set, ADD_SIZE field is present and the full block size is HEAD_SIZE+ADD_SIZE. Declared block types: HEAD_TYPE=0x72 marker block HEAD_TYPE=0x73 archive header HEAD_TYPE=0x74 file header HEAD_TYPE=0x75 old style comment header HEAD_TYPE=0x76 old style authenticity information HEAD_TYPE=0x77 old style subblock HEAD_TYPE=0x78 old style recovery record HEAD_TYPE=0x79 old style authenticity information HEAD_TYPE=0x7a subblock Comment block is actually used only within other blocks and doesn't exist separately. Archive processing is made in the following manner: 1. Read and check marker block 2. Read archive header 3.
    [Show full text]