DOCSLIB.ORG

Compressed RAM for Embedded Systems

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Compressed RAM for Embedded Systems CRAMES: Compressed RAM for Embedded Systems Robert Dick http://robertdick.org/ or http://ziyang.eecs.northwestern.edu/∼dickrp/ Department of Electrical Engineering and Computer Science Northwestern University Application Application Application Swap Page Page Decomp Comp CRAMES Virtual Memory Introduction CRAMES design Experimental evaluation Collaborators on project NEC Northwestern University NEC Labs America Lei Yang Dr. Haris Lekatsas Lan S. Bai Dr. Srimat Chakradhar Robert P. Dick 2 Robert Dick CRAMES Introduction Motivation CRAMES design Past work Experimental evaluation Outline 1. Introduction Motivation Past work 2. CRAMES design Design overview Pattern-based partial match compression 3. Experimental evaluation Experimental setup Commercialization and future directions 3 Robert Dick CRAMES Introduction Motivation CRAMES design Past work Experimental evaluation Problem background RAM quantity limits application functionality RAM price dropping but usage growing faster Secure Internet access, email, music, and games How much RAM? Functionality Cost Power consumption Size 4 Robert Dick CRAMES Introduction Motivation CRAMES design Past work Experimental evaluation Ideal hardware–software design process Ideal case Hardware and software engineers collaborate on system-level design from start to finish We teach the advantages of this in our classes It doesn’t always happen 5 Robert Dick CRAMES HW engineers SW engineers Introduction Motivation CRAMES design Past work Experimental evaluation Real hardware–software design process 6 Robert Dick CRAMES HW engineers SW engineers Introduction Motivation CRAMES design Past work Experimental evaluation Real hardware–software design process 6 Robert Dick CRAMES Introduction Motivation CRAMES design Past work Experimental evaluation Options Option 1: Add more memory Implications: Hardware redesign, miss shipping target, get fired Option 2: Rip out memory-hungry application features Implications: Lose market to competitors, fail to recoup design and production costs, get fired Option 3: Make it seem as if memory increased without changing hardware, without changing applications, and without performance or power consumption penalties Nobody knew how to do this 7 Robert Dick CRAMES Introduction Motivation CRAMES design Past work Experimental evaluation Goals Allow application RAM requirements to overrun initial estimates even after hardware design Reduce physical RAM, negligible performance and energy cost Improve functionality or performance with same physical RAM 8 Robert Dick CRAMES Introduction Motivation CRAMES design Past work Experimental evaluation HW RAM compression Tremaine 2001, Benini 2002, Moore 2003 Hardware (de)compression unit between cache and RAM Hardware redesign and application-specific compression hardware Past work claimed application-specific hardware essential to keep power and performance overhead low 9 Robert Dick CRAMES Introduction Motivation CRAMES design Past work Experimental evaluation Code compression Lekatsas 2000, Xu 2004 Store code compressed, decompress during execution Compress off-line, decompress on-line For RAM, less important than on-line data compression 10 Robert Dick CRAMES Introduction Motivation CRAMES design Past work Experimental evaluation Compression for swap performance Compressed caching Douglis 1993, Russinovich 1996, Wilson 1999, Kjelso 1999 Add compressed software cache to VM Swap compression RamDoubler, Cortez 2000, Roy 2001, Chihaia 2005 Compress swapped-out pages and store them in software cache Both techniques Target: general purpose system with disks Goal: improve system performance Interface to backing store (disk) 11 Robert Dick CRAMES Introduction Design overview CRAMES design Pattern-based partial match compression Experimental evaluation Outline 1. Introduction Motivation Past work 2. CRAMES design Design overview Pattern-based partial match compression 3. Experimental evaluation Experimental setup Commercialization and future directions 12 Robert Dick CRAMES Introduction Design overview CRAMES design Pattern-based partial match compression Experimental evaluation Design principles Page selection Scheduling compression and decompression Organizing compressed and uncompressed regions Dynamically adjust compressed region size Compression scheme High performance Energy efficient Good compression ratio Low memory requirement 13 Robert Dick CRAMES Introduction Design overview CRAMES design Pattern-based partial match compression Experimental evaluation CRAMES design Page Fault Memory Low 8 1 1 8 Kernel Swapping System 7 2 2 7 CRAMES Device Driver 6 6 3 3 4 Block Memory Block Decompressor Manager Compressor 5 5 4 Virtual Memory MMU RAM 14 Robert Dick CRAMES Introduction Design overview CRAMES design Pattern-based partial match compression Experimental evaluation System configuration Slightly slower Ramdisk with Original CRAMES file system 20 MB Ramdisk Compressed RAM with file system device with file system Main memory 12 MB 12 MB working area 10 MB Main memory working area Slightly slower Main memory 10 MB main memory working area working area 20 MB Compressed 20 MB swap device 10 MB System/User view System view User view 15 Robert Dick CRAMES Compression Ratio Working Memory 7600kB 0.39 10000 3700kB 0.34 bzip2 1000 zlib default 256KB 0.29 zilb level 1 100 Compression (bytes) 64KB zlib level 9 2 44KB 0.24 Decompression RLE log 16KB 16KB LZRW-1A 10 0.19 Compression Ratio Compression LZO 00 0 0.14 1 512 1024 2048 4096 8192 LZO bzip2 zlib default LZO LZRW-1A RLE Block Size (bytes) best existing Compression Algorithms compression Compression Time algorithm for Decompression Time 0.0025 0.0012 CRAMES 0.001 bzip2 0.002 bzip2 zlib default zilb default 0.0008 zlib level 1 0.0015 zlib level 1 zlib level 9 zlib level 9 0.0006 RLE 0.001 RLE Time (sec) Time Time (sec) Time 0.0004 LZRW-1A LZRW-1A 0.0005 LZO LZO 0.0002 0 0 512 1024 2048 4096 8192 512 1024 2048 4096 8192 Block Size (bytes) Block Size (bytes) Introduction Design overview CRAMES design Pattern-based partial match compression Experimental evaluation Compression algorithm Compression Ratio Working Memory 7600kB 0.39 10000 3700kB 0.34 bzip2 1000 zlib default 256KB 0.29 zilb level 1 100 Compression (bytes) 64KB zlib level 9 2 44KB 0.24 Decompression RLE log 16KB 16KB LZRW-1A 10 0.19 Compression Ratio Compression LZO 00 0 0.14 1 512 1024 2048 4096 8192 bzip2 zlib default LZO LZRW-1A RLE Block Size (bytes) Compression Algorithms Compression Time Decompression Time 0.0025 0.0012 0.001 bzip2 0.002 bzip2 zlib default zilb default 0.0008 zlib level 1 0.0015 zlib level 1 zlib level 9 zlib level 9 0.0006 RLE 0.001 RLE Time (sec) Time Time (sec) Time 0.0004 LZRW-1A LZRW-1A 0.0005 LZO LZO 0.0002 0 0 512 1024 2048 4096 8192 512 1024 2048 4096 8192 Block Size (bytes) Block Size (bytes) 16 Robert Dick CRAMES Compression Ratio Working Memory 7600kB 0.39 10000 3700kB 0.34 bzip2 1000 zlib default 256KB 0.29 zilb level 1 100 Compression (bytes) 64KB zlib level 9 2 44KB 0.24 Decompression RLE log 16KB 16KB LZRW-1A 10 0.19 Compression Ratio Compression LZO 00 0 0.14 1 512 1024 2048 4096 8192 LZO bzip2 zlib default LZO LZRW-1A RLE Block Size (bytes) best existing Compression Algorithms compression Compression Time algorithm for Decompression Time 0.0025 0.0012 CRAMES 0.001 bzip2 0.002 bzip2 zlib default zilb default 0.0008 zlib level 1 0.0015 zlib level 1 zlib level 9 zlib level 9 0.0006 RLE 0.001 RLE Time (sec) Time Time (sec) Time 0.0004 LZRW-1A LZRW-1A 0.0005 LZO LZO 0.0002 0 0 512 1024 2048 4096 8192 512 1024 2048 4096 8192 Block Size (bytes) Block Size (bytes) Introduction Design overview CRAMES design Pattern-based partial match compression Experimental evaluation Compression algorithm Compression Ratio Working Memory 7600kB 0.39 10000 3700kB 0.34 bzip2 1000 zlib default 256KB 0.29 zilb level 1 100 Compression (bytes) 64KB zlib level 9 2 44KB 0.24 Decompression RLE log 16KB 16KB LZRW-1A 10 0.19 Compression Ratio Compression LZO 00 0 0.14 1 512 1024 2048 4096 8192 bzip2 zlib default LZO LZRW-1A RLE Block Size (bytes) Compression Algorithms Compression Time Decompression Time 0.0025 0.0012 0.001 bzip2 0.002 bzip2 zlib default zilb default 0.0008 zlib level 1 0.0015 zlib level 1 zlib level 9 zlib level 9 0.0006 RLE 0.001 RLE Time (sec) Time Time (sec) Time 0.0004 LZRW-1A LZRW-1A 0.0005 LZO LZO 0.0002 0 0 512 1024 2048 4096 8192 512 1024 2048 4096 8192 Block Size (bytes) Block Size (bytes) 16 Robert Dick CRAMES Introduction Design overview CRAMES design Pattern-based partial match compression Experimental evaluation Weak link: Compression algorithm LZO average performance penalty 9.5% when RAM reduced to 40% Developed simulation environment to permit profiling Compression and decompression were taking most time Needed a better compression algorithm 17 Robert Dick CRAMES Introduction Design overview CRAMES design Pattern-based partial match compression Experimental evaluation Weak link: Compression algorithm LZO average performance penalty 9.5% when RAM reduced to 40% Developed simulation environment to permit profiling Compression and decompression were taking most time Needed a better compression algorithm 17 Robert Dick CRAMES Introduction Design overview CRAMES design Pattern-based partial match compression Experimental evaluation Pattern-based partial dictionary match coding Input 32-bit word N Y Output dictionary Is something similar Output value location and in dictionary? difference Eliminate LRU dictionary entry Update dictionary entry Place entry in dictionary Move entry to front of dictionary
Load more

Recommended publications
  • Data Compression for Remote Laboratories
    Paper—Data Compression for Remote Laboratories Data Compression for Remote Laboratories https://doi.org/10.3991/ijim.v11i4.6743 Olawale B. Akinwale Obafemi Awolowo University, Ile-Ife, Nigeria [email protected] Lawrence O. Kehinde Obafemi Awolowo University, Ile-Ife, Nigeria [email protected] Abstract—Remote laboratories on mobile phones have been around for a few years now. This has greatly improved accessibility of these remote labs to students who cannot afford computers but have mobile phones. When money is a factor however (as is often the case with those who can’t afford a computer), the cost of use of these remote laboratories should be minimized. This work ad- dressed this issue of minimizing the cost of use of the remote lab by making use of data compression for data sent between the user and remote lab. Keywords—Data compression; Lempel-Ziv-Welch; remote laboratory; Web- Sockets 1 Introduction The mobile phone is now the de facto computational device of choice. They are quite powerful, connected devices which are almost always with us. This is so much so that about every important online service has a mobile version or at least a version compatible with mobile phones and tablets (for this paper we would adopt the phrase mobile device to represent mobile phones and tablets). This has caused online labora- tory developers to develop online laboratory solutions which are mobile device- friendly [1, 2, 3, 4, 5, 6, 7, 8]. One of the main benefits of online laboratories is the possibility of using fewer re- sources to serve more people i.e.
    [Show full text]
  • The Pillars of Lossless Compression Algorithms a Road Map and Genealogy Tree
    International Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Number 6 (2018) pp. 3296-3414 © Research India Publications. http://www.ripublication.com The Pillars of Lossless Compression Algorithms a Road Map and Genealogy Tree Evon Abu-Taieh, PhD Information System Technology Faculty, The University of Jordan, Aqaba, Jordan. Abstract tree is presented in the last section of the paper after presenting the 12 main compression algorithms each with a practical This paper presents the pillars of lossless compression example. algorithms, methods and techniques. The paper counted more than 40 compression algorithms. Although each algorithm is The paper first introduces Shannon–Fano code showing its an independent in its own right, still; these algorithms relation to Shannon (1948), Huffman coding (1952), FANO interrelate genealogically and chronologically. The paper then (1949), Run Length Encoding (1967), Peter's Version (1963), presents the genealogy tree suggested by researcher. The tree Enumerative Coding (1973), LIFO (1976), FiFO Pasco (1976), shows the interrelationships between the 40 algorithms. Also, Stream (1979), P-Based FIFO (1981). Two examples are to be the tree showed the chronological order the algorithms came to presented one for Shannon-Fano Code and the other is for life. The time relation shows the cooperation among the Arithmetic Coding. Next, Huffman code is to be presented scientific society and how the amended each other's work. The with simulation example and algorithm. The third is Lempel- paper presents the 12 pillars researched in this paper, and a Ziv-Welch (LZW) Algorithm which hatched more than 24 comparison table is to be developed.
    [Show full text]
  • The Deep Learning Solutions on Lossless Compression Methods for Alleviating Data Load on Iot Nodes in Smart Cities
    sensors Article The Deep Learning Solutions on Lossless Compression Methods for Alleviating Data Load on IoT Nodes in Smart Cities Ammar Nasif *, Zulaiha Ali Othman and Nor Samsiah Sani Center for Artificial Intelligence Technology (CAIT), Faculty of Information Science & Technology, University Kebangsaan Malaysia, Bangi 43600, Malaysia; [email protected] (Z.A.O.); [email protected] (N.S.S.) * Correspondence: [email protected] Abstract: Networking is crucial for smart city projects nowadays, as it offers an environment where people and things are connected. This paper presents a chronology of factors on the development of smart cities, including IoT technologies as network infrastructure. Increasing IoT nodes leads to increasing data flow, which is a potential source of failure for IoT networks. The biggest challenge of IoT networks is that the IoT may have insufficient memory to handle all transaction data within the IoT network. We aim in this paper to propose a potential compression method for reducing IoT network data traffic. Therefore, we investigate various lossless compression algorithms, such as entropy or dictionary-based algorithms, and general compression methods to determine which algorithm or method adheres to the IoT specifications. Furthermore, this study conducts compression experiments using entropy (Huffman, Adaptive Huffman) and Dictionary (LZ77, LZ78) as well as five different types of datasets of the IoT data traffic. Though the above algorithms can alleviate the IoT data traffic, adaptive Huffman gave the best compression algorithm. Therefore, in this paper, Citation: Nasif, A.; Othman, Z.A.; we aim to propose a conceptual compression method for IoT data traffic by improving an adaptive Sani, N.S.
    [Show full text]
  • Data Compression for Remote Laboratories
    CORE Metadata, citation and similar papers at core.ac.uk Provided by Online-Journals.org (International Association of Online Engineering) Paper—Data Compression for Remote Laboratories Data Compression for Remote Laboratories https://doi.org/10.3991/ijim.v11i4.6743 Olawale B. Akinwale Obafemi Awolowo University, Ile-Ife, Nigeria [email protected] Lawrence O. Kehinde Obafemi Awolowo University, Ile-Ife, Nigeria [email protected] Abstract—Remote laboratories on mobile phones have been around for a few years now. This has greatly improved accessibility of these remote labs to students who cannot afford computers but have mobile phones. When money is a factor however (as is often the case with those who can’t afford a computer), the cost of use of these remote laboratories should be minimized. This work ad- dressed this issue of minimizing the cost of use of the remote lab by making use of data compression for data sent between the user and remote lab. Keywords—Data compression; Lempel-Ziv-Welch; remote laboratory; Web- Sockets 1 Introduction The mobile phone is now the de facto computational device of choice. They are quite powerful, connected devices which are almost always with us. This is so much so that about every important online service has a mobile version or at least a version compatible with mobile phones and tablets (for this paper we would adopt the phrase mobile device to represent mobile phones and tablets). This has caused online labora- tory developers to develop online laboratory solutions which are mobile device- friendly [1, 2, 3, 4, 5, 6, 7, 8].
    [Show full text]
  • Compression: Putting the Squeeze on Storage
    Compression: Putting the Squeeze on Storage Live Webcast September 2, 2020 11:00 am PT 1 | ©2020 Storage Networking Industry Association. All Rights Reserved. Today’s Presenters Ilker Cebeli John Kim Brian Will Moderator Chair, SNIA Networking Storage Forum Intel® QuickAssist Technology Samsung NVIDIA Software Architect Intel 2 | ©2020 Storage Networking Industry Association. All Rights Reserved. SNIA-At-A-Glance 3 3 | ©2020 Storage Networking Industry Association. All Rights Reserved. NSF Technologies 4 4 | ©2020 Storage Networking Industry Association. All Rights Reserved. SNIA Legal Notice § The material contained in this presentation is copyrighted by the SNIA unless otherwise noted. § Member companies and individual members may use this material in presentations and literature under the following conditions: § Any slide or slides used must be reproduced in their entirety without modification § The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations. § This presentation is a project of the SNIA. § Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. § The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK. 5 | ©2020 Storage Networking Industry Association.
    [Show full text]
  • Accelerating Compressed File Pattern Matching with Parabix Techniques
    Accelerating Compressed File Pattern Matching with Parabix Techniques by Xiangyu Wu B.Eng., Tongji University, China, 2015 Thesis Submitted in Partial Fulfillment of the Requirements for the Degree of Master of Science in the School of Computing Science Faculty of Applied Sciences © Xiangyu Wu 2018 SIMON FRASER UNIVERSITY Fall 2018 All rights reserved. However, in accordance with the Copyright Act of Canada, this work may be reproduced without authorization under the conditions for "Fair Dealing". Therefore, limited reproduction of this work for the purposes of private study, research, education, satire, parody, criticism, review and news reporting is likely to be in accordance with the law, particularly if cited appropriately. Approval Name: Xiangyu Wu Degree: Master of Science Title: Accelerating Compressed File Pattern Matching with Parabix Techniques Examining Committee: Chair: Kay Wiese Associate Professor Robert Cameron Senior Supervisor Professor William Sumner Supervisor Assistant Professor Fred Popowich Examiner Professor Date Defended: October 5, 2018 ii Abstract Data compression is a commonly used technique in the big data environment. On the other hand, efficient information retrieval of those compressed big data is also a standard requirement. In this thesis, we proposed a new method of compressed file pattern matching inspired by the bitstream pattern matching approach from ICGrep, a high- performance regular expression matching tool based on Parabix techniques. Instead of using the traditional way that fully decompresses the compressed file before pattern matching, our approach handles many complex procedures in the small compressed space. We selected LZ4 as a sample compression format and implemented a compressed file pattern matching tool LZ4 Grep, which showed small performance improvement.
    [Show full text]
  • 1 Introduction
    HELSINKI UNIVERSITY OF TECHNOLOGY Faculty of Electronics, Communications, and Automation Department of Communications and Networking Le Wang Evaluation of Compression for Energy-aware Communication in Wireless Networks Master’s Thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Technology. Espoo, May 11, 2009 Supervisor: Professor Jukka Manner Instructor: Sebastian Siikavirta 2 HELSINKI UNIVERSITY OF TECHNOLOGY ABSTRACT OF MASTER’S THESIS Author: Le Wang Title: Evaluation of Compression for Energy-aware Communication in Wireless Networks Number of pages: 75 p. Date: 11th May 2009 Faculty: Faculty of Electronics, Communications, and Automation Department: Department of Communications and Networks Code: S-38 Supervisor: Professor Jukka Manner Instructor: Sebastian Siikavirta Abstract In accordance with the development of ICT-based communication, energy efficient communication in wireless networks is being required for reducing energy consumption, cutting down greenhouse emissions and improving business competitiveness. Due to significant energy consumption of transmitting data over wireless networks, data compression techniques can be used to trade the overhead of compression/decompression for less communication energy. Careless and blind compression in wireless networks not only causes an expansion of file sizes, but also wastes energy. This study aims to investigate the usages of data compression to reduce the energy consumption in a hand-held device. By con- ducting experiments as the methodologies, the impacts of transmission on energy consumption are explored on wireless interfaces. Then, 9 lossless compression algo- rithms are examined on popular Internet traffic in the view of compression ratio, speed and consumed energy. Additionally, energy consumption of uplink, downlink and overall system is investigated to achieve a comprehensive understanding of compression in wireless networks.
    [Show full text]
  • Data Compression in Solid State Storage
    Data Compression in Solid State Storage John Fryar [email protected] Santa Clara, CA August 2013 1 Acknowledgements This presentation would not have been possible without the counsel, hard work and graciousness of the following individuals and/or organizations: Raymond Savarda Sandgate Technologies Santa Clara, CA August 2013 2 Disclaimers The opinions expressed herein are those of the author and do not necessarily represent those of any other organization or individual unless specifically cited. A thorough attempt to acknowledge all sources has been made. That said, we’re all human… Santa Clara, CA August 2013 3 Learning Objectives At the conclusion of this tutorial the audience will have been exposed to: • The different types of Data Compression • Common Data Compression Algorithms • The Deflate/Inflate (GZIP/GUNZIP) algorithms in detail • Implementation Options (Software/Hardware) • Impacts of design parameters in Performance • SSD benefits and challenges • Resources for Further Study Santa Clara, CA August 2013 4 Agenda • Background, Definitions, & Context • Data Compression Overview • Data Compression Algorithm Survey • Deflate/Inflate (GZIP/GUNZIP) in depth • Software Implementations • HW Implementations • Tradeoffs & Advanced Topics • SSD Benefits and Challenges • Conclusions Santa Clara, CA August 2013 5 Definitions Item Description Comments Open A system which will compress Must strictly adhere to standards System data for use by other entities. on compress / decompress I.E. the compressed data will algorithms exit the system Interoperability among vendors mandated for Open Systems Closed A system which utilizes Can support a limited, optimized System compressed data internally but subset of standard. does not expose compressed Also allows custom algorithms data to the outside world No Interoperability req’d.
    [Show full text]
  • Better Adaptive Text Compression Scheme
    Journal of Education and Science Vol. 27, No.2, 2018, pp. 48-57 ISSN 1812-125X http://edusj.mosuljournals.com Better Adaptive Text Compression Scheme Duha Amir Sultan Dept. of Biology/ Education College of Girls/ University of Mosul تاريخ اﻻستﻻم 17/04/2012 تاريخ القبول 2012/11/07 الخﻻصة ظهرت طر ق عدي دي لكبس البيدااداتو حاةدي أش ر ددددددددددددددهر طديق اللرق ر ط اللريقد ال قترة أش قبل العال ي ش Ziv ح Lempel حالت رطلق عليها اسددددددددد LZ77و ث طورت طيق اللريق أش قبددل العددال يش Storer و Szymanski لتظهر طريقدد يدديدددي ط LZSS و حالت رعلددن اتددا رفضدل أقارا ساداسقاتها سعي رط طبقن عل اصدو ثيير حخدة ف طاتيش اللريقتيش حي ي ع اللرق اﻷخرى الت تنضوي تحن تصنيف اللرق اليدناأيك ي لكبس البيااات دت أاح ال لف أش اليادددار ل الي يش اد اط رطول تلابق بيش أةزط البيااا ت )الن ص ال شدددمر أادددبقاص حالن ص اليي سدديشددمر اللريق ال قترة ف طيا البحث تعت ي أبير البح ث سإت اطيشو أش الياددار ل الي يش ث أش الي يش ل اليادددددددددددددارو عل الرر رط طيق اللريق تحتان حقتا رطول اوعا أا و ﻻ راها رعل ن اتي ثبس اكبر للبيااات Abstract A data compression scheme suggested by Ziv and Lempel, LZ77, is applied to text compression. A slightly modified version suggested by Storer and Szymanski ,LZSS, is found to achieve compression ratios as good as most existing schemes for a wide range of texts. In these two methods and all other dynamic methods, a text file is searched from left to right to find the longest match between the lookahead buffer (the previously encoded text) and the characters to be encoded.
    [Show full text]
  • Module 9: Compression
    Module 9: Compression CS 240 - Data Structures and Data Management Reza Dorrigiv, Daniel Roche School of Computer Science, University of Waterloo Winter 2010 Reza Dorrigiv, Daniel Roche (CS, UW) CS240 - Module 9 Winter 2010 1 / 33 Data Storage and Transmission The problem: How to store and transmit data? Source text The original data, string of characters S from the source alphabet ΣS Coded text The encoded data, string of characters C from the coded alphabet ΣC Encoding An algorithm mapping source texts to coded texts Decoding An algorithm mapping coded texts back to their original source text Note: Source \text" can be any sort of data (not always text!) Usually the coded alphabet ΣC is just binary: f0; 1g. Reza Dorrigiv, Daniel Roche (CS, UW) CS240 - Module 9 Winter 2010 2 / 33 Processing speed Reliability (e.g. error-correcting codes) Security (e.g. encryption) Size Encoding schemes that try to minimize jCj, the size of the coded text, perform data compression. We will measure the compression ratio: jCj · lg jΣC j jSj · lg jΣS j Judging Encoding Schemes We can always measure efficiency of encoding/decoding algorithms. What other goals might there be? Reza Dorrigiv, Daniel Roche (CS, UW) CS240 - Module 9 Winter 2010 3 / 33 Encoding schemes that try to minimize jCj, the size of the coded text, perform data compression. We will measure the compression ratio: jCj · lg jΣC j jSj · lg jΣS j Judging Encoding Schemes We can always measure efficiency of encoding/decoding algorithms. What other goals might there be? Processing speed Reliability (e.g. error-correcting codes) Security (e.g.
    [Show full text]
  • Data Compression for Climate Data
    Data compression for climate data Article Published Version Creative Commons: Attribution-Noncommercial 3.0 Open access Kuhn, M., Kunkel, J. M. and Ludwig, T. (2016) Data compression for climate data. Supercomputing Frontiers and Innovations, 3 (1). pp. 75-94. ISSN 2313-8734 doi: https://doi.org/10.14529/jsfi160105 Available at http://centaur.reading.ac.uk/77684/ It is advisable to refer to the publisher’s version if you intend to cite from the work. See Guidance on citing . Identification Number/DOI: https://doi.org/10.14529/jsfi160105 <https://doi.org/10.14529/jsfi160105> Publisher: South Urals University All outputs in CentAUR are protected by Intellectual Property Rights law, including copyright law. Copyright and IPR is retained by the creators or other copyright holders. Terms and conditions for use of this material are defined in the End User Agreement . www.reading.ac.uk/centaur CentAUR Central Archive at the University of Reading Reading’s research outputs online DOI: 10.14529/jsfi160105 Data Compression for Climate Data Michael Kuhn1, Julian Kunkel2, Thomas Ludwig2 © The Authors 2016. This paper is published with open access at SuperFri.org The different rates of increase for computational power and storage capabilities of super- computers turn data storage into a technical and economical problem. As storage capabilities are lagging behind, investments and operational costs for storage systems have increased to keep up with the supercomputers’ I/O requirements. One promising approach is to reduce the amount of data that is stored. In this paper, we take a look at the impact of compression on performance and costs of high performance systems.
    [Show full text]
  • An Optimized VLSI Based High Performance Architecture for Lossless Data Compression
    International Journal of Advanced Science and Technology Vol. 29, No. 5, (2020), pp. 9503-9510 An Optimized VLSI Based High Performance Architecture for Lossless Data Compression 1*R.Umamaheswaran, 2**S.R.Santhosh kumaran, 3R.Parameshwaran, 4G. Lakshminarayanan, 5S.Thiruppathirajan, 6R. Manikandan 1Scientific Secretary, ISRO, Bengaluru, India. Mail Id: [email protected], 9496040095 2,3,4Department of ECE, National Institute of Technology-Tiruchirappalli, India Mail Id: [email protected] , 9487679810, [email protected], 8056303057 [email protected], 9486001109 5Avionics Entity, VSSC, Department of Space, Thiruvananthapuram, India Mail Id: [email protected], 9446101839 6School of Computing, SASTRA Deemed University, India [email protected], 8667872608 Abstract Data compression is method of encoding rules, which allows substantial reduction in total number of bits to transmit or store file. To archive or transmit binary files or text, lossless data compression is utilized, which is need to keep data intact at any time. LZRW1 (Lempel Ziv Ross Williams) and LZW (Lempel Ziv Welch) was mainly used for lossless data Compression. But LZW takes more time to Compress and decompress data. LZRW refers to variants of LZ77 lossless data compression techniques with emphasis on increasing compression speed by use of hash tables and other methods. In order to provide high throughput of data, a hash table is implemented by replacing the dictionary in LZW. This method has worst case running time. It adapts quickly and neglects initialisation in making it fast as well as powerful for small blocks of data and large ones. The speed of compression has been increased by four times that of LZW.The compression ratio obtained also be relatively same as that of LZW.The proposed algorithm was implemented in system verilog, simulated by using Questa sim tool and synthesised in Xilinx Vivado.
    [Show full text]