Evaluation of Lossless and Lossy Algorithms for the Compression of Scientific Datasets in Netcdf-4 Or HDF5 files

Total Page:16

File Type:pdf, Size:1020Kb

Evaluation of Lossless and Lossy Algorithms for the Compression of Scientific Datasets in Netcdf-4 Or HDF5 files Geosci. Model Dev., 12, 4099–4113, 2019 https://doi.org/10.5194/gmd-12-4099-2019 © Author(s) 2019. This work is distributed under the Creative Commons Attribution 4.0 License. Evaluation of lossless and lossy algorithms for the compression of scientific datasets in netCDF-4 or HDF5 files Xavier Delaunay1, Aurélie Courtois1, and Flavien Gouillon2 1Thales, 290 allée du Lac, 31670 Labège, France 2CNES, Centre Spatial de Toulouse, 18 avenue Edouard Belin, 31401 Toulouse, France Correspondence: Xavier Delaunay ([email protected]) Received: 5 October 2018 – Discussion started: 20 November 2018 Revised: 7 June 2019 – Accepted: 3 July 2019 – Published: 23 September 2019 Abstract. The increasing volume of scientific datasets re- format is widely used throughout the global scientific com- quires the use of compression to reduce data storage and munity because of its useful features. The fourth version of transmission costs, especially for the oceanographic or me- the netCDF library, denoted netCDF-4/HDF5 (as it is based teorological datasets generated by Earth observation mission on the HDF5 layer), offers “Deflate” and “Shuffle” algo- ground segments. These data are mostly produced in netCDF rithms as native compression features. However, the com- files. Indeed, the netCDF-4/HDF5 file formats are widely pression ratio achieved does not fully meet ground process- used throughout the global scientific community because of ing requirements, which are to significantly reduce the stor- the useful features they offer. HDF5 in particular offers a dy- age and dissemination cost as well as the I/O times between namically loaded filter plugin so that users can write com- two modules in the processing chain. pression/decompression filters, for example, and process the In response to the ever-increasing volume of data, scien- data before reading or writing them to disk. This study eval- tists are keen to compress data. However, they have certain uates lossy and lossless compression/decompression meth- requirements: both compression and decompression have to ods through netCDF-4 and HDF5 tools on analytical and be fast. Lossy compression is acceptable only if the compres- real scientific floating-point datasets. We also introduce the sion ratios are higher than those of lossless algorithms and if Digit Rounding algorithm, a new relative error-bounded data the precision, or data loss, can be controlled. There is a trade- reduction method inspired by the Bit Grooming algorithm. off between the data volume and the accuracy of the com- The Digit Rounding algorithm offers a high compression ra- pressed data. Nevertheless, scientists can accept small losses tio while keeping a given number of significant digits in the if they remain below the data’s noise level. Noise is difficult dataset. It achieves a higher compression ratio than the Bit to compress and of little interest to scientists, so they do not Grooming algorithm with slightly lower compression speed. consider data degradation that remains under the noise level as a loss (Baker et al., 2016). In order to increase the com- pression ratio within the processing chain, “clipping” meth- ods may be used to degrade the data before compression. 1 Introduction These methods increase the compression ratio by removing the least significant digits in the data. Indeed, at some level, Ground segments processing scientific mission data are fac- these least significant digits may not be scientifically mean- ing challenges due to the ever-increasing resolution of on- ingful in datasets corrupted by noise. board instruments and the volume of data to be processed, This paper studies compression and clipping methods that stored and transmitted. This is the case for oceanographic can be applied to scientific datasets in order to maximize and meteorological missions, for instance. Earth observation the compression ratio while preserving scientific data content mission ground segments produce very large files mostly in and numerical accuracy. It focuses on methods that can be ap- netCDF format, which is standard in the oceanography field plied to scientific datasets, i.e. vectors or matrices of floating- and widely used by the meteorological community. This file Published by Copernicus Publications on behalf of the European Geosciences Union. 4100 X. Delaunay et al.: Digit rounding compression algorithm point numbers. First, lossless compression algorithms can be applied to any kind of data. The standard is the “Deflate” al- gorithm (Deutsch, 1996) native in netCDF-4/HDF5 libraries. It is widely used in compression tools such as zip, gzip, and zlib libraries, and has become a benchmark for lossless data compression. Recently, alternative lossless compression al- gorithms have emerged. These include Google Snappy, LZ4 (Collet, 2013) or Zstandard (Collet and Turner, 2016). To achieve faster compression than the Deflate algorithm, none of these algorithms use Huffman coding. Second, prepro- cessing methods such as Shuffle, available in HDF5, or Bit- shuffle (Masui et al., 2015) are used to optimize lossless com- pression by rearranging the data bytes or bits into a “more compressible” order. Third, some lossy/lossless compression algorithms, such as FPZIP (Lindstrom and Isenburg, 2006), ZFP (Lindstrom, 2014) or Sz (Tao et al., 2017), are specifi- cally designed for scientific data – and in particular floating- point data – and can control data loss. Fourth, data reduction methods such as Linear Packing (Caron, 2014a), Layer Pack- Figure 1. Compression chain showing the data reduction, prepro- ing (Silver and Zender, 2017), Bit Shaving (Caron, 2014b), cessing and lossless coding steps. and Bit Grooming (Zender, 2016a) lose some data con- tent without necessarily reducing its volume. Preprocessing methods and lossless compression can then be applied to ob- 2 Compression algorithms tain a higher compression ratio. This paper focuses on compression methods implemented Compression schemes for scientific floating-point datasets for netCDF-4 or HDF5 files. These scientific file formats usually entail several steps: data reduction, preprocessing, are widespread among the oceanographic and meteorolog- and lossless coding. These three steps can be chained as il- ical communities. HDF5 offers a dynamically loaded filter lustrated in Fig. 1. The lossless coding step is reversible. It plugin that allows users to write compression/decompression does not degrade the data while reducing its volume. It can filters (among others), and to process data before read- be implemented by lossless compression algorithms such as ing or writing them to disk. Consequently, many compres- Deflate, Snappy, LZ4 or Zstandard. The preprocessing step sion/decompression filters – such as Bitshuffle, Zstandard, is also reversible. It rearranges the data bytes or bits to en- LZ4, and Sz – have been implemented by members of the hance lossless coding efficiency. It can be implemented by HDF5 user community and are freely available. The netCDF algorithms such as Shuffle or Bitshuffle. The data reduction Operator toolkit (NCO) (Zender, 2016b) also offers some step is not reversible because it entails data losses. The goal compression features, such as Bit Shaving, Decimal Round- is to remove irrelevant data such as noise or other scientifi- ing and Bit Grooming. cally meaningless data. Data reduction can reduce data vol- The rest of this paper is divided into five sections. Sec- ume, depending on the algorithm used. For instance, the Lin- tion 2 presents the lossless and lossy compression schemes ear Packing and Sz algorithms reduce data volume, but Bit for scientific floating-point datasets. Section 3 introduces the Shaving and Bit Grooming algorithms do not. Digit Rounding algorithm, which is an improvement of the This paper evaluates lossless compression algorithms De- Bit Grooming algorithm that optimizes the number of man- flate, LZ4, and Zstandard; Deflate because it is the bench- tissa bits preserved. Section 4 defines the performance met- mark algorithm, LZ4 because it is a widely-used, very-high- rics used in this paper. Section 5 describes the performance speed compressor, and Zstandard because it provides bet- assessment of a selection of lossless and lossy compression ter results than Deflate both in terms of compression ra- methods on synthetic datasets. It presents the datasets and tios and of compression/decompression speeds. The Deflate compression results before making some recommendations. algorithm uses LZ77 dictionary coding (Ziv and Lempel, Section 6 provides some compression results obtained with 1977) and Huffman entropy coding (Huffman, 1952). LZ77 real CFOSAT and SWOT datasets. Finally, Sect. 7 provides and Huffman coding exploit different types of redundancies our conclusions. to enable Deflate to achieve high compression ratios. How- ever, the computational cost of the Huffman coder is high and makes Deflate compression rather slow. LZ4 is a dic- tionary coding algorithm designed to provide high compres- sion/decompression speeds rather than a high compression ratio. It does this without an entropy coder. Zstandard is a Geosci. Model Dev., 12, 4099–4113, 2019 www.geosci-model-dev.net/12/4099/2019/ X. Delaunay et al.: Digit rounding compression algorithm 4101 fast lossless compressor offering high compression ratios. It zero in the binary representation of the quantized floating- makes use of dictionary coding (repcode modeling) and a point value. It is also similar to Sz’s error-controlled quan- finite-state entropy coder (tANS) (Duda, 2013). It offers a tization (Tao et
Recommended publications
  • Data Compression: Dictionary-Based Coding 2 / 37 Dictionary-Based Coding Dictionary-Based Coding
    Dictionary-based Coding already coded not yet coded search buffer look-ahead buffer cursor (N symbols) (L symbols) We know the past but cannot control it. We control the future but... Last Lecture Last Lecture: Predictive Lossless Coding Predictive Lossless Coding Simple and effective way to exploit dependencies between neighboring symbols / samples Optimal predictor: Conditional mean (requires storage of large tables) Affine and Linear Prediction Simple structure, low-complex implementation possible Optimal prediction parameters are given by solution of Yule-Walker equations Works very well for real signals (e.g., audio, images, ...) Efficient Lossless Coding for Real-World Signals Affine/linear prediction (often: block-adaptive choice of prediction parameters) Entropy coding of prediction errors (e.g., arithmetic coding) Using marginal pmf often already yields good results Can be improved by using conditional pmfs (with simple conditions) Heiko Schwarz (Freie Universität Berlin) — Data Compression: Dictionary-based Coding 2 / 37 Dictionary-based Coding Dictionary-Based Coding Coding of Text Files Very high amount of dependencies Affine prediction does not work (requires linear dependencies) Higher-order conditional coding should work well, but is way to complex (memory) Alternative: Do not code single characters, but words or phrases Example: English Texts Oxford English Dictionary lists less than 230 000 words (including obsolete words) On average, a word contains about 6 characters Average codeword length per character would be limited by 1
    [Show full text]
  • Package 'Brotli'
    Package ‘brotli’ May 13, 2018 Type Package Title A Compression Format Optimized for the Web Version 1.2 Description A lossless compressed data format that uses a combination of the LZ77 algorithm and Huffman coding. Brotli is similar in speed to deflate (gzip) but offers more dense compression. License MIT + file LICENSE URL https://tools.ietf.org/html/rfc7932 (spec) https://github.com/google/brotli#readme (upstream) http://github.com/jeroen/brotli#read (devel) BugReports http://github.com/jeroen/brotli/issues VignetteBuilder knitr, R.rsp Suggests spelling, knitr, R.rsp, microbenchmark, rmarkdown, ggplot2 RoxygenNote 6.0.1 Language en-US NeedsCompilation yes Author Jeroen Ooms [aut, cre] (<https://orcid.org/0000-0002-4035-0289>), Google, Inc [aut, cph] (Brotli C++ library) Maintainer Jeroen Ooms <[email protected]> Repository CRAN Date/Publication 2018-05-13 20:31:43 UTC R topics documented: brotli . .2 Index 4 1 2 brotli brotli Brotli Compression Description Brotli is a compression algorithm optimized for the web, in particular small text documents. Usage brotli_compress(buf, quality = 11, window = 22) brotli_decompress(buf) Arguments buf raw vector with data to compress/decompress quality value between 0 and 11 window log of window size Details Brotli decompression is at least as fast as for gzip while significantly improving the compression ratio. The price we pay is that compression is much slower than gzip. Brotli is therefore most effective for serving static content such as fonts and html pages. For binary (non-text) data, the compression ratio of Brotli usually does not beat bz2 or xz (lzma), however decompression for these algorithms is too slow for browsers in e.g.
    [Show full text]
  • A Survey Paper on Different Speech Compression Techniques
    Vol-2 Issue-5 2016 IJARIIE-ISSN (O)-2395-4396 A Survey Paper on Different Speech Compression Techniques Kanawade Pramila.R1, Prof. Gundal Shital.S2 1 M.E. Electronics, Department of Electronics Engineering, Amrutvahini College of Engineering, Sangamner, Maharashtra, India. 2 HOD in Electronics Department, Department of Electronics Engineering , Amrutvahini College of Engineering, Sangamner, Maharashtra, India. ABSTRACT This paper describes the different types of speech compression techniques. Speech compression can be divided into two main types such as lossless and lossy compression. This survey paper has been written with the help of different types of Waveform-based speech compression, Parametric-based speech compression, Hybrid based speech compression etc. Compression is nothing but reducing size of data with considering memory size. Speech compression means voiced signal compress for different application such as high quality database of speech signals, multimedia applications, music database and internet applications. Today speech compression is very useful in our life. The main purpose or aim of speech compression is to compress any type of audio that is transfer over the communication channel, because of the limited channel bandwidth and data storage capacity and low bit rate. The use of lossless and lossy techniques for speech compression means that reduced the numbers of bits in the original information. By the use of lossless data compression there is no loss in the original information but while using lossy data compression technique some numbers of bits are loss. Keyword: - Bit rate, Compression, Waveform-based speech compression, Parametric-based speech compression, Hybrid based speech compression. 1. INTRODUCTION -1 Speech compression is use in the encoding system.
    [Show full text]
  • ROOT I/O Compression Improvements for HEP Analysis
    EPJ Web of Conferences 245, 02017 (2020) https://doi.org/10.1051/epjconf/202024502017 CHEP 2019 ROOT I/O compression improvements for HEP analysis Oksana Shadura1;∗ Brian Paul Bockelman2;∗∗ Philippe Canal3;∗∗∗ Danilo Piparo4;∗∗∗∗ and Zhe Zhang1;y 1University of Nebraska-Lincoln, 1400 R St, Lincoln, NE 68588, United States 2Morgridge Institute for Research, 330 N Orchard St, Madison, WI 53715, United States 3Fermilab, Kirk Road and Pine St, Batavia, IL 60510, United States 4CERN, Meyrin 1211, Geneve, Switzerland Abstract. We overview recent changes in the ROOT I/O system, enhancing it by improving its performance and interaction with other data analysis ecosys- tems. Both the newly introduced compression algorithms, the much faster bulk I/O data path, and a few additional techniques have the potential to significantly improve experiment’s software performance. The need for efficient lossless data compression has grown significantly as the amount of HEP data collected, transmitted, and stored has dramatically in- creased over the last couple of years. While compression reduces storage space and, potentially, I/O bandwidth usage, it should not be applied blindly, because there are significant trade-offs between the increased CPU cost for reading and writing files and the reduces storage space. 1 Introduction In the past years, Large Hadron Collider (LHC) experiments are managing about an exabyte of storage for analysis purposes, approximately half of which is stored on tape storages for archival purposes, and half is used for traditional disk storage. Meanwhile for High Lumi- nosity Large Hadron Collider (HL-LHC) storage requirements per year are expected to be increased by a factor of 10 [1].
    [Show full text]
  • Download Media Player Codec Pack Version 4.1 Media Player Codec Pack
    download media player codec pack version 4.1 Media Player Codec Pack. Description: In Microsoft Windows 10 it is not possible to set all file associations using an installer. Microsoft chose to block changes of file associations with the introduction of their Zune players. Third party codecs are also blocked in some instances, preventing some files from playing in the Zune players. A simple workaround for this problem is to switch playback of video and music files to Windows Media Player manually. In start menu click on the "Settings". In the "Windows Settings" window click on "System". On the "System" pane click on "Default apps". On the "Choose default applications" pane click on "Films & TV" under "Video Player". On the "Choose an application" pop up menu click on "Windows Media Player" to set Windows Media Player as the default player for video files. Footnote: The same method can be used to apply file associations for music, by simply clicking on "Groove Music" under "Media Player" instead of changing Video Player in step 4. Media Player Codec Pack Plus. Codec's Explained: A codec is a piece of software on either a device or computer capable of encoding and/or decoding video and/or audio data from files, streams and broadcasts. The word Codec is a portmanteau of ' co mpressor- dec ompressor' Compression types that you will be able to play include: x264 | x265 | h.265 | HEVC | 10bit x265 | 10bit x264 | AVCHD | AVC DivX | XviD | MP4 | MPEG4 | MPEG2 and many more. File types you will be able to play include: .bdmv | .evo | .hevc | .mkv | .avi | .flv | .webm | .mp4 | .m4v | .m4a | .ts | .ogm .ac3 | .dts | .alac | .flac | .ape | .aac | .ogg | .ofr | .mpc | .3gp and many more.
    [Show full text]
  • Divx Codec Package
    Divx codec package Videos. How To Use DivX Mux GUI · How to Stream DivX Plus HD (MKV) files to your Xbox · more. Guides. There are no guides available. Search FAQs. Téléchargement gratuit. Inclut DivX Codec et tout ce dont vous avez besoin pour lire les fichiers DivX, AVI ou MKV dans n'importe quel lecteur multimédia. Kostenloser Download. Umfasst DivX Codec und alles, was Du zur Wiedergabe von DivX-, AVI- oder MKV-Dateien in einem beliebigen Media-Player brauchst. H codecs compress digital video files so that they only use half the space of MPEG-2, to deliver the same quality video. An H encoder delivers. Download grátis. Inclui DivX Codec e tudo o mais de que você precisa para reproduzir arquivos DivX, AVI ou MKV em qualquer player de mídia. Free video software downloads to play & stream DivX (AVI) & DivX Plus HD (MKV) video. Find devices to play DivX video and Hollywood movies in DivX format. You can do it all in one go and be ready for any video format that comes your way. Codec Pack All-in-1 includes: DivX ; XviD Codec Media Player Codec Pack for Microsoft Windows, 10, , 8, 7, Vista, XP, , x | h | HEVC | 10bit x | x | h | AVCHD | AVC | DivX | XviD. They feature improved HEVC and AVC decoders for better stability and the DivX codec pack has been removed for consistency around the. Codec Pack All in 1, free and safe download. Codec Pack All in 1 latest version: A free Video program for Windows. Codec Pack All in 1 is a good, free Windows.
    [Show full text]
  • Arxiv:2004.10531V1 [Cs.OH] 8 Apr 2020
    ROOT I/O compression improvements for HEP analysis Oksana Shadura1;∗ Brian Paul Bockelman2;∗∗ Philippe Canal3;∗∗∗ Danilo Piparo4;∗∗∗∗ and Zhe Zhang1;y 1University of Nebraska-Lincoln, 1400 R St, Lincoln, NE 68588, United States 2Morgridge Institute for Research, 330 N Orchard St, Madison, WI 53715, United States 3Fermilab, Kirk Road and Pine St, Batavia, IL 60510, United States 4CERN, Meyrin 1211, Geneve, Switzerland Abstract. We overview recent changes in the ROOT I/O system, increasing per- formance and enhancing it and improving its interaction with other data analy- sis ecosystems. Both the newly introduced compression algorithms, the much faster bulk I/O data path, and a few additional techniques have the potential to significantly to improve experiment’s software performance. The need for efficient lossless data compression has grown significantly as the amount of HEP data collected, transmitted, and stored has dramatically in- creased during the LHC era. While compression reduces storage space and, potentially, I/O bandwidth usage, it should not be applied blindly: there are sig- nificant trade-offs between the increased CPU cost for reading and writing files and the reduce storage space. 1 Introduction In the past years LHC experiments are commissioned and now manages about an exabyte of storage for analysis purposes, approximately half of which is used for archival purposes, and half is used for traditional disk storage. Meanwhile for HL-LHC storage requirements per year are expected to be increased by factor 10 [1]. arXiv:2004.10531v1 [cs.OH] 8 Apr 2020 Looking at these predictions, we would like to state that storage will remain one of the major cost drivers and at the same time the bottlenecks for HEP computing.
    [Show full text]
  • Installation Manual
    CX-20 Installation manual ENABLING BRIGHT OUTCOMES Barco NV Beneluxpark 21, 8500 Kortrijk, Belgium www.barco.com/en/support www.barco.com Registered office: Barco NV President Kennedypark 35, 8500 Kortrijk, Belgium www.barco.com/en/support www.barco.com Copyright © All rights reserved. No part of this document may be copied, reproduced or translated. It shall not otherwise be recorded, transmitted or stored in a retrieval system without the prior written consent of Barco. Trademarks Brand and product names mentioned in this manual may be trademarks, registered trademarks or copyrights of their respective holders. All brand and product names mentioned in this manual serve as comments or examples and are not to be understood as advertising for the products or their manufacturers. Trademarks USB Type-CTM and USB-CTM are trademarks of USB Implementers Forum. HDMI Trademark Notice The terms HDMI, HDMI High Definition Multimedia Interface, and the HDMI Logo are trademarks or registered trademarks of HDMI Licensing Administrator, Inc. Product Security Incident Response As a global technology leader, Barco is committed to deliver secure solutions and services to our customers, while protecting Barco’s intellectual property. When product security concerns are received, the product security incident response process will be triggered immediately. To address specific security concerns or to report security issues with Barco products, please inform us via contact details mentioned on https://www.barco.com/psirt. To protect our customers, Barco does not publically disclose or confirm security vulnerabilities until Barco has conducted an analysis of the product and issued fixes and/or mitigations. Patent protection Please refer to www.barco.com/about-barco/legal/patents Guarantee and Compensation Barco provides a guarantee relating to perfect manufacturing as part of the legally stipulated terms of guarantee.
    [Show full text]
  • The Basic Principles of Data Compression
    The Basic Principles of Data Compression Author: Conrad Chung, 2BrightSparks Introduction Internet users who download or upload files from/to the web, or use email to send or receive attachments will most likely have encountered files in compressed format. In this topic we will cover how compression works, the advantages and disadvantages of compression, as well as types of compression. What is Compression? Compression is the process of encoding data more efficiently to achieve a reduction in file size. One type of compression available is referred to as lossless compression. This means the compressed file will be restored exactly to its original state with no loss of data during the decompression process. This is essential to data compression as the file would be corrupted and unusable should data be lost. Another compression category which will not be covered in this article is “lossy” compression often used in multimedia files for music and images and where data is discarded. Lossless compression algorithms use statistic modeling techniques to reduce repetitive information in a file. Some of the methods may include removal of spacing characters, representing a string of repeated characters with a single character or replacing recurring characters with smaller bit sequences. Advantages/Disadvantages of Compression Compression of files offer many advantages. When compressed, the quantity of bits used to store the information is reduced. Files that are smaller in size will result in shorter transmission times when they are transferred on the Internet. Compressed files also take up less storage space. File compression can zip up several small files into a single file for more convenient email transmission.
    [Show full text]
  • Implementing Compression on Distributed Time Series Database
    Implementing compression on distributed time series database Michael Burman School of Science Thesis submitted for examination for the degree of Master of Science in Technology. Espoo 05.11.2017 Supervisor Prof. Kari Smolander Advisor Mgr. Jiri Kremser Aalto University, P.O. BOX 11000, 00076 AALTO www.aalto.fi Abstract of the master’s thesis Author Michael Burman Title Implementing compression on distributed time series database Degree programme Major Computer Science Code of major SCI3042 Supervisor Prof. Kari Smolander Advisor Mgr. Jiri Kremser Date 05.11.2017 Number of pages 70+4 Language English Abstract Rise of microservices and distributed applications in containerized deployments are putting increasing amount of burden to the monitoring systems. They push the storage requirements to provide suitable performance for large queries. In this paper we present the changes we made to our distributed time series database, Hawkular-Metrics, and how it stores data more effectively in the Cassandra. We show that using our methods provides significant space savings ranging from 50 to 95% reduction in storage usage, while reducing the query times by over 90% compared to the nominal approach when using Cassandra. We also provide our unique algorithm modified from Gorilla compression algorithm that we use in our solution, which provides almost three times the throughput in compression with equal compression ratio. Keywords timeseries compression performance storage Aalto-yliopisto, PL 11000, 00076 AALTO www.aalto.fi Diplomityön tiivistelmä Tekijä Michael Burman Työn nimi Pakkausmenetelmät hajautetussa aikasarjatietokannassa Koulutusohjelma Pääaine Computer Science Pääaineen koodi SCI3042 Työn valvoja ja ohjaaja Prof. Kari Smolander Päivämäärä 05.11.2017 Sivumäärä 70+4 Kieli Englanti Tiivistelmä Hajautettujen järjestelmien yleistyminen on aiheuttanut valvontajärjestelmissä tiedon määrän kasvua, sillä aikasarjojen määrä on kasvanut ja niihin talletetaan useammin tietoa.
    [Show full text]
  • Lossless Compression of Audio Data
    CHAPTER 12 Lossless Compression of Audio Data ROBERT C. MAHER OVERVIEW Lossless data compression of digital audio signals is useful when it is necessary to minimize the storage space or transmission bandwidth of audio data while still maintaining archival quality. Available techniques for lossless audio compression, or lossless audio packing, generally employ an adaptive waveform predictor with a variable-rate entropy coding of the residual, such as Huffman or Golomb-Rice coding. The amount of data compression can vary considerably from one audio waveform to another, but ratios of less than 3 are typical. Several freeware, shareware, and proprietary commercial lossless audio packing programs are available. 12.1 INTRODUCTION The Internet is increasingly being used as a means to deliver audio content to end-users for en­ tertainment, education, and commerce. It is clearly advantageous to minimize the time required to download an audio data file and the storage capacity required to hold it. Moreover, the expec­ tations of end-users with regard to signal quality, number of audio channels, meta-data such as song lyrics, and similar additional features provide incentives to compress the audio data. 12.1.1 Background In the past decade there have been significant breakthroughs in audio data compression using lossy perceptual coding [1]. These techniques lower the bit rate required to represent the signal by establishing perceptual error criteria, meaning that a model of human hearing perception is Copyright 2003. Elsevier Science (USA). 255 AU rights reserved. 256 PART III / APPLICATIONS used to guide the elimination of excess bits that can be either reconstructed (redundancy in the signal) orignored (inaudible components in the signal).
    [Show full text]
  • The H.264 Advanced Video Coding (AVC) Standard
    Whitepaper: The H.264 Advanced Video Coding (AVC) Standard What It Means to Web Camera Performance Introduction A new generation of webcams is hitting the market that makes video conferencing a more lifelike experience for users, thanks to adoption of the breakthrough H.264 standard. This white paper explains some of the key benefits of H.264 encoding and why cameras with this technology should be on the shopping list of every business. The Need for Compression Today, Internet connection rates average in the range of a few megabits per second. While VGA video requires 147 megabits per second (Mbps) of data, full high definition (HD) 1080p video requires almost one gigabit per second of data, as illustrated in Table 1. Table 1. Display Resolution Format Comparison Format Horizontal Pixels Vertical Lines Pixels Megabits per second (Mbps) QVGA 320 240 76,800 37 VGA 640 480 307,200 147 720p 1280 720 921,600 442 1080p 1920 1080 2,073,600 995 Video Compression Techniques Digital video streams, especially at high definition (HD) resolution, represent huge amounts of data. In order to achieve real-time HD resolution over typical Internet connection bandwidths, video compression is required. The amount of compression required to transmit 1080p video over a three megabits per second link is 332:1! Video compression techniques use mathematical algorithms to reduce the amount of data needed to transmit or store video. Lossless Compression Lossless compression changes how data is stored without resulting in any loss of information. Zip files are losslessly compressed so that when they are unzipped, the original files are recovered.
    [Show full text]