DOCSLIB.ORG

Lossless Source Coding

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Lossless Source Coding Lossless Source Coding Gadiel Seroussi Marcelo Weinberger Information Theory Research Hewlett-Packard Laboratories Palo Alto, California Seroussi/Weinberger – Lossless source coding 26-May-04 1 Notation • A = discrete (usually finite) alphabet • a = | A| = size of A (when finite) n n • x1 = x = x1x2 x3 Kxn = finite sequence over A ¥ ¥ • x1 = x = x1x2 x3 Kxt K = infinite sequence over A j • xi = xi xi+1Kx j = sub-sequence (i sometimes omitted if = 1) • pX(x) = Prob(X=x) = probability mass function (PMF) on A (subscript X and argument x dropped if clear from context) • X ~ p(x) : X obeys PMF p(x) • Ep [F] = expectation of F w.r.t. PMF p (subscript and [ ] may be dropped) n • pˆ n (x) = empirical distribution obtained from x1 x1 • log x = logarithm to base 2 of x, unless base otherwise specified • ln x = natural logarithm of x • H(X), H(p) = entropy of a random variable X or PMF p, in bits; also • H(p) = - p log p - (1-p) log (1-p), 0 £ p £ 1 : binary entropy function • D(p||q) = relative entropy (information divergence) between PMFs p and q Seroussi/Weinberger – Lossless source coding 26-May-04 2 LosslessLossless SourceSource CodingCoding 3. Universal Coding Seroussi/Weinberger – Lossless source coding 26-May-04 3 Universal Modeling and Coding q So far, it was assumed that a model of the data is available, and we aimed at compressing the data optimally w.r.t. the model q By Kraft’s inequality, the models we use can be expressed in terms of probability distributions: - L( s ) For every UD code with length function L(s), we have å2 £ 1 n string of length n over sÎA finite alphabet A Þ a code defines a probability distribution P(s) = 2-L(s) over An q Conversely, given a distribution P( ) (a model), there exists a UD code that assigns é- log p(s)ù bits to s (Shannon code) q Hence, P( ) serves as a model to encode s, and every code has an associated model l a (probabilistic) model is a tool to “understand” and predict the behavior of the data Seroussi/Weinberger – Lossless source coding 26-May-04 4 Universal Modeling and Coding (cont.) q Given a model P( ) on n-tuples, arithmetic coding provides an effective mean to sequentially assign a code word of length close to -log P(s) to s l we don’t need to see the whole string s = x1 x2 … xn : encode xt using conditional probability t t å P(x u) P(x ) n-t p(x | x x ...x ) = = uÎA t 1 2 t-1 P(xt -1 ) å P(xt -1v) vÎAn-t +1 l the model probabilities can vary arbitrarily and “adapt” to the data l in a strongly sequential model, P(xt) is independent of n q CODING SYSTEM = MODEL + CODING UNIT l two separate problems: design a model and use it to encode We will view data compression as a problem of assigning probabilities to data Seroussi/Weinberger – Lossless source coding 26-May-04 5 Coding with Model Classes q Universal data compression deals with the optimal description of data in the absence of a given model l in most practical applications, the model is not given to us q How do we make the concept of “optimality” meaningful? l there is always a code that assigns just 1 bit to the data at hand! The answer: Model classes q We want a “universal” code to perform as well as the best model in a given class C for any string s, where the best competing model changes from string to string l universality makes sense only w.r.t. a model class q A code with length function L(xn) is pointwise universal w.r.t. a n 1 n n class C if RC (L, x ) = [L(x ) - min LC (x )] ® 0 when n ® ¥ n CÎC pointwise redundancy Seroussi/Weinberger – Lossless source coding 26-May-04 6 How to Choose a Model Class? Universal coding tells us how to encode optimally w.r.t. to a class; it doesn’t tell us how to choose a class! q Some possible criteria: l complexity l prior knowledge on the data l some popular models were already presented q We will see that the bigger the class, the slower the best possible convergence rate of the redundancy to 0 l in this sense, prior knowledge is of paramount importance: don’t learn what you already know! Ultimately, the choice of model class is an art Seroussi/Weinberger – Lossless source coding 26-May-04 7 Example: Bernoulli Models A = {0,1}, C = {Pq , q Î L = [0,1] } i.i.d. distribution with parameter p(1) = q n 1 1 ) n min LC (x ) = min log n = log n = nH (x ) CÎC qÎL ) Pq (x ) Pq ( x n ) (x ) ML-estimate of q Therefore, our goal is to find a code such that L(x n ) ) - H (x n ) ® 0 n q Here is a trivial example of a universal code: log( n + 1) Use é ù bits to encode n1, and then “tune” your Shannon code to the parameter q = n1 / n, which is precisely the ML-estimate Þ this is equivalent to telling the decoder what the best model is! Seroussi/Weinberger – Lossless source coding 26-May-04 8 Bernoulli Models: Enumerative Code q The total code length for this code satisfies: n æ ö æ ö æ ö æ ö L(x ) n0 n0 n1 n1 log(n +1) + 2 ˆ n log(n +1) + 2 £ -ç ÷logç ÷ - ç ÷logç ÷ + = H (x ) + n è n ø è n ø è n ø è n ø n n ® 0 l wasteful: knowledge of n1 already discards part of the sequences, to which we should not reserve a code word æ n ö ç ÷ q A slightly better code: enumerate all ç ÷ sequences with n1ones, è n1 ø and describe the sequence with an index n é n ù L¢(x ) 1 æ ö élog(n +1)ù ˆ n log(n +1) + 2 = êlogç ÷ú + £ H (x ) + n n ê èn1 øú n n n æ n ö 2nHˆ ( x ) ç ÷ £ Stirling: ç ÷ Þ for sequences such that n /n is èn1 ø 2p (n1n0 ) / n 1 L¢(xn ) log n bounded away from 0 and 1, £ Hˆ (xn ) + + O(1/ n) n 2n Seroussi/Weinberger – Lossless source coding 26-May-04 9 Bernoulli Models: Mixture Code n q The enumerative code length is close to -log Q’(x ), where -1 n æ n ö 1 q Is Q( ) a probability assignment? Q¢(x ) = ç ÷ × èn1 ø n +1 -1 1 1 æ n ö 1 P (xn )dq = q n1 (1-q )n0 dq = ç ÷ × = Q¢(xn ) Þ Q¢(x n ) = 1 ò0 q ò0 ç ÷ å èn1 ø n +1 x nÎAn qAha!! So we get the same result by mixing all the models in the class and using the mixture as a model! qThis uniform mixture is very appealing because it can be sequentially implemented, independent of n: n !n ! n-1 n (xt ) +1 ¢ n 0 1 ¢ t ¢ t 0 Q (x ) = = Õ q (xt+1 | x ) where q (0 | x ) = (n +1)! t=0 t + 2 l Laplace’s rule of succession! l this results in a “plug-in” approach: estimate q and plug it in Seroussi/Weinberger – Lossless source coding 26-May-04 10 Bernoulli Models: Mixture Code (cont.) qMaybe we can make Stirling work for us with a different mixture, that puts more weight in the problematic regions where n1/n approaches 0 and 1? 1 qConsider Dirichlet’s density w(q ) = Þ 1 2 G( 2 ) q (1-q ) 1 1 1 1 1 n - 1 n - 1 G(n + )G(n + ) Q¢¢(xn ) = P (xn )dw(q ) = q 1 2 (1-q ) 0 2 dq = 0 2 1 2 ò0 q 1 2 ò0 1 2 G( 2 ) n!G( 2 ) Þ by Stirling for the Gamma function, for all sequences xn L¢¢(xn ) 1 log n = - log Q¢¢(x n ) £ Hˆ (xn ) + + O(1/ n) Can we do any better? n n 2n qThis mixture also has a plug-in interpretation: n-1 n (xt ) + 1 ¢¢ n ¢¢ t ¢¢ t 0 2 Q (x ) = Õ q (xt+1 | x ) where q (0 | x ) = t=0 t +1 Seroussi/Weinberger – Lossless source coding 26-May-04 11 Summary of Codes q Two-part code l describe best parameter and then code based on it: works whenever the number of possible optimal codes in the class is sub-exponential l the most natural approach (akin to Kolmogorov complexity), but not sequential l not efficient in the example: can be improved by describing the parameter more coarsely Þ trade-off: spend less bits on the parameter and use an approximation of best parameter l alternatively, don’t allocate code words to impossible sequences q Mixture code l can be implemented sequentially l a suitable prior on the parameter gave the smallest redundancy q Plug-in codes l use a biased estimate of the conditional probability in the model class based on the data seen so far l can often be interpreted as mixtures q But what’s the best we can do? Seroussi/Weinberger – Lossless source coding 26-May-04 12 Normalized Maximum-Likelihood Code q Goal: find a code that attains the best worst-case pointwise redundancy n 1 n n RC = min max RC (L, x ) = min max[L(x ) - min LC (x )] L xn ÎAn n L x n ÎAn CÎC q Consider the code defined by the probability assignment n -min LC (x ) CÎC n n 2 1 é -min LC ( x ) ù Q(x ) = R (L, x n ) = log 2 CÎC -min L (xn ) Þ C ê å ú C CÎC n n n å 2 ë x ÎA û xnÎAn This quantity depends only on the class! q Since the redundancy of this code is the same for all xn, it must be optimal in the minimax sense! q Drawbacks: - sequential probability assignment depends on horizon n - hard to compute Seroussi/Weinberger – Lossless source coding 26-May-04 13 Example: NML Code for the Bernoulli Class q We evaluate the minimax redundancy RC for the Bernoulli class: ) n é n ù é æ n ö i ù 1 -H ( x ) 1 -h( n ) 1 np RC = logê å 2 ú = logêå ç ÷2 ú = log + o(1/ n) n ëx n ÎAn û n ë i=0 è i ø û 2n 2 typical of sufficient statistics: Hint: Stirling + n n n n 1 P (x ) = p(x | s(x )) p (s(x )) dq 2 q q =G( 1 ) = p ò0 q (1-q ) 2 Þ the pointwise redundancy cannot vanish faster than (log n)/2n (Dirichlet mixture or NML code) Seroussi/Weinberger – Lossless source coding 26-May-04 14 Parametric Model Classes q A useful limitation to the model class is to assume C = {Pq, q Î Qd } a parameter space of dimension d q Examples: l Bernoulli: d = 1 , general i.i.d.
Load more

Recommended publications
  • 16.1 Digital “Modes”
    Contents 16.1 Digital “Modes” 16.5 Networking Modes 16.1.1 Symbols, Baud, Bits and Bandwidth 16.5.1 OSI Networking Model 16.1.2 Error Detection and Correction 16.5.2 Connected and Connectionless 16.1.3 Data Representations Protocols 16.1.4 Compression Techniques 16.5.3 The Terminal Node Controller (TNC) 16.1.5 Compression vs. Encryption 16.5.4 PACTOR-I 16.2 Unstructured Digital Modes 16.5.5 PACTOR-II 16.2.1 Radioteletype (RTTY) 16.5.6 PACTOR-III 16.2.2 PSK31 16.5.7 G-TOR 16.2.3 MFSK16 16.5.8 CLOVER-II 16.2.4 DominoEX 16.5.9 CLOVER-2000 16.2.5 THROB 16.5.10 WINMOR 16.2.6 MT63 16.5.11 Packet Radio 16.2.7 Olivia 16.5.12 APRS 16.3 Fuzzy Modes 16.5.13 Winlink 2000 16.3.1 Facsimile (fax) 16.5.14 D-STAR 16.3.2 Slow-Scan TV (SSTV) 16.5.15 P25 16.3.3 Hellschreiber, Feld-Hell or Hell 16.6 Digital Mode Table 16.4 Structured Digital Modes 16.7 Glossary 16.4.1 FSK441 16.8 References and Bibliography 16.4.2 JT6M 16.4.3 JT65 16.4.4 WSPR 16.4.5 HF Digital Voice 16.4.6 ALE Chapter 16 — CD-ROM Content Supplemental Files • Table of digital mode characteristics (section 16.6) • ASCII and ITA2 code tables • Varicode tables for PSK31, MFSK16 and DominoEX • Tips for using FreeDV HF digital voice software by Mel Whitten, KØPFX Chapter 16 Digital Modes There is a broad array of digital modes to service various needs with more coming.
    [Show full text]
  • Shannon's Coding Theorems
    Saint Mary's College of California Department of Mathematics Senior Thesis Shannon's Coding Theorems Faculty Advisors: Author: Professor Michael Nathanson Camille Santos Professor Andrew Conner Professor Kathy Porter May 16, 2016 1 Introduction A common problem in communications is how to send information reliably over a noisy communication channel. With his groundbreaking paper titled A Mathematical Theory of Communication, published in 1948, Claude Elwood Shannon asked this question and provided all the answers as well. Shannon realized that at the heart of all forms of communication, e.g. radio, television, etc., the one thing they all have in common is information. Rather than amplifying the information, as was being done in telephone lines at that time, information could be converted into sequences of 1s and 0s, and then sent through a communication channel with minimal error. Furthermore, Shannon established fundamental limits on what is possible or could be acheived by a communication system. Thus, this paper led to the creation of a new school of thought called Information Theory. 2 Communication Systems In general, a communication system is a collection of processes that sends information from one place to another. Similarly, a storage system is a system that is used for storage and later retrieval of information. Thus, in a sense, a storage system may also be thought of as a communication system that sends information from one place (now, or the present) to another (then, or the future) [3]. In a communication system, information always begins at a source (e.g. a book, music, or video) and is then sent and processed by an encoder to a format that is suitable for transmission through a physical communications medium, called a channel.
    [Show full text]
  • Information Theory and Coding
    Information Theory and Coding Introduction – What is it all about? References: C.E.Shannon, “A Mathematical Theory of Communication”, The Bell System Technical Journal, Vol. 27, pp. 379–423, 623–656, July, October, 1948. C.E.Shannon “Communication in the presence of Noise”, Proc. Inst of Radio Engineers, Vol 37 No 1, pp 10 – 21, January 1949. James L Massey, “Information Theory: The Copernican System of Communications”, IEEE Communications Magazine, Vol. 22 No 12, pp 26 – 28, December 1984 Mischa Schwartz, “Information Transmission, Modulation and Noise”, McGraw Hill 1990, Chapter 1 The purpose of telecommunications is the efficient and reliable transmission of real data – text, speech, image, taste, smell etc over a real transmission channel – cable, fibre, wireless, satellite. In achieving this there are two main issues. The first is the source – the information. How do we code the real life usually analogue information, into digital symbols in a way to convey and reproduce precisely without loss the transmitted text, picture etc.? This entails proper analysis of the source to work out the best reliable but minimum amount of information symbols to reduce the source data rate. The second is the channel, and how to pass through it using appropriate information symbols the source. In doing this there is the problem of noise in all the multifarious ways that it appears – random, shot, burst- and the problem of one symbol interfering with the previous or next symbol especially as the symbol rate increases. At the receiver the symbols have to be guessed, based on classical statistical communications theory. The guessing, unfortunately, is heavily influenced by the channel noise, especially at high rates.
    [Show full text]
  • Source Coding: Part I of Fundamentals of Source and Video Coding Full Text Available At
    Full text available at: http://dx.doi.org/10.1561/2000000010 Source Coding: Part I of Fundamentals of Source and Video Coding Full text available at: http://dx.doi.org/10.1561/2000000010 Source Coding: Part I of Fundamentals of Source and Video Coding Thomas Wiegand Berlin Institute of Technology and Fraunhofer Institute for Telecommunications | Heinrich Hertz Institute Germany [email protected] Heiko Schwarz Fraunhofer Institute for Telecommunications | Heinrich Hertz Institute Germany [email protected] Boston { Delft Full text available at: http://dx.doi.org/10.1561/2000000010 Foundations and Trends R in Signal Processing Published, sold and distributed by: now Publishers Inc. PO Box 1024 Hanover, MA 02339 USA Tel. +1-781-985-4510 www.nowpublishers.com [email protected] Outside North America: now Publishers Inc. PO Box 179 2600 AD Delft The Netherlands Tel. +31-6-51115274 The preferred citation for this publication is T. Wiegand and H. Schwarz, Source Coding: Part I of Fundamentals of Source and Video Coding, Foundations and Trends R in Signal Processing, vol 4, nos 1{2, pp 1{222, 2010 ISBN: 978-1-60198-408-1 c 2011 T. Wiegand and H. Schwarz All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, mechanical, photocopying, recording or otherwise, without prior written permission of the publishers. Photocopying. In the USA: This journal is registered at the Copyright Clearance Cen- ter, Inc., 222 Rosewood Drive, Danvers, MA 01923. Authorization to photocopy items for internal or personal use, or the internal or personal use of specific clients, is granted by now Publishers Inc for users registered with the Copyright Clearance Center (CCC).
    [Show full text]
  • Fundamental Limits of Video Coding: a Closed-Form Characterization of Rate Distortion Region from First Principles
    1 Fundamental Limits of Video Coding: A Closed-form Characterization of Rate Distortion Region from First Principles Kamesh Namuduri and Gayatri Mehta Department of Electrical Engineering University of North Texas Abstract—Classical motion-compensated video coding minimum amount of rate required to encode video. On methods have been standardized by MPEG over the the other hand, a communication technology may put years and video codecs have become integral parts of a limit on the maximum data transfer rate that the media entertainment applications. Despite the ubiquitous system can handle. This limit, in turn, places a bound use of video coding techniques, it is interesting to note on the resulting video quality. Therefore, there is a that a closed form rate-distortion characterization for video coding is not available in the literature. In this need to investigate the tradeoffs between the bit-rate (R) paper, we develop a simple, yet, fundamental character- required to encode video and the resulting distortion ization of rate-distortion region in video coding based (D) in the reconstructed video. Rate-distortion (R-D) on information-theoretic first principles. The concept of analysis deals with lossy video coding and it establishes conditional motion estimation is used to derive the closed- a relationship between the two parameters by means of form expression for rate-distortion region without losing a Rate-distortion R(D) function [1], [6], [10]. Since R- its generality. Conditional motion estimation offers an D analysis is based on information-theoretic concepts, elegant means to analyze the rate-distortion trade-offs it places absolute bounds on achievable rates and thus and demonstrates the viability of achieving the bounds derives its significance.
    [Show full text]
  • Beyond Traditional Transform Coding
    Beyond Traditional Transform Coding by Vivek K Goyal B.S. (University of Iowa) 1993 B.S.E. (University of Iowa) 1993 M.S. (University of California, Berkeley) 1995 A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Engineering|Electrical Engineering and Computer Sciences in the GRADUATE DIVISION of the UNIVERSITY OF CALIFORNIA, BERKELEY Committee in charge: Professor Martin Vetterli, Chair Professor Venkat Anantharam Professor Bin Yu Fall 1998 Beyond Traditional Transform Coding Copyright 1998 by Vivek K Goyal Printed with minor corrections 1999. 1 Abstract Beyond Traditional Transform Coding by Vivek K Goyal Doctor of Philosophy in Engineering|Electrical Engineering and Computer Sciences University of California, Berkeley Professor Martin Vetterli, Chair Since its inception in 1956, transform coding has become the most successful and pervasive technique for lossy compression of audio, images, and video. In conventional transform coding, the original signal is mapped to an intermediary by a linear transform; the final compressed form is produced by scalar quantization of the intermediary and entropy coding. The transform is much more than a conceptual aid; it makes computations with large amounts of data feasible. This thesis extends the traditional theory toward several goals: improved compression of nonstationary or non-Gaussian signals with unknown parameters; robust joint source–channel coding for erasure channels; and computational complexity reduction or optimization. The first contribution of the thesis is an exploration of the use of frames, which are overcomplete sets of vectors in Hilbert spaces, to form efficient signal representations. Linear transforms based on frames give representations with robustness to random additive noise and quantization, but with poor rate–distortion characteristics.
    [Show full text]
  • 19It501 Information Theory and Coding Techniques
    19IT501 INFORMATION THEORY AND CODING L T P C TECHNIQUES 3 0 0 3 PREAMBLE This course is used to design an application with error control. It is used to help to identify the multimedia concepts. It is one of the most important and direct applications of information theory and also make use of compression and decompression techniques. PREREQUISITE NIL COURSE OUTCOMES At the end of the course learners will be able to CO1 Construct Huffman coding applications Understand CO2 Recall Differential pulse code modulation Apply CO3 Design an application with error–control Apply CO4 Apply the concepts of multimedia communication Analyze CO5 Make Use of compression and decompression techniques Evaluate MAPPING OF COs WITH POs AND PSOs PROGRAMME COURSE PROGRAMME OUTCOMES SPECIFIC OUTCOMES OUTCOMES PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 PO10 PO11 PO12 PSO1 PSO2 CO1 - 2 3 2 3 1 - - - - 3 - 3 - CO2 1 1 3 2 2 1 - - - - - - - - CO3 1 1 3 2 2 1 - - - - 1 - - 1 CO4 1 3 3 1 1 1 - - - - 1 - 3 - CO5 - 2 3 1 3 - - - - - - - 3 - CO6 - 2 3 1 3 - - - - - 2 - - 1 1. LOW 2. MODERATE 3. SUBSTANTIAL CONCEPT MAP Information Theory and Coding Techniques Information Data and voice Error control Compression coding TheoryFundame coding Techniques Modulation ntals Codes Text Differential pulse Syndrome Image Source coding code Decoding Audio Huffman Adaptive Video cyclic codes coding Differential pulse Polynomial & Shannon coding code Parity check Channel coding Delta Polynomial Channel capacity Encoder 1. Source coding Development of Multimedia application SYLLABUS UNIT I INFORMATION ENTROPY FUNDAMENTALS 9 Uncertainty, Information and Entropy – Source coding Theorem – Huffman coding –Shannon Fanocoding – Discrete Memory less channels – channel capacity – channel coding Theorem – Channel capacity Theorem.
    [Show full text]
  • 01 2007 DCC.Pmd
    TPSK31: Getting the Trellis Coded Modulation Advantage A Tutorial on Information Theory, Coding, and Us with an homage to Claude Shannon and Ungerboeck’s genius applied to PSK31. By: Bob McGwier, N4HY As just about everyone who is alive and awake and not suffering from some awful brain disease knows, PSK31 is an extremely popular mode for doing keyboard to keyboard communications. Introduced in the late 1990’s (1,2) it quickly swept the amateur radio world. Its major characteristics are a 31.25 baud differentially encoded PSK signal with keyboard characters encoded into an compression algorithm called varicode by Peter (1) before being modulated with the differentially encoded psk. This differential psk has some serious advantages on HF, including insensitivity to sideband selection, no need for a training sequence to allowing break in decoding to occur and myriad others discussed elsewhere. The slow data rate allows it to use our awful HF channels on 40 and 20 meters effectively and it is more than fast enough to have a good conversation just as we have always done with RTTY. Further, it is very spectrally efficient. If we are going to improve on it, we better not mess with these characteristics. We will not. Peter will be the first person to tell you that he is not a trained communications theorist, engineer, or coding expert. Yet he is one of the great native intelligences in all of amateur radio. His inspirational muse (lend me some!) is one of the best and his perspiration coefficient are among the highest ever measured in amateur radio (this means he is smart and works hard, an extremely good combination).
    [Show full text]
  • Quantum Rate Distortion, Reverse Shannon Theorems, and Source-Channel Separation Nilanjana Datta, Min-Hsiu Hsieh, and Mark M
    1 Quantum rate distortion, reverse Shannon theorems, and source-channel separation Nilanjana Datta, Min-Hsiu Hsieh, and Mark M. Wilde Abstract—We derive quantum counterparts of two key the- of the original data, then the compression is said to be orems of classical information theory, namely, the rate distor- lossless. The simplest example of an information source is tion theorem and the source-channel separation theorem. The a memoryless one. Such a source can be characterized by a rate-distortion theorem gives the ultimate limits on lossy data random variable U with probability distribution pU (u) and compression, and the source-channel separation theorem implies f g that a two-stage protocol consisting of compression and channel each use of the source results in a letter u being emitted with coding is optimal for transmitting a memoryless source over probability pU (u). Shannon’s noiseless coding theorem states a memoryless channel. In spite of their importance in the P that the entropy H (U) u pU (u) log2 pU (u) of such classical domain, there has been surprisingly little work in these an information source is≡ the − minimum rate at which we can areas for quantum information theory. In the present paper, we prove that the quantum rate distortion function is given in compress signals emitted by it [49], [21]. terms of the regularized entanglement of purification. We also The requirement of a data compression scheme being loss- determine a single-letter expression for the entanglement-assisted less is often too stringent a condition, in particular for the quantum rate distortion function, and we prove that it serves case of multimedia data, i.e., audio, video and still images as a lower bound on the unassisted quantum rate distortion or in scenarios where insufficient storage space is available.
    [Show full text]
  • Entropy Coding and Different Coding Techniques
    Journal of Network Communications and Emerging Technologies (JNCET) www.jncet.org Volume 6, Issue 5, May (2016) Entropy Coding and Different Coding Techniques Sandeep Kaur Asst. Prof. in CSE Dept, CGC Landran, Mohali Sukhjeet Singh Asst. Prof. in CS Dept, TSGGSK College, Amritsar. Abstract – In today’s digital world information exchange is been 2. CODING TECHNIQUES held electronically. So there arises a need for secure transmission of the data. Besides security there are several other factors such 1. Huffman Coding- as transfer speed, cost, errors transmission etc. that plays a vital In computer science and information theory, a Huffman code role in the transmission process. The need for an efficient technique for compression of Images ever increasing because the is a particular type of optimal prefix code that is commonly raw images need large amounts of disk space seems to be a big used for lossless data compression. disadvantage during transmission & storage. This paper provide 1.1 Basic principles of Huffman Coding- a basic introduction about entropy encoding and different coding techniques. It also give comparison between various coding Huffman coding is a popular lossless Variable Length Coding techniques. (VLC) scheme, based on the following principles: (a) Shorter Index Terms – Huffman coding, DEFLATE, VLC, UTF-8, code words are assigned to more probable symbols and longer Golomb coding. code words are assigned to less probable symbols. 1. INTRODUCTION (b) No code word of a symbol is a prefix of another code word. This makes Huffman coding uniquely decodable. 1.1 Entropy Encoding (c) Every source symbol must have a unique code word In information theory an entropy encoding is a lossless data assigned to it.
    [Show full text]
  • MIT6 451S05 Fulllecnotes.Pdf
    Chapter 1 Introduction The advent of cheap high-speed global communications ranks as one of the most important developments of human civilization in the second half of the twentieth century. In 1950, an international telephone call was a remarkable event, and black-and-white television was just beginning to become widely available. By 2000, in contrast, an intercontinental phone call could often cost less than a postcard, and downloading large files instantaneously from anywhere in the world had become routine. The effects of this revolution are felt in daily life from Boston to Berlin to Bangalore. Underlying this development has been the replacement of analog by digital communications. Before 1948, digital communications had hardly been imagined. Indeed, Shannon’s 1948 paper 1 [7] may have been the first to use the word “bit.” Even as late as 1988, the authors of an important text on digital communications [5] could write in their first paragraph: Why would [voice and images] be transmitted digitally? Doesn’t digital transmission squander bandwidth? Doesn’t it require more expensive hardware? After all, a voice- band data modem (for digital transmission over a telephone channel) costs ten times as much as a telephone and (in today’s technology) is incapable of transmitting voice signals with quality comparable to an ordinary telephone [authors’ emphasis]. This sounds like a serious indictment of digital transmission for analog signals, but for most applications, the advantages outweigh the disadvantages . But by their second edition in 1994 [6], they were obliged to revise this passage as follows: Not so long ago, digital transmission of voice and video was considered wasteful of bandwidth, and the cost .
    [Show full text]
  • Source–Channel Coding in Networks
    Thesis for the degree of Doctor of Philosophy Source–Channel Coding in Networks Niklas Wernersson Communication Theory School of Electrical Engineering KTH (Royal Institute of Technology) Stockholm 2008 Wernersson, Niklas Source–Channel Coding in Networks Copyright c 2008 Niklas Wernersson except where otherwise stated. All rights reserved. TRITA-EE 2008:028 ISSN 1653-5146 ISBN 978-91-7415-002-5 Communication Theory School of Eletrical Engineering KTH (Royal Institute of Technology) SE-100 44 Stockholm, Sweden Telephone + 46 (0)8-790 7790 Abstract The aim of source coding is to represent information as accurately as possible us- ing as few bits as possible and in order to do so redundancy from the source needs to be removed. The aim of channel coding is in some sense the contrary, namely to introduce redundancy that can be exploited to protect the information when be- ing transmitted over a nonideal channel. Combining these two techniques leads to the area of joint source–channel coding which in general makes it possible to achieve a better performance when designing a communication system than in the case when source and channel codes are designed separately. In this thesis four particular areas in joint source–channel coding are studied: analog (i.e. continu- ous) bandwidth expansion, distributed source coding over noisy channels, multiple description coding (MDC) and soft decoding. A general analog bandwidth expansion code based on orthogonal polynomi- als is proposed and analyzed. The code has a performance comparable with other existing schemes. However, the code is more general in the sense that it is imple- mentable for a larger number of source distributions.
    [Show full text]