Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Lecture 5 Lossless Coding (II) May 20, 2009 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Outline Review Arithmetic Coding Dictionary Coding Run-Length Coding Lossless Image Coding Data Compression: Hall of Fame References 1 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Review Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Image and video encoding: A big picture Differential Coding Motion Estimation and Compensation A/D Conversion Context-Based Coding Color Space Conversion … Pre-Filtering Predictive Partitioning Coding … Input Image/Video Post- Pre- Lossy Lossless Processing Processing Coding Coding (Post-filtering) Quantization Entropy Coding Transform Coding Dictionary-Based Coding Model-Based Coding Run-Length Coding Encoded … … Image/Video 3 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding The ingredients of entropy coding A random source (X, P) A statistical model (X, P’) as an estimation of the random source An algorithm to optimize the coding performance (i.e., to minimize the average codeword length) At least one designer … 4 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding FLC, VLC and V2FLC FLC = Fixed-length coding/code(s)/codeword(s) • Each symbol xi emitted from a random source (X, P) is encoded as an n-bit codeword, where |X|≤2n. VLC = Variable-length coding/code(s)/codeword(s) • Each symbol xi emitted from a random source (X, P) is encoded as an ni-bit codeword. • FLC can be considered as a special case of VLC, where n1=…=n|X|. V2FLC = Variable-to-fixed length coding/code(s)/codeword(s) • A symbol or a string of symbols is encoded as an n-bit codeword. • V2FLC can also be considered as a special case of VLC. 5 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Static coding vs. Dynamic/Adaptive coding Static coding = The statistical model P’ is static, i.e., it does not change over time. Dynamic/Adaptive coding = The statistical model P’ is dynamically updated, i.e., it adapts itself to the context (i.e., changes over time). • Dynamic/Adaptive coding ⊂ Context-based coding Hybrid coding = Static + Dynamic coding • A codebook is maintained at the encoder side, and the encoder dynamically chooses a code for a number of symbols and inform the decoder about the choice. 6 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding A coding Shannon coding Shannon-Fano coding Huffman coding Arithmetic coding (Range coding) • Shannon-Fano-Elisa coding Universal coding • Exp-Golomb coding (H.264/MPEG-4 AVC, Dirac) • Elias coding family , Levenshtein coding , … Non-universal coding • Truncated binary coding, unary coding, … • Golomb coding ⊃ Rice coding Tunstall coding ⊂ V2FLC David Salomon, Variable-length Codes … for Data Compression, Springer, 2007 7 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shannon-Fano Code: An example X={A,B,C,D,E}, P={0.35,0.2,0.19,0.13,0.13}, Y={0,1} A Possible Code 0.35+0.2 0.19+0.13+0.13 A, B C, D, E A 00 B 01 0.13+0.13 0.35 0.2 0.19 D, E C 10 A B C D 110 0.13 0.13 E 111 D E 8 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Universal coding (code) A code is called universal if L≤C1(H+C2) for all possible values of H, where C1, C2≥1. • You may see a different definition somewhere, but the basic idea remains the same – a universal code works like an optimal code, except there is a bound defined by a constant C1. A universal code is called asymptotically optimal if C1→1 when H→∞. 9 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Coding positive/non-negative integers Naive binary coding • |X|=2k: k-bit binary representation of an integer. Truncated binary coding • |X|=2k+b: X={0,1,…,2k+b-1} 0 0 0 k k k 2 b 1 b0 bk 1 2 b b0 bk 2 + b 1 1 1 ⇒ ··· − − ⇒ //////////////////////··· − − ⇒ ··· − ⇒ ··· k k+1 | {zUnary} code (Stone-age binary coding) | {z } • |X|=∞: X=Z+={1,2,…} f(x)=0 0 1or1 1 0 ··· ··· x 1 x 1 − − | {z } | {z } 10 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Golomb coding and rice coding Golomb coding = Unary coding + Truncated binary coding • An integer x is divided into two parts (quotient and remainder) according to a parameter M: q = x/M , b c r=mod(x, M)=x-q*M. • Golumb code = unary code of q + truncated binary code of r. • When M=1, Golomb coding = unary coding. • When M=2k, Golomb coding = Rice coding. • Golomb code is the optimal code for the geometric distribution: Prob(x=i)=(1-p)i-1p, where 0<p<1. 11 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Exp-Golomb coding (Universal) Exp-Golomb coding ≠ Golomb coding Exp-Golomb coding of order k=0 is used in some video coding standards such as H.264. The encoding process • |X|=∞: X={0,1,2,…} k • Calculate q = x/ 2 +1 , and n q = log q . b c b 2 c • Exp-Golomb code = unary code of nq + nq LSBs of q + k-bit representation of r=mod(x,2k)=x-q*2k. 12 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Huffman code: An example X={1,2,3,4,5}, P=[0.4,0.2,0.2,0.1,0.1]. p1+2+3+4+5=1 A Possible Code p =0.6 1 00 1+2 p3+4+5=0.4 2 01 3 10 p1=0.4 p2=0.2 p4+5=0.2 p3=0.2 4 110 5 111 p4=0.1 p5=0.1 13 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Huffman code: An optimal code Relation between Huffman code and Shannon code: H≤LHuffman≤LHuffman-Fano≤LShannon<H+1 A stronger result (Gallager, 1978) • When pmax≥ 0.5, LHuffman≤H+pmax<H+1 • When pmax<0.5, LHuffman≤H+pmax+log2(2(log2e)/e)≈H+pmax+0.086<H+0.586 Huffman’s rules of optimal codes imply that Huffman code is optimal. • When each pi is a negative power of 2, Huffman code reaches the entropy. 14 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Huffman code: Small X problem Problem • When |X| is small, the coding performance is less obvious. • As a special case, when |X|=2, Huffman coding cannot compress the data at all – no matter what the probability is, each symbol has to be encoded as a single bit. Solutions • Solution 1: Work on Xn rather than X. • Solution 2: Dual tree coding = Huffman coding + Tunstall coding 15 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Huffman code: Variance problem Problem • There are multiple choices of two smallest probabilities, if more than two nodes have the same probability during any step of the coding process. • Huffman codes with a larger variance may cause trouble for data transmissions via a CBR (constant bit rate) channel – a larger buffer is needed. Solution • Shorter subtrees first. (A single node’s height is 0.) 16 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Modified Huffman code Problem • If |X| is too large, the construction of the Huffman tree will be too long and the memory used for the tree will be too demanding. Solution -v -v • Divide X into two set X1={si|p(si)>2 }, X2={si|p(si)≤2 }. • Perform Huffman coding for the new set X3=X1∪{X2}. • Append f(X2) as the prefix of naive binary representation of all symbols in X2. 17 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Huffman’s rules of making optimal codes Source statistics: P=P0=[p1,…,pm], where p1≥…≥pm-1≥pm. Rule 1: L1≤…≤Lm-1=Lm. Rule 2: If L1≤…≤Lm-2<Lm-1=Lm, Lm-1 and Lm differs from each other only for the last bit, i.e., f(xm-1)=b0 and f(xm)=b1, where b is a sequence of Lm-1 bits. Rule 3: Each possible bit sequence of length Lm-1 must be either a codeword or the prefix of some codewords. Answers: Read Section 5.2.1 (pp. 122-123) of the following book – Yun Q. Shi and Huifang Sun, Image and Video Compression for Multimedia Engineering, 2nd Edition, CRC Press, 2008 18 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Justify Huffman’s rules Rule 1 • If Li>Li+1, we can swap them to get a smaller average codeword length (when Pi=Pi+1 it does not make sense, though). • If Lm-1<Lm, then L1,…,Lm-1<Lm and the last bit of Lm is redundant. Rule 2 • If they do not have the same parent node, then the last bits of both codewords are redundant. Rule 3 • If there is an unused bit sequence of length Lm-1, we can use it for Lm. 19 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Arithmetic Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Why do we need a new coding algorithm? Problems with Huffman coding • Each symbol in X is represented by at least one bit. • Coding for Xn: A Huffman tree with |X|n nodes needs to be constructed. The value of n cannot be too large. • Encoding with a Huffman tree is quite easy, but decoding can be difficult especially when X is large. • Dynamic Huffman coding is slow due to the update of the Huffman tree from time to time. Solution: Arithmetic coding • Encode a number of symbols (can be a message of a very large size n) incrementally (progressively) as a binary fraction in a real range [low, high) ⊆ [0,1).