Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Outline

Lecture 5 Review Lossless Coding (II) Dictionary Coding Run-Length Coding Lossless Image Coding May 20, 2009

: Hall of Fame References

1

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Image and encoding: A big picture Differential Coding and Compensation A/D Conversion Context-Based Coding Conversion … Pre-Filtering Predictive Partitioning Coding Review … Input Image/Video Post- Pre- Lossy Lossless Processing Processing Coding Coding (Post-filtering)

Quantization Entropy Coding Dictionary-Based Coding Model-Based Coding Run-Length Coding Encoded … … Image/Video

3

1 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

The ingredients of entropy coding FLC, VLC and V2FLC

A random source (X, P) FLC = Fixed-length coding/(s)/codeword(s) • Each symbol xi emitted from a random source (X, P) is encoded as an A statistical model (X, P’) as an estimation n-bit codeword, where |X|≤2n. of the random source VLC = Variable-length coding/code(s)/codeword(s) • Each symbol x emitted from a random source (X, P) is encoded as an An algorithm to optimize the coding i ni-bit codeword.

performance (i.e., to minimize the average • FLC can be considered as a special case of VLC, where n1=…=n|X|. codeword length) V2FLC = Variable-to-fixed length coding/code(s)/codeword(s) • A symbol or a string of symbols is encoded as an n-bit codeword. • V2FLC can also be considered as a special case of VLC. At least one designer …

4 5

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Static coding vs. Dynamic/ A coding

Static coding = The statistical model P’ is static, i.e., it does not change over time. Shannon-Fano coding Dynamic/Adaptive coding = The statistical model Arithmetic coding (Range coding) P’ is dynamically updated, i.e., it adapts itself to • Shannon-Fano-Elisa coding the context (i.e., changes over time). Universal coding • Exp- (H.264/MPEG-4 AVC, Dirac) • Dynamic/Adaptive coding ⊂ Context-based coding • Elias coding family , Levenshtein coding , … Hybrid coding = Static + Dynamic coding Non-universal coding • Truncated binary coding, , … • A codebook is maintained at the encoder side, and the • Golomb coding ⊃ Rice coding encoder dynamically chooses a code for a number of ⊂ V2FLC David Salomon, Variable-length symbols and inform the decoder about the choice. … for Data Compression, Springer, 2007

6 7

2 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Shannon-Fano Code: An example Universal coding (code)

X={A,B,C,D,E}, A code is called universal if L≤C1(H+C2) for all possible values of H, where C , C ≥1. P={0.35,0.2,0.19,0.13,0.13}, Y={0,1} 1 2 • You may see a different definition somewhere, but the basic idea remains the same – a universal code works like an optimal code, except there is a bound defined by a constant C . A Possible Code 1 0.35+0.2 0.19+0.13+0.13 A universal code is called asymptotically optimal if C →1 A, B C, D, E A 00 1 when H→∞. B 01 0.13+0.13 0.35 0.2 0.19 D, E C 10 A B C D 110 0.13 0.13 E 111 D E

8 9

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Coding positive/non-negative integers Golomb coding and rice coding

Naive binary coding Golomb coding = Unary coding + Truncated • |X|=2k: k-bit binary representation of an integer. binary coding • An integer x is divided into two parts (quotient and Truncated binary coding remainder) according to a parameter M: q = x/M , b c • |X|=2k+b: X={0,1,…,2k+b-1} r=mod(x, M)=x-q*M. 0 0 0 k k k 2 b 1 b0 bk 1 2 b b0 bk 2 + b 1 1 1 ⇒ ··· − − ⇒ //////////////////////··· − − ⇒ ··· − ⇒ ··· • Golumb code = unary code of q + truncated binary k k+1 code of r. | {zUnary} code (Stone-age binary coding) | {z } • When M=1, Golomb coding = unary coding. • |X|=∞: X=Z+={1,2,…} • When M=2k, Golomb coding = Rice coding. f(x)=0 01or1 10 ··· ··· • Golomb code is the optimal code for the geometric x 1 x 1 − − distribution: Prob(x=i)=(1-p)i-1p, where 0

3 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Exp-Golomb coding (Universal) Huffman code: An example

Exp-Golomb coding ≠ Golomb coding X={1,2,3,4,5}, P=[0.4,0.2,0.2,0.1,0.1].

Exp-Golomb coding of order k=0 is used in p1+2+3+4+5=1 some video coding standards such as H.264. A Possible Code p =0.6 1 00 1+2 p3+4+5=0.4 The encoding process 2 01 • |X|=∞: X={0,1,2,…} 3 10 p =0.4 p =0.2 p =0.2 k 1 2 p =0.2 4+5 4 110 • Calculate q = x/ 2 +1 , and n q = log 2 q . 3 b c b c 5 111 • Exp-Golomb code = unary code of nq + nq LSBs of k k q + k-bit representation of r=mod(x,2 )=x-q*2 . p4=0.1 p5=0.1

12 13

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Huffman code: An optimal code Huffman code: Small X problem

Relation between Huffman code and Shannon code: Problem H L L L

4 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Huffman code: Variance problem Modified Huffman code

Problem Problem • There are multiple choices of two smallest • If |X| is too large, the construction of the Huffman tree will be too long and the memory used for the tree will be too demanding. probabilities, if more than two nodes have the same probability during any step of the coding process. Solution • Divide X into two set X ={s |p(s )>2-v}, X ={s |p(s )≤2-v}. • Huffman codes with a larger variance may cause 1 i i 2 i i • Perform Huffman coding for the new set X =X ∪{X }. trouble for data transmissions via a CBR (constant bit 3 1 2 • Append f(X ) as the prefix of naive binary representation of all rate) channel – a larger buffer is needed. 2 symbols in X2. Solution • Shorter subtrees first. (A single node’s height is 0.)

16 17

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Huffman’s rules of making optimal codes Justify Huffman’s rules

Source statistics: P=P0=[p1,…,pm], where p1≥…≥pm-1≥pm. Rule 1 If L >L , we can swap them to get a smaller average codeword Rule 1: L1≤…≤Lm-1=Lm. • i i+1 length (when Pi=Pi+1 it does not make sense, though). Rule 2: If L1≤…≤Lm-2

Answers: Read Section 5.2.1 (pp. 122-123) of the following book – Yun Q. • If there is an unused bit sequence of length Lm-1, we can use it for Shi and Huifang Sun, Image and Video Compression for Multimedia Lm. Engineering, 2nd Edition, CRC Press, 2008

18 19

5 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Why do we need a new coding algorithm?

Problems with Huffman coding • Each symbol in X is represented by at least one bit. • Coding for Xn: A Huffman tree with |X|n nodes needs to be constructed. The value of n cannot be too large. Arithmetic Coding • Encoding with a Huffman tree is quite easy, but decoding can be difficult especially when X is large. • Dynamic Huffman coding is slow due to the update of the Huffman tree from time to time. Solution: Arithmetic coding • Encode a number of symbols (can be a message of a very large size n) incrementally (progressively) as a binary fraction in a real range [low, high) ⊆ [0,1).

21

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

The name of the game: The history Associating input symbol(s) with intervals

Shannon-Fano-Elias coding (sometimes simply Elias coding) Given a source (X,P): P=[p(x1),…,p(xi),…,p(xm)] • Shannon actually mentioned something about it in 1948. ☺ • Elias invented the recursive implementation, but didn’t publish it at all. For any symbol xi∈X • Abramson introduced Elias’ idea in his book “ and Coding” (1963) as a note on pages 61-62. ☺ • The associated sub-interval is [pc(xi), Practical arithmetic coding pc(xi+1)=pc(xi)+p(xi)), where pc(x1)=0 and • Rissanen and Pasco proposed the finite-precision implementation in 1976 pc(xm+1)=pc(xm)+p(xm)=1. independently. The name “arithmetic coding” was coined by Rissanen. n • The idea was patented by the IBM (Langdon and Rissanen as the inventors) in For any n-symbol (n>1) word x=(xi1,…,xin)∈X 1970s and afterwards more related patents were filed. … The associated sub-interval is [p (x), p (x)+p(x)), • Witten, Neal and Cleary published source code in C in 1987, which made • c c arithmetic coding more popular. ☺ where pc(x) is the starting point of the interval Range coding corresponding to x and p(x)=p(xi1)…p(xin) for a • It was proposed by Martin in 1979, but it is just a different way (a patent-free one memoryless random source. ☺☺) of looking at arithmetic coding and implementing it.

22 23

6 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Arithmetic coding: An example Arithmetic coding: Which fraction?

X={A,B,C,D}, P=[0.6,0.2,0.1,0.1]. Which fraction in [pc(x), pc(x)+p(x)) should be chosen? l • One proper choice: 1/2 ≤p(x)/2 ⇒⇒l = log2(1/p(x)) +1 choose f(x) to be a binary integer i such dthat (i- e l l What is this? 1)/2 ≤pc(x)

24 25

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Arithmetic coding: Termination problem Arithmetic coding: Performance

Termination of the original message x Arithmetic coding: Hb(P)≤L

• Transmit the size of x to the decoder separately. Huffman coding: Hb(P)≤L Huffman coding! • Make use of semantic information.

26 27

7 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Arithmetic coding: Noticeable features Arithmetic coding: Practical issues

Non-block coding (Incremental coding) Finite-precision problem • The output of an encoder is not the concatenation of codewords of • Approximating probabilities and the arithmetic operations in consecutive symbols. finite-precision ⇒ loss of coding efficiency, but negligible if the precision is high enough. “Range” coding Common bits of the two end points can be removed immediately • The encoded message is actually a range in [0,1]. • and sent out. Separation between coding and the source’s statistics • A renormalization process is needed to re-calculate the new • The probability is used to divide a range with the coding process “values” of the two end points. (instead of before it). Carry-propagation problem • ⇒ Arithmetic coding is inherently dynamic/adaptive/context- A carry (caused by the probability addition) may propagate over q based coding. • bits (such as 0.011111110… ⇒ 0.100000000…), so an extra register has to be used to track this problem.

28 29

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Binary arithmetic coding Binary arithmetic coding

X=Y={0,1}, P=[p0,1-p0 ] n-bit finite precision is used: initially, [0,1) ⇒ [0,2n-1)

Update the range as usual: low’=low or low+(high-low)*p0, high’=low+(high-low)*p0 or high. Do nothing if high-low≥2n-1 Do renormalization when high-low<2n-1 • When low, high<2n-1 ⇒ output a 0-bit, and [0,2n-1) ⇒ [0,2n-1). • When low, high≥2n-1 ⇒ output a 1-bit, subtract 2n-1 from low, high, and [2n-1,2n-1) ⇒ [0,2n-1). • low<2n-1 (low=01…) but high≥2n-1 (high=10…) ⇒ store an unknown bit (which is the complement of the next known bit) in the buffer, subtract 2n-2 from low, high, and [2n-2,2n-1+2n-2) ⇒ [0,2n-1) After all symbols are encoded, terminate the process by sending out a sufficient number of bits to represent the value of low. 30 31

8 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Multiplication-free implementation Q- arithmetic coders

For binary arithmetic coding, we can further approximate Q-coder high-low≈α=¾ (⇐ high-low=½~1) • The first multiplication-free binary arithmetic coder. Then, the update of low and high can be approximated • It is less efficient than ideal arithmetic coders, but the loss of without doing the multiplication. compression efficiency is less than 6%. • Why was it named after “Q”? – high-low=A, pLPS=Q ⇒ low’=low The optimal value of α is actually depends on the or low+(A-Q), A’=A-Q or Q. probability p0, and in most cases smaller than ¾ (typical • The renormalization process ensures A∈[0.75,1.5) ⇒ A≈1 (α=⅔). value is ⅔ or around 0.7). QM-coder (JPEG/JBIG) high=low+A AQ≈Q When p0>1/(2α)=⅔, it is possible αp0>½>high-low, so • An enhanced edition of Q-coder. high’

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

The basic idea

Use a static/dynamic dictionary and encode a symbol or a sequence of symbols as the index of the dictionary. Dictionary Coding The dictionary can be implicit or explicit. The dictionary can be considered as an (maybe very rough) approximation of the probability distribution of the source.

35

9 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Static dictionary coding: Digram coding Adaptive dictionary coding

A static dictionary of size 2n contains |X| LZ (Lempel-Ziv) family symbols in X and also 2n-|X| most LZ77 (LZ1) and LZ78 (LZ2) : LZ77 + Huffman Coding 2 frequently used symbols in X (i.e., pairs of LZMA (Lempel-Ziv-Markov chain-Algorithm): DEFLATE + symbols in ). Arithmetic Coding X Code/Index Entry LZW (Lempel-Ziv-Welch): Improved LZ78 A 000 An example LZSS (Lempel-Ziv-Storer-Szymanski): Improved LZ77 B 001 LZO (Lempel-Ziv-Oberhumer): Fast LZ • |X|={A,B,C,D,E}, n=3 C 010 LZRW (Lempel-Ziv-Ross Williams): Improved LZ77 • ABDEBE ⇒ 101011100111 D011 E 100 LZJB (Lempel-Ziv-Jeff Bonwick): Improved LZRW1 AB 101 … AD 110 BE 111 36 37

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

LZ77 (Sliding window compression) LZ78

A sliding window = a search buffer + a look-ahead buffer Problems with LZ77 Only exploit local redundancies. • Nothing can be captured out of the sliding window. Use implicit dictionary. Solution: Using an explicit dictionary Encoding: “BADA…” ⇒ (o,l,“A”),… (o=l=0 for no • At the beginning, the dictionary is empty. match) Sliding window • An encoded symbol or a sequence of symbols that cannot be found in the dictionary is added into the l dictionary as a new entry. • The output is (i,c), where i is the dictionary index and c A E D B A D C E B A D A E A is the next symbol following the dictionary entry. Search buffer Look-ahead buffer

o 38 39

10 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

LZW

Main advantage over LZ78 • No need to transmit the next symbol c, but only i. Encoding procedure Run-Length Coding • Step 1: Initialize the dictionary to include all single symbols in X. • Step 2: Search the input sequence of symbols in the dictionary until xi+1…xj can be found but xi+1…xjxj+1 cannot, then output the index of xi+1…xj and add xi+1…xjxj+1 as a new entry in the dictionary. • Step 3: Repeat Step 2 until all symbols are encoded.

40

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

1-D run-length coding

Statistical model: Discrete Markov Source S={S1,S2} with 4 transition probabilities: P(Si/Sj), i, j∈{1,2}.

P(S /S ) 2 1 Lossless Image Coding P(S2/S2) P(S1/S1) S1 S2

P(S1/S2)

Runs: Repetition of symbols S1 or S2

How to code: (Si, Run1,Run2,…) • All the run-lengths form a new random source, which can be further coded with a (modified) Huffman coder or a universal code.

42

11 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

How entropy coding is used Lossless image coding standards

Huffman/Arithmetic coding + … JBIG (Joint Bi-level Image Experts Group): QM-coder Lossless JPEG family: Huffman coding/Arithmetic coding Dictionary coding + … JBIG 2: MQ-coder Universal codes + Predictive coding GIF (Graphics Interchange Format): LZW • Spatial difference tends to have a monotonic PNG (Portable Network Graphics): DELATE = LZ77 + probability distribution Huffman coding Universal codes + Run-length coding TIFF (Tagged Image File Format): RLE, LZW • The run-length pairs tends to have a monotonic Compressed BMP and PCX: RLE probability distribution …

44 45

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Lossy coding followed by lossless coding Differential Coding Motion Estimation and Compensation A/D Conversion Context-Based Coding Color Space Conversion … Pre-Filtering Predictive Partitioning Coding … Data Compression: Hall of Fame Input Image/Video Post- Pre- Lossy Lossless Processing Processing Coding Coding (Post-filtering)

Quantization Entropy Coding Transform Coding Dictionary-Based Coding Model-Based Coding Run-Length Coding Encoded … … Image/Video

46

12 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Hall of Fame

Abraham Lempel (1936-) • IEEE Richard W. Hamming Medal (2007) • Fellow of IEEE (1982) References Jacob Ziv (1931-) • IEEE Richard W. Hamming Medal (1995) • Israel Prize (1993) • Member (1981) and former President (1995-2004) of Israel National Academy of Sciences and Humanities • Fellow of IEEE (1973)

48

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

References for further reading

Norman L. Biggs, Codes: An Introduction to Information Communication and Cryptography, Sections 4.6-4.8, Springer, 2008

Khalid Sayood, Introduction to Data Compression, Chapter 4 “Arithmetic Coding” and Chapter 5 “Dictionary Techniques”, 3rd Edition, Morgan Kaufmann, 2005 Yun Q. Shi and Huifang Sun, Image and Video Compression for Multimedia Engineering: Fundamentals, Algorithms, and Standards, Chapter 6 “Run-Length and Dictionary Coding - Information Theory Results (III)”, 2nd Edition, CRC Press, 2008 David S. Taubman and Michael W. Marcellin, JPEG2000: Fundamentals, Standards and Practice, Section 2.3 “Arithmetic Coding”, Kluwer Academic Publishers, 2002 Tinku Acharya and Ping-Sing Tsai, JPEG2000 Standard for Image Compression: Concepts, Algorithms and VLSI Architectures, Sections 2.3-2.5, John Wiley & Sons, Inc., 2004 Mohammed Ghanbari, Standard : Image Compression to Advanced Video Coding, Section 3.4.2 “Arithmetic coding”, IEE, 2003

50

13