ECE 5578 Multimedia Communication Lec 05 Arithmetic Coding Zhu Li Dept of CSEE, UMKC web: http://l.web.umkc.edu/lizhu phone: x2346 Z. Li, Multimedia Communciation, 2018 p.1 Outline Lecture 04 ReCap Arithmetic Coding About Homework-1 and Lab Z. Li, Multimedia Communciation, 2018 p.2 JPEG Coding Block (8x8 pel) based coding DCT transform to find sparse * representation Quantization reflects human visual system Zig-Zag scan to convert 2D to 1D string Run-Level pairs to have even more = compact representation Hoffman Coding on Level Category Quant Table: Fixed on the Level with in the category Z. Li, Multimedia Communciation, 2018 p.3 Coding of AC Coefficients Zigzag scanning: Example 8 24 -2 0 0 0 0 0 -31 -4 6 -1 0 0 0 0 0 -12 -1 2 0 0 0 0 0 0 -2 -1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Example: zigzag scanning result 24 -31 0 -4 -2 0 6 -12 0 0 0 -1 -1 0 0 0 2 -2 0 0 0 0 0 -1 EOB (Run, level) representation: (0, 24), (0, -31), (1, -4), (0, -2), (1, 6), (0, -12), (3, -1), (0, -1), (3, 2), (0, -2), (5, -1), EOB Z. Li, Multimedia Communciation, 2018 p.4 Coding of AC Coefficients Run / Base Run / Base … Run / Base codeword Catg. codeword Catg. Codeword Cat. EOB 1010 - - … ZRL 1111 1111 001 0/1 00 1/1 1100 … 15/1 1111 1111 1111 0101 0/2 01 1/2 11011 … 15/2 1111 1111 1111 0110 0/3 100 1/3 1111001 … 15/3 1111 1111 1111 0111 0/4 1011 1/4 111110110 … 15/4 1111 1111 1111 1000 0/5 11010 1/5 11111110110 … 15/5 1111 1111 1111 1001 … … … … … … … ZRL: represent 16 zeros when number of zeros exceeds 15. Example: 20 zeros followed by -1: (ZRL), (4, -1). (Run, Level) sequence: (0, 24), (0, -31), (1, -4), …… Run/Cat. Sequence: 0/5, 0/5, 1/3, … 24 is the 24-th entry in Category 5 (0, 24): 11010 11000 -4 is the 3-th entry in Category 3 (1, -4): 1111001 011 Z. Li, Multimedia Communciation, 2018 p.5 Outline Lecture 04 ReCap Arithmetic Coding . Basic Encoding and Decoding . Uniqueness and Efficiency . Scaling and Incremental Coding . Integer Implementation About Homework-1 and Lab Z. Li, Multimedia Communciation, 2018 p.6 Arithmetic Coding – The SciFi Story When I was in my 5th grade…. Aliens visit earth…. Z. Li, Multimedia Communciation, 2018 p.7 Huffman Coding Replacing an input symbol with a codeword Need a probability distribution Hard to adapt to changing statistics Need to store the codeword table Minimum codeword length is 1 bit Arithmetic Coding Replace the entire input with a single floating-point number Does not need the probability distribution Adaptive coding is very easy No need to keep and send codeword table Fractional codeword length Z. Li, Multimedia Communciation, 2018 p.8 Introduction Recall table look-up decoding of Huffman code . N: alphabet size . L: Max code word length 1 . Divide [0, 2^L] into N intervals 00 . One interval for one symbol . Interval size is roughly 010 011 proportional to symbol prob. 000 010 011 100 Arithmetic coding applies this idea recursively Normalizes the range [0, 2L] to [0, 1]. Map an input sequence (multiple symbols) to a unique tag in [0, 1). abcd….. dcba….. 0 1 Z. Li, Multimedia Communciation, 2018 p.9 Arithmetic Coding Symbol set and prob: a (0.8), b(0.02), c(0.18) Disjoint and complete partition of the range [0, 1) 0 1 [0, 0.8), [0.8, 0.82), [0.82, 1) Each interval corresponds to one symbol a b c Interval size is proportional to symbol probability The first symbol restricts the tag 0 1 position to be in one of the intervals The reduced interval is partitioned 0 1 recursively as more symbols are processed. 0 1 Observation: once the tag falls into an interval, it never gets out of it Z. Li, Multimedia Communciation, 2018 p.10 Some Questions to think about: Why compression is achieved this way? How to implement it efficiently? How to decode the sequence? Why is it better than Huffman code? Z. Li, Multimedia Communciation, 2018 p.11 Example: Symbol Prob. 1 2 3 1 0.8 2 0.02 0 0.8 0.82 1.0 3 0.18 Map to real line range [0, 1) Order does not matter Decoder needs to use the same order • Disjoint but complete partition: • 1: [0, 0.8): 0, 0.799999…9 • 2: [0.8, 0.82): 0.8, 0.819999…9 • 3: [0.82, 1): 0.82, 0.999999…9 Z. Li, Multimedia Communciation, 2018 p.12 Concept of the encoding (not practical) Input sequence: “1321” 1 2 3 Range 1 0 0.8 0.82 1.0 1 2 3 Range 0.8 0 0.64 0.656 0.8 1 2 3 Range 0.144 0.656 0.7712 0.77408 0.8 1 2 3 Range 0.00288 0.7712 0.773504 0.7735616 0.77408 Termination: Encode the lower end or midpoint to signal the end. Difficulties: 1. Shrinking of interval requires very high precision for long sequence. 2. No output is generated until the entire sequence has been processed. Z. Li, Multimedia Communciation, 2018 p.13 Encoder Pseudo Code Cumulative Density Function (CDF) Probability Mass Function For continuous distribution: 0.4 x = £ = 0.2 0.2 0.2 FX (x) P(X x) ò p(x)dx -¥ For discrete distribution: 1 2 3 4 X i F (i) = P(X £ i) = P(X = k) 1.0 X å 0.8 k =-¥ CDF P(X = i) = F (i) - F (i -1). 0.4 X X 0.2 Properties: Non-decreasing 1 2 3 4 X Piece-wise constant Each segment is closed at the lower end. Z. Li, Multimedia Communciation, 2018 p.14 Encoder Pseudo Code Keep track of LOW=0.0, HIGH=1.0; while (not EOF) { LOW, HIGH, RANGE n = ReadSymbol(); Any two are RANGE = HIGH - LOW; sufficient, e.g., HIGH = LOW + RANGE * CDF(n); LOW = LOW + RANGE * CDF(n-1); LOW and RANGE. } output LOW; Input HIGH LOW RANGE Initial 1.0 0.0 1.0 1 0.0+1.0*0.8=0.8 0.0+1.0*0 = 0.0 0.8 3 0.0 + 0.8*1=0.8 0.0 + 0.8*0.82=0.656 0.144 2 0.656+0.144*0.82=0.77408 0.656+0.144*0.8=0.7712 0.00288 1 0.7712+0.00288*0=0.7712 0.7712+0.00288*0.8=0.773504 0.002304 Z. Li, Multimedia Communciation, 2018 p.15 Concept of the Decoding Arithmetic Decoding (conceptual) Suppose encoder encodes the lower end: 0.7712 Zoom 1 2 3 In by * CDF Decode 1 0 0.8 0.82 1.0 1 2 3 Decode 3 * CDF 0 0.64 0.656 0.8 1 2 3 Decode 2 * CDF 0.656 0.7712 0.77408 0.8 1 2 3 Decode 1 0.7712 0.773504 0.7735616 0.77408 Drawback: need to recalculate all thresholds each time. Z. Li, Multimedia Communciation, 2018 p.16 Simplified Decoding (Floating Pt Ops) x - low Normalize RANGE to [0, 1) each time x ¬ range No need to recalculate the thresholds. Receive 0.7712 1 2 3 Decode 1 x =(0.7712-0) / 0.8 0 0.8 0.82 1.0 = 0.964 1 2 3 Decode 3 0 0.8 0.82 1.0 x =(0.964-0.82) / 0.18 = 0.8 1 2 3 Decode 2 x =(0.8-0.8) / 0.02 0 0.8 0.82 1.0 = 0 Decode 1. 1 2 3 Stop. 0 0.8 0.82 1.0 Z. Li, Multimedia Communciation, 2018 p.17 Decoder Pseudo Code Low = 0; high = 1; x = GetEncodedNumber(); While (x ≠ low) { n = DecodeOneSymbol(x); output symbol n; x = (x - CDF(n-1)) / (CDF(n) - CDF(n-1)); }; 1 00 010 011 000 010 011 100 But this method still needs high-precision floating point operations. It also needs to read all input at the beginning of the decoding. Z. Li, Multimedia Communciation, 2018 p.18 Outline Lecture 04 ReCap Arithmetic Coding . Basic Encoding and Decoding . Uniqueness and Efficiency . Scaling and Incremental Coding About Homework-1 and Lab Z. Li, Multimedia Communciation, 2018 p.19 Uniqueness 1 2 3 Range 0.00288 0.7712 0.773504 0.7735616 0.77408 Termination: Encode the midpoint (or lower end) to signal the end. How to represent the final tag uniquely and efficiently? Answer: Take the binary value of the tag T(X) and truncate to (X is the sequence {x1, …, xm}, not individual symbol) é 1 ù 1 bit longer than l(X ) = êlog ú +1 bits. Shannon code ê p(X )ú 1 Proof: Assuming midpoint tag: T(X) = F(X -1) + p(X ), p(X) > 0. 2 First show the truncated code is unique, that is, code is within [F(X-1), F(X)). 1). ëT(X)û l( X ) £ T(X ) < F(X ) So ëT(X)û l( X ) is below the high end of the interval. Z. Li, Multimedia Communciation, 2018 p.20 Uniqueness and Efficiency æ é 1 ù ö æ 1 ö -ç êlog ú+1÷ -ç log +1÷ ç p( X ) ÷ ç ÷ 1 2).