Arithmetic coding Elias coding Arithmetic coding as finite-precision Elias coding Redundancy due to finite precision Multiplication-free arithmetic coding Context-adaptive coding Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 1 Elias coding Entropy coding algorithm for sequences of symbols x with general (conditional) probabilities 1 0.1111 0.111 Representation of x by a subinterval of the unit 0.1101 interval [0,1) 0.11 Width of the subinterval is approximately equal to 0.1011 0.101 the probability fX(x) 0.1001 Subinterval for x can be determined by recursive 0.1 subdivision algorithm 0.0111 0.011 Represent x by shortest binary fraction in the 0.0101 fX x subinterval 0.01 0.0011 Subinterval of width f (x) is guaranteed to contain X 0.001 one number that can be represented by L binary 0.0001 digits, with 0 Lflog x 0 2 X Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 2 Example: Elias coding of memoryless binary source 13 ff0 1 XX44 4 16 37 148 cann 1 16 64 256 1024 1 1 1 1 1 1 4 0 0 cn 0 1 7 28 121 16 64 256 1024 Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 3 Example: Elias coding of memoryless ternary source 1 1 f X a f X b f X c 4 2 Courtesy: T. Wiegand Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 4 Elias coding: choose binary fraction in subinterval Uniquely identify interval [cn,cn+an) by a binary fraction 0.bbbbbb b [ cn , c n a n ) Ln bits Length of bit-string 1 Ln log2 1 hX x0:n1 1 0:n1 an Bit-string ˆ LLnn cn2 2 c n 1 c n Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 5 Successive decoding Elias bit string can be decoded symbol by symbol, starting with the first symbol For each symbol n=0,1,2, . • Calculate the intervals [cn,cn+an) corresponding to all possible xn ˆ • Determine the interval for which cn c n, c n a n • Emit the corresponding xn Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 6 Elias coding for memoryless source Elias coding algorithm Start with unit interval Initialize ca000 and 1. n-th symbol Interval For each n 0,1, to be width encoded Update an1 a n f X x n Update cn1 c n a n F X x n Interval lower limit Cumulative distribution (excluding current symbol) i1 FfX i X j j0 where XK {0 , 1 , 2 ,..., 1 } Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 7 Elias coding for changing probabilities Elias coding algorithm Start with unit interval Initialize ca000 and 1. n-th symbol Interval For each n 0,1, to be width Update a a f x encoded n1 n Xn n Update cn1 c n a n F X x n Interval n lower limit Cumulative distribution (excluding current symbol) i1 Ff Xnn i X j j0 where { , , ,..., } XKn 0 1 2 1 Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 8 Elias coding for Markov random processes Elias coding algorithm for Markov-p sources Start with unit interval Initialize ca000 and 1. For each n 0,1,... Up to p previous Set p min p , n symbols Interval Update a a f x ,x width n1 n Xn |X n p:1 n n n p : n 1 Update c c a F x ,x n1 n n Xn |X n p:1 n n n p : n 1 Interval lower limit n-th symbol Cumulative conditional to be distribution (excluding encoded current symbol) Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 9 Arithmetic coding Elias coding not practical for long symbol strings: required arithmetic precision grows with string length Finite precision implementations: “arithmetic coding” Widely used in modern image and video compression algorithms: JBIG, JPEG, JPEG-2000, H.263, H.264/AVC xn Arithmetic bit stream c Arithmetic coder decoder PMF fx Xnn Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 10 Finite precision for arithmetic coding N-bit precision is permissible, if rounding down Initialize ca000 and 1. For each n 0,1, Update an1 a n f X x n Update cn1 c n a n F X x n At most N+P LSBs change, except for a possible carry affecting MSBs. P-bit approximation of Each bit can be affected by probabilities at most one carry. Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 11 Finite precision for arithmetic coding (cont.) Output code string cn . xxxxxxxxx xxx 01111111 1 cccccccc c MSBs that willrn 1 bits N P LSBs that no longer changethat might be represent lower affected by a interval boundary for carry further computation Represent by Represent by Send state variable rn state variable Cn Maximum value of rn can be limited by bit stuffing Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 12 Inefficiency due to interval rounding Recall: Subinterval of width fX(x) is guaranteed to contain one number that can be represented by Ln binary digits, with Ln log2 fX x0:n1 0:n1 Hence, rounding one interval value an+1 increases bit string by N 1 an f X xn 2 1 1 log log log 1 21N 21N 2 2 N 1 2 a 2 ln2 n1 Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 13 Limited precision probabilities Efficiency loss fXX x f x Elog2 fXX x 1 E log 2 f x E log 2 1 f x f x XX HX( ) 1 fxX E ln 2 fxX fXX x f x E fXX x f x 0 fXX x xx f x provided that rounded probabilities still add to 1 Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 14 Example: inefficiency for binary source due to representing smaller probability by 2-P rate increase rate - relative bit relative log2 p Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 15 Multiplication-free binary arithmetic coding Interval width is normalized to 1/ 2 A 1 by binary shift after each symbol Approximation, with p ( L ) p ( M ) : Ap()() L p L p L ApM( ) A (1 pL ( )) A pL ( ) ApL Corresponds to approximating of p(L) within a factor of 2: p L p L A Factor can be lumped into external probabilities Optimum depends somewhat on data, typically 23 or 0.708 or 34 JBIG Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 16 MPS-LPS switch Problem: interval width for symbol M can become negative 1 A p L 0 if p L A ,1 2 pL 1 Solution: switch the role of M and L, if 2 Problem: interval for symbol M can be smaller than interval for L if 1 11 p L A p L p L A 42, 2 Solution: “Conditional exchange” of symbols, when problem occurs Improves compression efficiency, implemented in JPEG and JBIG Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 17 Sufficiency of binary coders B0 MSB K B1 Consider r.v. X 0,1, ,2 1 equivalent to a r.v. B BK 1 No loss by conditionally encoding bits (in any order) LSB HXH B HBHBBHBBBB | | , , 0 1 0 KK 1 0 1 2 Supply arithmetic coder with symbol/probability pairs b, p , b , p , , b , p with p f 0,b 0 0 1 1 K 1 K 1 k Bkk |B0: 1 0: k 1 Total number of (conditional) probabilities to be estimated per symbol x 1 2 2KK1 2 1 . same as with K-ary encoding of x Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 18 Adaptive probability estimation Consider stationary binary Markov-k process X n with probabilities “Context vector,” ffX|XX0,xx 0: k 1 X | 0, ( n k ): n 1 k0: k 1 n ( n k ): n 1 2k possible combinations Context labeling function, enumerating all possible context vectors k xx: 0,1 0,1, ,2k 1 Backward probability estimation separately for each context Probability of symbol Cn, x x[n]=0 0 (n k ): n 1 pn0 C n,,xx C n 0 (n k ): n 1 1 ( n k ): n 1 Count of zeroes that have previously Bias for low counts, occured in that context typically 1 Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 19 Adaptive probability estimation (cont.) Tracking changing statistics, compromise between • Rapid adaptation (small past sample counts) • Accurate estimates (large past sample counts) Scaled count estimation algorithm Initialize CC010 For n 0,1, If xn 1 C11C 1 else C00C 1 If min C0 ,CCCC 1 min or max C 0 , 1 max C0 C1 C01 ; C 22 C0 Estimate pn0 CC01 2 Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 20 Context adaptive coding 2-d extension of Markov model “causal half-plane” n “Coding context” n xn Markov conditional independence property f X n,X f X n,X X n X n X n X n n n Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 21 Context adaptive coding (cont.) Coding context vector can be mapped into context label directly without loss • Feasible for binary images • Example: JBIG uses 10 binary pixels as context label For 8-bit images, number of different contexts 256 • Context might have to be clustered • Combine with prediction Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no.