Arithmetic coding

 Elias coding  Arithmetic coding as finite-precision Elias coding  Redundancy due to finite precision  Multiplication-free arithmetic coding  Context-

Bernd Girod: EE398A Image and Compression Arithmetic Coding no. 1 Elias coding

 Entropy coding algorithm for sequences of symbols x with general (conditional) probabilities 1 0.1111 0.111  Representation of x by a subinterval of the unit 0.1101 interval [0,1) 0.11  Width of the subinterval is approximately equal to 0.1011 0.101 the probability fX(x) 0.1001  Subinterval for x can be determined by recursive 0.1 subdivision algorithm 0.0111 0.011  Represent x by shortest binary fraction in the 0.0101 fX x subinterval 0.01 0.0011  Subinterval of width f (x) is guaranteed to contain X 0.001 one number that can be represented by L binary 0.0001 digits, with 0 Lflog x 0 2 X  

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 2 Example: Elias coding of memoryless binary source

13 ff0   1 XX44

4 16 37 148 cann 1 16 64 256 1024

1 1 1 1 1

1 4 0 0 cn 0 1 7 28 121 16 64 256 1024

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 3 Example: Elias coding of memoryless ternary source

1 1 f X a f X b f X c 4 2

Courtesy: T. Wiegand Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 4 Elias coding: choose binary fraction in subinterval

 Uniquely identify interval [cn,cn+an) by a binary fraction

0.bbbbbb b [ c , c a ) n n n Ln  Length of -string   1   Ln  log2  1 hX x0:n1  1 a  0:n1   n   Bit-string ˆ LLnn cn2 2 c n  1  c n

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 5 Successive decoding

 Elias bit string can be decoded symbol by symbol, starting with the first symbol

 For each symbol n=0,1,2, . . .

• Calculate the intervals [cn,cn+an) corresponding to all possible xn ˆ • Determine the interval for which cn c n, c n a n 

• Emit the corresponding xn

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 6 Elias coding for memoryless source

Elias coding algorithm Start with unit interval

Initialize ca00 0 and 1. n-th symbol Interval For each n  0,1, to be width encoded Update an1  a n f X x n 

Update cn1  c n a n F X x n  Interval lower limit

Cumulative distribution (excluding current symbol) i1 FfX i   X j  j0

where XK {0 ,  1 ,  2 ,...,   1 }

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 7 Elias coding for changing probabilities

Elias coding algorithm Start with unit interval

Initialize ca00 0 and 1.

n-th symbol Interval For each n  0,1, to be width Update a a f x encoded n1 n Xn  n 

Update cn1  c n a n F X x n  Interval n lower limit

Cumulative distribution (excluding current symbol) i1 Ff Xnn i  X j  j0 where  { ,  ,  ,...,  } XKn 0 1 2 1

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 8 Elias coding for Markov random processes

Elias coding algorithm for Markov-p sources Start with unit interval

Initialize ca00 0 and 1. For each n  0,1,... Up to p previous Set p  min p , n symbols Interval Update a a f x ,x width n1 n Xn |X n p:1 n  n n  p : n  1  Update c c a F x ,x n1 n n Xn |X n p:1 n  n n  p : n  1  Interval lower limit

n-th symbol Cumulative conditional to be distribution (excluding encoded current symbol)

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 9 Arithmetic coding

 Elias coding not practical for long symbol strings: required arithmetic precision grows with string length  Finite precision implementations: “arithmetic coding”  Widely used in modern image and video compression algorithms: JBIG, JPEG, JPEG-2000, H.263, H.264/AVC

xn Arithmetic bit stream c Arithmetic coder decoder

PMF fx Xnn  

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 10 Finite precision for arithmetic coding

N-bit precision is permissible, if rounding down

Initialize ca00 0 and 1. For each n  0,1,

Update an1  a n f X x n 

Update cn1  c n a n F X x n 

At most N+P LSBs change, except for a possible carry affecting MSBs. P-bit approximation of Each bit can be affected by probabilities at most one carry.

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 11 Finite precision for arithmetic coding (cont.)

 Output code string c . xxxxxxxxx xxx 01111111 1 cccccccc c n MSBs that willrn  1 bits N P LSBs that no longer changethat might be represent lower affected by a interval boundary for carry further computation

Represent by Represent by Send state variable rn state variable Cn

 Maximum value of rn can be limited by bit stuffing

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 12 Inefficiency due to interval rounding

 Recall: Subinterval of width fX(x) is guaranteed to contain one number that can be represented by Ln binary digits, with

Ln  log2 fX x0:n1  0:n1

 Hence, rounding one interval value an+1 increases bit string by

N 1 an f X xn  2 1 1 log  log  log 1 21N  21N 2 2 N 1 2   a 2 ln2 n1

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 13 Limited precision probabilities

 Efficiency loss     fXX x   f x  Elog2 fXX x 1    E  log 2 f x  E  log 2  1   f x   f x  XX          HX ( )

1 fxX    E  ln 2 fxX  

fXX x f x E fXX x    f x  0 fXX x xx f x

provided that rounded probabilities still add to 1

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 14 Example: inefficiency for binary source

due to representing smaller probability by 2-P

rate increase rate

- relative bit relative

log p 2  

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 15 Multiplication-free binary arithmetic coding

 Interval width is normalized to 1/ 2  A 1 by binary shift after each symbol  Approximation, with p ( L )  p ( M ) : Ap()() L  p L  p L

ApM( ) A (1  pL ( ))  A   pL ( )  ApL   

 Corresponds to approximating of p(L) within a factor of 2:  p L  p L A  Factor  can be lumped into external probabilities

 Optimum  depends somewhat on data, typically 23  or   0.708 or   34 JBIG

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 16 MPS-LPS switch

 Problem: interval width for symbol M can become negative 1  A p L 0 if p L  A   ,1 2 

 pL  1 Solution: switch the role of M and L, if   2

 Problem: interval for symbol M can be smaller than interval for L if

1 11 p L  A  p  L  p  L  A  42 ,  2  Solution: “Conditional exchange” of symbols, when problem occurs  Improves compression efficiency, implemented in JPEG and JBIG

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 17 Sufficiency of binary coders

B0 MSB K B1  Consider r.v. X   0,1, ,2 1  equivalent to a r.v. B   

 BK 1  No loss by conditionally encoding bits (in any order) LSB HXH   B

HBHBBHBBBB 0  1| 0    KK 1 | 0 , 1 , 2   Supply arithmetic coder with symbol/probability pairs

b0, p 0 , b 1 , p 1 , , bK 1 , p K  1 with p k f B |B  0,b 0: k  1  kk0: 1

 Total number of (conditional) probabilities to be estimated per symbol x KK1 1 2   2  2  1 . . . same as with K-ary encoding of x

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 18 Adaptive probability estimation

 Consider stationary binary Markov-k process  X n  with probabilities

“Context vector,” ffX|XX0,xx 0: k 1  X | 0, ( n  k ): n  1  k0: k 1 n ( n  k ): n  1 2k possible combinations  Context labeling function, enumerating all possible context vectors k  xx:  0,1  0,1, ,2k  1  Backward probability estimation separately for each context

Probability of symbol Cn, x  x[n]=0 0 (n k ): n 1  pn0        C0 n,,xx (n k ): n  1    C 1 n  ( n  k ): n  1           Count of zeroes that have previously Bias for low counts, occured in that context typically 1

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 19

Adaptive probability estimation (cont.)

 Tracking changing statistics, compromise between • Rapid adaptation (small past sample counts) • Accurate estimates (large past sample counts)  Scaled count estimation algorithm

Initialize CC01 0 For n  0,1,

If xn  1 C11C 1 else

C00C 1

If min C0 ,CCCC 1  min or max C 0 , 1 max

C0 C1 C01 ; C 22 C0  Estimate pn0    CC01 2 

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 20 Context adaptive coding

 2-d extension of Markov model “causal half-plane”   n

“Coding context”

  n xn  Markov conditional independence property f X n,X  f X n,X     n  X n X    n  X n X    n  n

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 21 Context adaptive coding (cont.)

 Coding context vector can be mapped into context label directly without loss • Feasible for binary images • Example: JBIG uses 10 binary pixels as context label   For 8-bit images, number of different contexts 256 • Context might have to be clustered • Combine with prediction

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 22 Reading

 W. B. Pennebaker, J. L. Mitchell, G. G. Langdon, Jr., R. B. Arps, “An overview of the basic principles of the Q-Coder adaptive binary arithmetic coder,” IBM J. Res. Develop., vol. 32, no. 6, November 1988.  Witten, Radford, Neal, Cleary, “Arithmetic Coding for ” Communications of the ACM, vol. 30, no. 6, pp. 520-540, June 1987.  Moffat, Neal, Witten, “Arithmetic Coding Revisited,” ACM Transactions on Information Systems, vol. 16, vo. 3, pp. 256–294, July 1998.

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 23