Arithmetic coding
Elias coding Arithmetic coding as finite-precision Elias coding Redundancy due to finite precision Multiplication-free arithmetic coding Context-adaptive coding
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 1 Elias coding
Entropy coding algorithm for sequences of symbols x with general (conditional) probabilities 1 0.1111 0.111 Representation of x by a subinterval of the unit 0.1101 interval [0,1) 0.11 Width of the subinterval is approximately equal to 0.1011 0.101 the probability fX(x) 0.1001 Subinterval for x can be determined by recursive 0.1 subdivision algorithm 0.0111 0.011 Represent x by shortest binary fraction in the 0.0101 fX x subinterval 0.01 0.0011 Subinterval of width f (x) is guaranteed to contain X 0.001 one number that can be represented by L binary 0.0001 digits, with 0 Lflog x 0 2 X
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 2 Example: Elias coding of memoryless binary source
13 ff0 1 XX44
4 16 37 148 cann 1 16 64 256 1024
1 1 1 1 1
1 4 0 0 cn 0 1 7 28 121 16 64 256 1024
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 3 Example: Elias coding of memoryless ternary source
1 1 f X a f X b f X c 4 2
Courtesy: T. Wiegand Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 4 Elias coding: choose binary fraction in subinterval
Uniquely identify interval [cn,cn+an) by a binary fraction
0.bbbbbb b [ c , c a ) n n n Ln bits Length of bit-string 1 Ln log2 1 hX x0:n1 1 a 0:n1 n Bit-string ˆ LLnn cn2 2 c n 1 c n
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 5 Successive decoding
Elias bit string can be decoded symbol by symbol, starting with the first symbol
For each symbol n=0,1,2, . . .
• Calculate the intervals [cn,cn+an) corresponding to all possible xn ˆ • Determine the interval for which cn c n, c n a n
• Emit the corresponding xn
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 6 Elias coding for memoryless source
Elias coding algorithm Start with unit interval
Initialize ca00 0 and 1. n-th symbol Interval For each n 0,1, to be width encoded Update an1 a n f X x n
Update cn1 c n a n F X x n Interval lower limit
Cumulative distribution (excluding current symbol) i1 FfX i X j j0
where XK {0 , 1 , 2 ,..., 1 }
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 7 Elias coding for changing probabilities
Elias coding algorithm Start with unit interval
Initialize ca00 0 and 1.
n-th symbol Interval For each n 0,1, to be width Update a a f x encoded n1 n Xn n
Update cn1 c n a n F X x n Interval n lower limit
Cumulative distribution (excluding current symbol) i1 Ff Xnn i X j j0 where { , , ,..., } XKn 0 1 2 1
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 8 Elias coding for Markov random processes
Elias coding algorithm for Markov-p sources Start with unit interval
Initialize ca00 0 and 1. For each n 0,1,... Up to p previous Set p min p , n symbols Interval Update a a f x ,x width n1 n Xn |X n p:1 n n n p : n 1 Update c c a F x ,x n1 n n Xn |X n p:1 n n n p : n 1 Interval lower limit
n-th symbol Cumulative conditional to be distribution (excluding encoded current symbol)
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 9 Arithmetic coding
Elias coding not practical for long symbol strings: required arithmetic precision grows with string length Finite precision implementations: “arithmetic coding” Widely used in modern image and video compression algorithms: JBIG, JPEG, JPEG-2000, H.263, H.264/AVC
xn Arithmetic bit stream c Arithmetic coder decoder
PMF fx Xnn
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 10 Finite precision for arithmetic coding
N-bit precision is permissible, if rounding down
Initialize ca00 0 and 1. For each n 0,1,
Update an1 a n f X x n
Update cn1 c n a n F X x n
At most N+P LSBs change, except for a possible carry affecting MSBs. P-bit approximation of Each bit can be affected by probabilities at most one carry.
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 11 Finite precision for arithmetic coding (cont.)
Output code string c . xxxxxxxxx xxx 01111111 1 cccccccc c n MSBs that willrn 1 bits N P LSBs that no longer changethat might be represent lower affected by a interval boundary for carry further computation
Represent by Represent by Send state variable rn state variable Cn
Maximum value of rn can be limited by bit stuffing
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 12 Inefficiency due to interval rounding
Recall: Subinterval of width fX(x) is guaranteed to contain one number that can be represented by Ln binary digits, with
Ln log2 fX x0:n1 0:n1
Hence, rounding one interval value an+1 increases bit string by
N 1 an f X xn 2 1 1 log log log 1 21N 21N 2 2 N 1 2 a 2 ln2 n1
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 13 Limited precision probabilities
Efficiency loss fXX x f x Elog2 fXX x 1 E log 2 f x E log 2 1 f x f x XX HX ( )
1 fxX E ln 2 fxX
fXX x f x E fXX x f x 0 fXX x xx f x
provided that rounded probabilities still add to 1
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 14 Example: inefficiency for binary source
due to representing smaller probability by 2-P
rate increase rate
- relative bit relative
log p 2
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 15 Multiplication-free binary arithmetic coding
Interval width is normalized to 1/ 2 A 1 by binary shift after each symbol Approximation, with p ( L ) p ( M ) : Ap()() L p L p L
ApM( ) A (1 pL ( )) A pL ( ) ApL
Corresponds to approximating of p(L) within a factor of 2: p L p L A Factor can be lumped into external probabilities
Optimum depends somewhat on data, typically 23 or 0.708 or 34 JBIG
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 16 MPS-LPS switch
Problem: interval width for symbol M can become negative 1 A p L 0 if p L A ,1 2
pL 1 Solution: switch the role of M and L, if 2
Problem: interval for symbol M can be smaller than interval for L if
1 11 p L A p L p L A 42 , 2 Solution: “Conditional exchange” of symbols, when problem occurs Improves compression efficiency, implemented in JPEG and JBIG
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 17 Sufficiency of binary coders
B0 MSB K B1 Consider r.v. X 0,1, ,2 1 equivalent to a r.v. B
BK 1 No loss by conditionally encoding bits (in any order) LSB HXH B
HBHBBHBBBB 0 1| 0 KK 1 | 0 , 1 , 2 Supply arithmetic coder with symbol/probability pairs
b0, p 0 , b 1 , p 1 , , bK 1 , p K 1 with p k f B |B 0,b 0: k 1 kk0: 1
Total number of (conditional) probabilities to be estimated per symbol x KK1 1 2 2 2 1 . . . same as with K-ary encoding of x
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 18 Adaptive probability estimation
Consider stationary binary Markov-k process X n with probabilities
“Context vector,” ffX|XX0,xx 0: k 1 X | 0, ( n k ): n 1 k0: k 1 n ( n k ): n 1 2k possible combinations Context labeling function, enumerating all possible context vectors k xx: 0,1 0,1, ,2k 1 Backward probability estimation separately for each context
Probability of symbol Cn, x x[n]=0 0 (n k ): n 1 pn0 C0 n,,xx (n k ): n 1 C 1 n ( n k ): n 1 Count of zeroes that have previously Bias for low counts, occured in that context typically 1
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 19
Adaptive probability estimation (cont.)
Tracking changing statistics, compromise between • Rapid adaptation (small past sample counts) • Accurate estimates (large past sample counts) Scaled count estimation algorithm
Initialize CC01 0 For n 0,1,
If xn 1 C11C 1 else
C00C 1
If min C0 ,CCCC 1 min or max C 0 , 1 max
C0 C1 C01 ; C 22 C0 Estimate pn0 CC01 2
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 20 Context adaptive coding
2-d extension of Markov model “causal half-plane” n
“Coding context”
n xn Markov conditional independence property f X n,X f X n,X n X n X n X n X n n
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 21 Context adaptive coding (cont.)
Coding context vector can be mapped into context label directly without loss • Feasible for binary images • Example: JBIG uses 10 binary pixels as context label For 8-bit images, number of different contexts 256 • Context might have to be clustered • Combine with prediction
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 22 Reading
W. B. Pennebaker, J. L. Mitchell, G. G. Langdon, Jr., R. B. Arps, “An overview of the basic principles of the Q-Coder adaptive binary arithmetic coder,” IBM J. Res. Develop., vol. 32, no. 6, November 1988. Witten, Radford, Neal, Cleary, “Arithmetic Coding for Data Compression” Communications of the ACM, vol. 30, no. 6, pp. 520-540, June 1987. Moffat, Neal, Witten, “Arithmetic Coding Revisited,” ACM Transactions on Information Systems, vol. 16, vo. 3, pp. 256–294, July 1998.
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 23