Data Compression Techniques Part 1: Entropy Coding Lecture 2: Variable-Length Codes Continued

Juha K¨arkk¨ainen

01.11.2017

1 / 16 Kraft’s Inequality

When constructing a variable-length code, we are not really interested in what the individual codewords are as long as they satisfy two conditions:

I The code is a prefix code (or at least a uniquely decodable code).

I The codeword lengths are chosen to minimize the average codeword length. Kraft’s inequality gives an exact condition for the existence of a prefix code in terms of the codeword lengths. Theorem (Kraft’s Inequality)

There exists a binary prefix code with codeword lengths `1, `2, . . . , `σ if and only if σ X 2−`i ≤ 1 . i=1

2 / 16 Proof of Kraft’s Inequality

Consider a binary search on the real interval [0, 1). In each step, the current interval is split into two halves and one of the halves is chosen as the new interval.

We can associate a search of ` steps with a binary string of length `:

I Zero corresponds to choosing the left half.

I One corresponds to choosing the right half. For any binary string w, let I(w) be the final interval of the associated search.

Example 1011 corresponds to the search sequence [0, 1), [1/2, 2/2), [2/4, 3/4), [5/8, 6/8), [11/16, 12/16) and I(1011) = [11/16, 12/16).

3 / 16 Consider the set {I(w) | w ∈ {0, 1}`} of all intervals corresponding to binary strings of lengths `. Clearly, an interval [x, y) belongs to this set if and only if −` I The length y − x of the interval is 2 . −` I Both end points x and y are multiples of 2 . Such intervals are called dyadic intervals. For any two binary strings w and w 0, the intervals I(w) and I(w 0) overlap if and only if one of the strings is a prefix of the other. Example I(1011) = [11/16, 12/16) ⊂ I(101) = [5/8, 6/8) Now we are ready to prove the theorem starting with the “only if” part. Let C be a prefix code with the codewords w1, w2,..., wσ of lengths `1, `2, . . . , `σ. Since C is a prefix code, the intervals I(wi ), i ∈ [1..σ], do not overlap with each other, and their total length is at most 1. Thus σ σ X −`i X 2 = |I(wi )| ≤ 1 . i=1 i=1

4 / 16 Finally, we prove the “if” part. Let `1, `2, . . . , `σ be arbitrary integers satisfying the inequality. We will construct a prefix code with those codeword lengths. We start by sorting the lengths in nondecreasing order, i.e., we assume that `1 ≤ `2 ≤ · · · ≤ `σ. For i ∈ [1..σ], let h  i−1 −`i X −`j [xi , yi ) = Li−1, Li−1 + 2 where Li−1 = 2 j=1

Example The lengths 1, 2, 3 result the intervals [0, 1/2), [1/2, 3/4), [3/4, 7/8).

The intervals [xi , yi ) are dyadic: −` I The length of the interval [xi , yi ) is 2 i . −` I Both end points xi and yi are multiples 2 i . This is because, for all j < i, 2−`j is a multiple of 2−`i . Thus [xi , yi ) = I(wi ) for some binary string wi of length `i . Since the intervals do not overlap, the binary strings are not prefixes of each other. Thus w1, w2,..., wσ are valid codewords for a prefix code with the required lengths. 5 / 16 McMillan’s Inequality Prefix codes are a subclass of uniquely decodable codes. McMillan’s inequality generalizes the result for all uniquely decodable codes. Theorem (McMillan’s Inequality) There exists a uniquely decodable binary code with codeword lengths `1, `2, . . . , `σ if and only if σ X 2−`i ≤ 1 . i=1 Proof omitted. Thus considering only prefix codes (rather than all uniquely decodable codes) is not a significant restriction with respect to compression. Corollary For any uniquely decodable binary code, there exists a binary prefix code with the same codeword lengths. Thus a Huffman code is optimal among all uniquely decodable codes. 6 / 16 Redundancy in Prefix Codes

Using Kraft’s inequality, we can also quantify redundancy in prefix codes. Definition A prefix code satisfying Kraft’s inequality with strict inequality (P 2−`i < 1) is called redundant. A prefix code satisfying Kraft’s inequality with equality (P 2−`i = 1) is called complete. In other words, a prefix code is redundant if some part of the interval [0, 1) is not covered by any codeword interval.

Most good codes including Huffman code are complete.

7 / 16 A redundant code can always be improved by reducing some codeword length. Lemma

For any redundant prefix code with codeword lengths `1, `2, . . . , `σ, there 0 0 0 exists a complete prefix code with codeword lengths `1, `2, . . . , `σ such 0 0 that `i ≤ `i for all i ∈ [1..σ] and `i < `i for some i.

Proof.

Assume `σ is the longest codeword length in the redundant code. For all i ∈ [1..σ], 2−`i is a multiple of 2−`σ . Thus the redundancy gap 1 − P 2−`i −`σ is a multiple of 2 too. If we reduce `σ by one, we increase the sum P 2−`i by 2−`σ and still satisfy Kraft’s inequality. If the resulting code is still redundant, repeat the procedure until it becomes complete. In a complete code, a codeword length can be reduced only if some other codeword lengths are simultaneously increased or removed.

8 / 16 Canonical Prefix Codes A prefix code constructed from codeword lengths using the procedure in the “if” part of the proof of Kraft’s inequality is called canonical. A Huffman code is often replaced with a canonical Huffman code, where the Huffman algorithm is used only to compute the codeword lengths. Example Initial Huffman code Canonical Huffman code C(a) = 10 C(a) = 00 C(e) = 00 C(e) = 01 C(i) = 011 C(o) = 10 C(o) = 11 C(i) = 110 C(u) = 0100 C(u) = 1110 C(y) = 0101 C(y) = 1111 A canonical code enables some practical optimizations: I Store the code by listing only codeword lengths. I Faster decoding by using the fact that codewords with the same length form a lexicographical range. 9 / 16 Integer Codes

Next we look at some variable-length codes for integers. We want to have a prefix code when the source alphabet is the set N of natural numbers.

We cannot use the Huffman algorithm on an infinite alphabet. Even if there is an upper bound on the integers, the alphabet may be too large for efficient computation and storage of the code.

Instead, we can define the code as a simple function. Each of the following codes are complete canonical prefix codes, where the codeword length grows monotonically with the integer value. The codes differ in how fast the growth is.

The codes described here are for all non-negative integers including zero. If the

minimum integer value nmin is larger than zero, we can shift the codes, i.e., use the code for n − nmin to encode n. The case nmin = 1 is common in the literature.

10 / 16 Let us first define the standard binary codes that are used as a building block in the other codes. This is not a prefix code for natural numbers but a family of fixed-length codes for finite integer ranges. Definition (Binary code) For any k ≥ 0 and n ∈ [0..2k − 1]

binaryk (n) = k-bit binary representation of n. Our first proper prefix code is the unary code. Definition (Unary code) For any n ∈ N unary(n) = 1n0 The length of the unary code for n is n + 1. Example binary4(6) = 0110 and unary(6) = 1111110.

11 / 16 The unary code lengths grow too fast for many purposes. Elias γ (gamma) and δ (delta) codes achieve a logarithmic growth rates. Definition (Elias γ and δ code) For any n ∈ N

k γ(n) = unary(k) · binaryk (n − 2 + 1) k δ(n) = γ(k) · binaryk (n − 2 + 1) where k = blog(n + 1)c. The lengths of the codes are

|γ(n)| = 2blog(n + 1)c + 1 |δ(n)| = dlog(n + 2)e + 2blogdlog(n + 2)ec

Example If n = 6, then k = 2 and γ(6) = 110 11 and δ(6) = 101 11.

12 / 16 Golomb–Rice codes are a family of prefix codes GR0, GR1, GR2, ... with a codeword length growth rate between unary code and Elias codes. Definition (Golomb–Rice codes) For any k ≥ 0 and any n ∈ N

k GRk (n) = unary(q) · binaryk (n − q2 ) where q = bn/2k c. The codeword lengths are

k |GRk (n)| = bn/2 c + k + 1

Example

GR1(6) = 1110 0, GR2(6) = 10 10, GR3(6) = 0 110 There are many possible variants and extension of the codes we have seen.

13 / 16 Example

n unary(n) γ(n) δ(n) GR2(n) GR4(n) 0 0 0 0 000 00000 1 10 100 1000 001 00001 2 110 101 1001 010 00010 3 1110 11000 10100 011 00011 4 11110 11001 10101 1000 00100 5 111110 11010 10110 1001 00101 6 1111110 11011 10111 1010 00110 7 11111110 1110000 11000000 1011 00111 8 111111110 1110001 11000001 11000 01000 9 1111111110 1110010 11000010 11001 01001 10 111...10 1110011 11000011 11010 01010 20 111...10 111100101 110010101 11111000 100100 30 111...10 111101111 110011111 1111111010 101110 40 111...10 11111001001 1101001001 1111111111000 1101000 50 111...10 11111010011 1101010011 111...1010 11100010 60 111...10 11111011101 1101011101 111...1000 11101100

14 / 16 All of these codes are complete canonical prefix codes for natural numbers. Example The intervals corresponding to the codewords (as in the proof of Kraft’s inequality) of length at most 7 for unary, γ and GR2 codes.

unary

γ

GR2

When used for a finite range of integers, the codes are redundant as they leave a short unused gap at the end of unit interval. However, the redundancy gap is insignificant for a large integer range such as [0..232).

15 / 16 VByte Code For efficiency reasons, we may sometimes want to encode integers using a variable number of bytes instead of bits. The most popular such code is called VByte: I One bit in each byte tells whether the byte is the last one in the codeword. I The remaining 7 bits from each byte in the codeword are concatenated and interpreted as a single binary number. I In a simple but redundant variant, the binary number is the encoded integer. 7 I In a non-redundant variant, the integers [0...2 − 1] are encoded with one byte, the integers [27...27 + 214 − 1] with two bytes, the integers [27 + 214...27 + 214 + 221 − 1] with three bytes etc. VByte encoding is faster to encode and decode than binary codes.

When encoding a lot of very small integers, VByte code can use a lot more space than the bit codes, but still much less than a fixed-length code if we also need to code some large integers. 16 / 16