Information theory

LING 572 Fei Xia

1 Information theory

• Reading: M&S 2.2

• It is the use of probability theory to quantify and measure “information”.

• Basic concepts: – Entropy – Cross entropy and relative entropy – Joint entropy and conditional entropy – Entropy of the language and perplexity – Mutual information 2 Entropy

• Entropy is a measure of the uncertainty associated with a distribution. H(X )   p(x)log p(x) x Here, X is a random variable, x is a possible outcome of X.

• The lower bound on the number of bits that it takes to transmit messages.

• An example: – Display the results of a 8-horse race. – Goal: minimize the number of bits to encode the results. 3 An example

• Uniform distribution: pi=1/8.

1 1 H(X )   p(x)log p(x)  8*( log2 )  3 bits x 8 8 • Non-uniform distribution: (1/2, 1/4, 1/8, 1/16, 1/64, 1/64, 1/64, 1/64)

1 1 1 1 1 1 1 1 1 1 H(X )  ( log  log  log  log  4* log )  2 bits 2 2 4 4 8 8 16 16 64 64 (0, 10, 110, 1110, 111100, 111101, 111110, 111111)

Uniform distribution has a higher entropy. MaxEnt: make the distribution as “uniform” as possible. 4 Cross Entropy

• Entropy: H (X )   p(x)log p(x) x • Cross Entropy: H c (X )   p(x)log q(x) x

Here, p(x) is the true probability; q(x) is our estimate of p(x).

H c (X )  H (X )

5 Relative Entropy

• Also called Kullback-Leibler divergence: p(x) KL( p || q)  p(x)log  H (X )  H(X )  2 q(x) c

• A “distance” measure between probability functions p and q; the closer p(x) and q(x) are, the smaller the relative entropy is.

• KL divergence is asymmetric, so it is not a proper distance metric: KL( p,q)  KL(q, p)

6 Joint and conditional entropy

• Joint entropy: H(X ,Y)   p(x, y)log p(x, y) x y

• Conditional entropy: H (Y | X )  H (X ,Y )  H (X )

7 Entropy of a language (per-word entropy)

• The entropy of a language L:

 p(x1n )log p(x1n ) H (L, p)   lim x1n n n

• If we make certain assumptions that the language is “nice”, then the cross entropy can be calculated as: (Shannon-Breiman-Mcmillan Theorem) log p(x ) log p(x ) H (L, p)   lim 1n   1n n n n

8 Per-word entropy (cont)

• p(x1n) can be calculated by n-gram models

• Ex: unigram model

9 Perplexity

H(L,p) • Perplexity PP(x1n) is 2 .

• Perplexity is the weighted average number of choices a random variable has to make.

• Perplexity is often used to evaluate a language model; lower perplexity is preferred.

10 Mutual information

• It measures how much is in common between X and Y: p(x, y) I(X ;Y )   p(x, y)log x y p(x) p(y)  H (X )  H (Y )  H (X ,Y )  I(Y; X )  H (X )  H (X |Y )  H (Y )  H (Y | X )

• I(X;Y) = KL( p(x,y) || p(x)p(y) )

• If X and Y are independent, I(X;Y) is 0. 11 Summary on Information theory

• Reading: M&S 2.2

• It is the use of probability theory to quantify and measure “information”.

• Basic concepts: – Entropy – Cross entropy and relative entropy – Joint entropy and conditional entropy – Entropy of the language and perplexity – Mutual information 12 Additional slides

13 Conditional entropy

H (Y | X )   p(x)H (Y | X  x) x   p(x) p(y | x)log p(y | x) x y   p(x, y)log p(y | x) x y   p(x, y)log p(x, y) / p(x) x y   p(x, y)(log p(x, y)  log p(x)) x y   p(x, y)log p(x, y)   p(x, y)log p(x) x y x y   p(x, y)log p(x, y)   p(x)log p(x) x y x

 H (X ,Y )  H (X ) 14 Mutual information

p(x, y) I(X ;Y )   p(x, y)log x y p(x) p(y)   p(x, y)log p(x, y)  p(x, y)log p(x)  p(x, y)log p(y) x y x y y x  H (X ,Y )  log p(x) p(x, y)  log p(y) p(x, y) x y y x  H (X ,Y )  (log p(x)) p(x)  (log p(y)) p(y) x y  H (X )  H (Y )  H (X ,Y )  I(Y; X )

15