ECE644 – Lecture 2
Asymptotic Equipartition Property (AEP) Data Compression
01/23/02 Information Theory, Spring 2002, 1 Hairong Qi
Recap ()= − () () H X ∑ p x log2 p x ( )= − () () H X ,Y ∑∑p x, y log2 p x, y x, y H ()Y | X = ∑ p ()(x H Y | X = x )= −∑∑p ()()x, y log p y | x x xy () ()p()x D p || q = ∑ p x log () x q x () ()= () p x, y I X ;Y ∑∑p x, y log ()() x, y p x p y
01/23/02 Information Theory, Spring 2002, 2 Hairong Qi
Venn Diagram
H ()X H ()X ,Y H ()Y
H ()()(X ,Y = H X + H Y | X ) = H ()Y + H (X | Y ) I()()()X ;Y = H X − H X | Y = H ()Y − H (Y | X ) = H ()X + H ()Y − H (X ,Y )
H ()X |Y I ()X ;Y H ()Y | X
01/23/02 Information Theory, Spring 2002, 3 Hairong Qi
1 Asymptotic Equipartition Property
1 1 log Weak law of large AEP ()m n p X1, X 2 , X n numbers Is close to the For i.i.d. random entropy H 1 n variables, ∑ X i is n i=1 close to its expected value EX for large values of n ()m −nH p X1, X 2 , , X n is close to 2
01/23/02 Information Theory, Spring 2002, 4 Hairong Qi
The AEP
m () If X1, X 2 , are i.i.d. ~ p x , then
1 1 log → H ()X ()m n p X1, X 2 , X n
01/23/02 Information Theory, Spring 2002, 5 Hairong Qi
The Typical Set
()n The typical set Aε with respect to p()x is the set of ()m ∈ℜn sequences x1, x2 , , xn with the following property −n()H ()X +ε ≤ ()m ≤ −n()H ()X −ε 2 p x1, x2 , , xn 2 Other properties: 1 1) H()X −ε ≤ − log p()()x , x ,m, x ≤ H X + ε n 1 2 n ()n 2) Pr{}Aε >1−ε for n sufficiently large ()n n()H ()X +ε 3) Aε ≤ 2
01/23/02 Information Theory, Spring 2002, 6 Hairong Qi
2 Data Compression - Problem
Let X1, X2, …, Xn be i.i.d. random variables drawn from the probability mass function p(x) Try to find short descriptions for such sequences of random variables
01/23/02 Information Theory, Spring 2002, 7 Hairong Qi
Data Compression - AEP
Divide all sequence into two sets: the typical set and the non-typical set
all sequence ℜn
Non-typical set
()n What is Aε ?
01/23/02 Information Theory, Spring 2002, 8 Hairong Qi
Expected Length of the Codeword
E(l(X n ))= ??
01/23/02 Information Theory, Spring 2002, 9 Hairong Qi
3 Calculation of Typical Set
Consider a sequence of I.I.d. binary random
variables X1, X2, …, Xn, where the probability that Xi=1 is 0.6 – (a) Calculate H(X) – (b) With n=25 and ε=0.1, which sequences fall in the typical set? – (c) What is the probability of the typical set? – (d) How many elements are there in the typical set?
01/23/02 Information Theory, Spring 2002, 10 Hairong Qi
Homework 1 (Due 1/30/02)
1) Prove the three equations on Lecture1 slide 17 2) Problem 16 3) Calculation of typical set described in slide 10 of lecture 2 4) Read handout and identify research area 5) Reading: “Claude Shannon: Reluctant Father of the Digital Age” by M. Mitchell Waldrop 6) Experiment: “Shannon’s experiment to calculate the Entropy of English” 7) Reading: “A mathematical theory of communication” by Claude E. Shannon
01/23/02 Information Theory, Spring 2002, 11 Hairong Qi
4