Asymptotic Equipartition Property An Analogous of the Law of Large Numbers Radu Tr^ımbit¸a¸s UBB October 2012 Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 1 / 19 Asymptotic equipartition property - IntroductionI 1 The law of large numbers states that for i.i.d. RVs n ∑ Xi is close to E (X ) for large values of n. The AEP states that states that 1 log 1 is close to the n p(X1,X2,...,Xn) entropy H. The probability p (X1, X2, ... , Xn) assigned to an observed sequence will be close to 2−nH . This enables us to divide the set of all sequences into two sets, the typical set, where the sample entropy is close to the true entropy, and the nontypical set, which contains the other sequences. Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 2 / 19 Asymptotic equipartition property - IntroductionII Definition 1 (Convergence of random variables) Given a sequence of RVs, X1, X2, ... we say that the sequences X1, X2, ... converges to a random variable X p 1 In probability (Xn −! X ) if for every # > 0, P (jXn − X j > #) ! 0. 2 2 In mean square if E (Xn − X ) ! 0. a.s. 3 With probability 1 (also called almost surelyX n −! X ) if P(limn!¥ Xn = X ) = 1. Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 3 / 19 Asymptotic equipartition property theoremI Theorem 2 (AEP) If X1, X2, ... are i.i.d.∼ p(x), then 1 p − log p (X , X , ... , X ) −! H(X ) (1) n 1 2 n Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 4 / 19 Asymptotic equipartition property theoremII Proof. Xi i.i.d =) log p(Xi ) i.i.d, by the weak law of large numbers 1 1 − log p (X1, X2, ... , Xn) = − ∑ log p(Xi ) n n i p −! −E (log p(X ) = H(X ). Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 5 / 19 Asymptotic equipartition property theorem III Definition 3 (n) The typical set A# with respect to p(x) is the set of sequences n (x1, x2, ... , xn) 2 X with the property −n(H(X )+#) −n(H(X )−#) 2 ≤ p(x1, x2, ... , xn) ≤ 2 . (2) Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 6 / 19 Asymptotic equipartition property theoremIV Theorem 4 (n) (1) If (x1, x2, ... , xn) 2 A# , then 1 H(X ) − # ≤ − log p(x , x , ... , x ) ≤ H(X ) + #. n 1 2 n (n) (2) P A# > 1 − # for n sufficiently large. (n) n(H(X )+#) (3) A# ≤ 2 , where jAj denotes the cardinal of A. (n) n(H(X )−#) (4) A# ≥ (1 − #) 2 for n sufficiently large. Thus, the typical set has probability nearly 1, all elements of the typical set are nearly equiprobable, and the number of elements in the typical set is nearly 2nH . Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 7 / 19 Asymptotic equipartition property theoremV Proof. (n) (1) from definition of A# . (n) (2) follows from Theorem 2, since P((X1, X2, ... , Xn) 2 A# ! 1 as n ! ¥. Thus, 8d > 09n0 such that for all n ≥ n0 1 P − log p (X1, X2, ... , Xn) − H(X ) < # > 1 − d. n Setting d = # complets the proof of (2). Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 8 / 19 Asymptotic equipartition property theoremVI Proof - continuation. (3) −n(H(X )+#) −n(H(X )+#) (n) 1 = ∑ p(x) ≥ ∑ p(x) ≥ ∑ 2 = 2 A# x2X n (n) (n) x2A# x2A# (n) n(H(X )+#) Hence A# ≤ 2 . (n) (4) For sufficiently large n, P A# > 1 − #, so that (n) −n(H(X )−#) −n(H(X )−#) (n) 1 − # < P A# ≤ ∑ 2 = 2 A# (n) x2A# which yields to (n) n(H(X )−#) A# ≥ (1 − #) 2 . Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 9 / 19 Consequences of the AEP: Data compressionI X1, X2, ... , Xn ∼ p(x), i.i.d We wish short descriptions of such sequences n (n) We divide sequences in X in two sets: the typical set A# and its complement (Figure 1) All elements in each set are ordered (e.g. lexicografic order) (n) We represent each sequence of A# by its index of the sequence in the set. n(H+#) (n) Since there are ≤ 2 sequences in A# , the indexing requires ≤ n(H + #) + 1 bits We prefix each sequence by 0: ≤ n(H + #) + 2 bits required (Figure 2) (n) Similarly, for the complement of A# we need n log jX j + 1; we prefix each sequence by 1 Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 10 / 19 Consequences of the AEP: Data compressionII Features of the coding scheme one-to-one and easy decodadle; the initial bit acts as a flag to indicate the length of the codeword (n) For complement of A# we have a brute-force enumeration, without C (n) n taking into account that the number of sequences in A# ≤ jX j Typical sequences have short descriptions of length≈ nH n n Notations x - a sequence x1, x2, ... , xn; `(x ) is the length of the codeword corresponding to xn Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 11 / 19 Consequences of the AEP: Data compression III Theorem 5 Let X n i.i.d ∼ p(x), and # > 0. The there exists a code that maps sequences xn of length n into binary strings such that the mappings is one-to-one (and therefore invertible) and 1 E ` (X n) ≤ H(X ) + # (3) n for n sufficiently large. We can represent sequences X n using nH(X ) bits on the average. Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 12 / 19 Consequences of the AEP: Data compressionIV Proof. (n) If n is sufficiently large so that P A# ≥ 1 − #, the expected length of codeword is E (` (X n)) = ∑ p(xn)`(xn) xn = ∑ p(xn)`(xn) + ∑ p(xn)`(xn) (n) C xn 2A n (n) # x 2 A# ≤ ∑ p(xn) (n (H + #) + 2) + ∑ p(xn) (n log jX j + 2) (n) C xn 2A n (n) # x 2 A# (n) (n)C = P A# (n (H + #) + 2) + P A# (n log jX j + 2) ≤ n (H + #) + #n log jX j + 2 = n H + #0 0 2 But, # = # + # log jX j + n can be made arbitrarily small. Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 13 / 19 Typical sets and source coding Figure: Typical sets and source coding Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 14 / 19 Source coding using the typical sets Figure: Source code using the typical set Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 15 / 19 High-probability sets and the typical setsI (n) It is not clear that A# is the smallest set that contains most of the probability. Definition 6 (n) n For each n = 1, 2, ... , let Bd ⊂ X with (n) P Bd ≥ 1 − d. (4) Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 16 / 19 High-probability sets and the typical setsII Theorem 7 1 0 Let X1, X2, ... , Xn be i.i.d. ∼ p(x). For d < 2 and any d > 0, if (n) P Bd > 1 − d, then 1 (n) log B > H − d0 (5) n d for n sufficiently large. Definition 8 . The notation an = bn means 1 a lim log n = 0. n!¥ n bn Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 17 / 19 High-probability sets and the typical sets III . Thus, an = bn implies an and bn are equal to the first order in the exponent. We can restate the above theorem: if dn ! 0 and #n ! 0, then n . n . B( ) = A( ) = 2nH . dn #n Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 18 / 19 References Thomas M. Cover, Joy A. Thomas, Elements of Information Theory, 2nd edition, Wiley, 2006. David J.C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003. Robert M. Gray, Entropy and Information Theory, Springer, 2009 Radu Tr^ımbit¸a¸s (UBB) Asymptotic Equipartition Property October 2012 19 / 19.