Kolmogorov Complexity

Home , Kolmogorov complexity, Random sequence

Pei Wang

Pei Wang

School of Mathematical Sciences Peking University [email protected]

May 15, 2012 Kolmogorov Outline Complexity Pei Wang Definition Models of Computation Definition Models of Computation Definitions Definitions Examples

Examples Kolmogorov Complexity and Kolmogorov Complexity and Entropy Entropy Preﬁx Code Kolmogorov Complexity and Preﬁx Code Entropy

Kolmogorov Complexity and Entropy Algorithmically Random and Incompressible Algorithmically Random and Incompressible Random and Incompressible Random and Incompressible Asymptotic Equipartition Property A Random Bernoulli Asymptotic Equipartition Property Sequence

A Random Bernoulli Sequence Ω

Kolmogorov Ω Complexity and Universal Kolmogorov Complexity and Universal Probability Probability Summary Summary References Kolmogorov A Turing Machine Complexity Pei Wang I a ﬁnite-state machine operating on a ﬁnite symbol

set, Definition Models of Computation I at each unit of time, the machine inspects the Definitions program tape, writes some symbols on a work tape, Examples Kolmogorov changes its state according to its transition table, and Complexity and Entropy calls for more program. Prefix Code Kolmogorov Complexity and I reads from left to right only. Entropy Algorithmically Random and Incompressible Random and Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Figure: A Turing Machine Summary

References

Most of the materials is copy-edited from[Cover et al., 1991]. Kolmogorov A Turing Machine Complexity Pei Wang I a ﬁnite-state machine operating on a ﬁnite symbol

Ω

Kolmogorov Complexity and Universal Probability

Figure: A Turing Machine Summary

References

Most of the materials is copy-edited from[Cover et al., 1991]. Kolmogorov A Turing Machine Complexity Pei Wang I a ﬁnite-state machine operating on a ﬁnite symbol

Ω

Kolmogorov Complexity and Universal Probability

Figure: A Turing Machine Summary

References

Most of the materials is copy-edited from[Cover et al., 1991]. Kolmogorov A Turing Machine Complexity Pei Wang

Definition I The programs this Turing Machine reads form a Models of Computation Definitions prefix-free set: Examples

I no program leading to a halting computation can be Kolmogorov Complexity and the prefix of another such program. Entropy Prefix Code Kolmogorov Complexity and I We can view the Turing machine as a map: Entropy Algorithmically I from a set of finite-length binary strings to the set of Random and finite- or infinite-length binary strings, Incompressible Random and Incompressible I The computation may halt or not. Asymptotic Equipartition Property A Random Bernoulli Sequence

Deﬁnition Ω ∗ ∗ ∞ The set of functions f : {0, 1} → {0, 1} ∪ {0, 1} Kolmogorov Complexity and computable by Turing machines is called the set of Universal partial recursive functions . Probability Summary

References Kolmogorov A Turing Machine Complexity Pei Wang

Definition I The programs this Turing Machine reads form a Models of Computation Definitions prefix-free set: Examples

References Kolmogorov A Turing Machine Complexity Pei Wang

Definition I The programs this Turing Machine reads form a Models of Computation Definitions prefix-free set: Examples

References Kolmogorov A Turing Machine Complexity Pei Wang

Definition I The programs this Turing Machine reads form a Models of Computation Definitions prefix-free set: Examples

References Kolmogorov A Turing Machine Complexity Pei Wang

Definition I The programs this Turing Machine reads form a Models of Computation Definitions prefix-free set: Examples

References Kolmogorov A Turing Machine Complexity Pei Wang

Definition I The programs this Turing Machine reads form a Models of Computation Definitions prefix-free set: Examples

References Kolmogorov Turing Machine Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions I Turing argued that this machine could mimic the Examples

computational ability of a human being. Kolmogorov Complexity and Entropy I After Turing’s work, it turned out that every new Preﬁx Code Kolmogorov Complexity and computational system could be reduced to a Turing Entropy Algorithmically machine, and conversely. Random and Incompressible Random and I In particular,the familiar digital computer with its Incompressible Asymptotic Equipartition CPU, memory, and input output devices could be Property A Random Bernoulli simulated by and could simulate a Turing machine. Sequence Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Turing Machine Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions I Turing argued that this machine could mimic the Examples

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Turing Machine Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions I Turing argued that this machine could mimic the Examples

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Turing’s Thesis Complexity Pei Wang

Definition Models of Computation Definitions I The class of algorithmically computable numerical Examples Kolmogorov functions (in the intuitive sense) coincides with the Complexity and Entropy class of partial recursive functions Prefix Code Kolmogorov Complexity and [Li and Vitányi, 2008]. Entropy Algorithmically Random and I All (sufficiently complex) computational models are Incompressible Random and equivalent in the sense that they can compute the Incompressible Asymptotic Equipartition same family of functions. Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Turing’s Thesis Complexity Pei Wang

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov The Universal Turing Machine Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and I Universal: Entropy Preﬁx Code I All but the most trivial computers are universal,in the Kolmogorov Complexity and sense that they can mimic the actions of other Entropy Algorithmically computers. Random and Incompressible Random and I The universal Turing machine: Incompressible Asymptotic Equipartition Property I the conceptually simplest universal computer. A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov The Universal Turing Machine Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov The Universal Turing Machine Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov The Universal Turing Machine Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Outline Complexity Pei Wang Definition Models of Computation Definition Models of Computation Definitions Definitions Examples

Examples Kolmogorov Complexity and Kolmogorov Complexity and Entropy Entropy Preﬁx Code Kolmogorov Complexity and Preﬁx Code Entropy

A Random Bernoulli Sequence Ω

Kolmogorov Ω Complexity and Universal Kolmogorov Complexity and Universal Probability Probability Summary Summary References Kolmogorov Kolmogorov Complexity of a String Complexity Pei Wang

Definition Models of Computation Definition Definitions Examples The Kolmogorov complexity KU (x) of a string x with Kolmogorov respect to a universal computer U is defined as: Complexity and Entropy Prefix Code Kolmogorov Complexity and KU (x) = min `(p), Entropy p:U(p)=x Algorithmically Random and Incompressible Random and Incompressible I x is a finite-length binary string, Asymptotic Equipartition Property A Random Bernoulli I U is a universal computer, Sequence I `(x) is the length of the string x, Ω Kolmogorov I U(p) is the output of the computer U when presented Complexity and Universal with a program p. Probability

Summary

References Kolmogorov Kolmogorov Complexity of a String Complexity Pei Wang

Summary

References Kolmogorov Kolmogorov Complexity of a String Complexity Pei Wang

Summary

References Kolmogorov Kolmogorov Complexity of a String Complexity Pei Wang

Summary

References Kolmogorov Kolmogorov Complexity of a String Complexity Pei Wang

Summary

References Kolmogorov Kolmogorov Complexity of a String Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Remark Kolmogorov Complexity and I A useful technique for thinking about Kolmogorov Entropy Preﬁx Code Kolmogorov Complexity and complexity is the following–if one person can Entropy

describe a sequence to another person in such a Algorithmically Random and manner as to lead unambiguously to a computation Incompressible Random and of that sequence in a ﬁnite amount of time. Incompressible Asymptotic Equipartition Property A Random Bernoulli I The number of bits in that communication is an upper Sequence bound on the Kolmogorov complexity. Ω Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Kolmogorov Complexity of a String Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Remark Kolmogorov Complexity and I A useful technique for thinking about Kolmogorov Entropy Preﬁx Code Kolmogorov Complexity and complexity is the following–if one person can Entropy

Summary

References Kolmogorov Kolmogorov Complexity of a String Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Remark Kolmogorov Complexity and I A useful technique for thinking about Kolmogorov Entropy Preﬁx Code Kolmogorov Complexity and complexity is the following–if one person can Entropy

Summary

References Kolmogorov Kolmogorov Complexity of a String Complexity Pei Wang

Definition Models of Computation Definitions Examples Example Kolmogorov Complexity and One can say"Print out the first 1,239,875,981,825,931 Entropy Prefix Code Kolmogorov Complexity and bits of the square root of e." We see that the Kolmogorov Entropy

complexity of this huge number is no greater than Algorithmically Random and (8)(73) = 584 bits. Incompressible Random and Incompressible Asymptotic Equipartition The fact that there is a simple algorithm to calculate the Property A Random Bernoulli square root of e provides the saving in descriptive Sequence complexity. Ω Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Conditional Kolmogorov Complexity Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Definition Entropy If we assume that the computer already knows the length Prefix Code Kolmogorov Complexity and of x, we can define the conditional Kolmogorov Entropy Algorithmically complexity knowing `(x) as: Random and Incompressible Random and Incompressible KU (x|`(x)) = min `(p), Asymptotic Equipartition p:U(p,`(x))=x Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universality of Kolmogorov Complexity Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy Theorem Preﬁx Code Kolmogorov Complexity and If U is a universal computer, for any other computer A Entropy there exists a constant cA such that Algorithmically Random and Incompressible Random and KU (x) ≤ KA(x) + cA. Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universality of Kolmogorov Complexity Complexity Pei Wang

Deﬁnition Models of Computation Proof. Assume p is a program for computer A to print x. Deﬁnitions A Examples

Thus, A(pA) = x. sA is a simulation program which tells Kolmogorov Complexity and computer U how to simulate computer A. The program Entropy for U to print x is p = s p and its length is Preﬁx Code A A Kolmogorov Complexity and Entropy

Algorithmically `(p) = `(sA) + `(pA) = cA + `(pA), Random and Incompressible Random and where cA is the length of the simulation program. Hence, Incompressible Asymptotic Equipartition Property A Random Bernoulli KU (x) = min `(p) ≤ min (`(p) + cA) = KA(x) + cA Sequence p:U(p)=x p:A(p)=x Ω

Kolmogorov for all strings x. Complexity and Universal Probability

Summary

References Kolmogorov Universality of Kolmogorov Complexity Complexity Pei Wang

Definition Models of Computation Definitions Remark Examples Kolmogorov Complexity and I The constant cA in the theorem may be very large. Entropy Prefix Code Kolmogorov Complexity and I The crucial point is that the length of this simulation Entropy program is independent of the length of x. Algorithmically Random and Incompressible Random and I For sufficiently long x, the length of this simulation Incompressible Asymptotic Equipartition program can be neglected. Property A Random Bernoulli Sequence

I We discuss Kolmogorov complexity without talking Ω about the constants. Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universality of Kolmogorov Complexity Complexity Pei Wang

I We discuss Kolmogorov complexity without talking Ω about the constants. Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universality of Kolmogorov Complexity Complexity Pei Wang

I We discuss Kolmogorov complexity without talking Ω about the constants. Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universality of Kolmogorov Complexity Complexity Pei Wang

I We discuss Kolmogorov complexity without talking Ω about the constants. Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universality of Kolmogorov Complexity Complexity Pei Wang

I We discuss Kolmogorov complexity without talking Ω about the constants. Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universality of Kolmogorov Complexity Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Remark Complexity and If A and U are both universal, we have Entropy Preﬁx Code Kolmogorov Complexity and |KU (x) − KA(x)| < c for all x. Hence, we will drop all Entropy

mention of U in all further definitions. We will assume that Algorithmically Random and the unspecified computer U is a fixed universal computer. Incompressible Random and Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Upper Bound on Conditional Complexity Complexity Pei Wang

Definition Models of Computation Theorem Definitions (Conditional complexity is less than the length of the Examples Kolmogorov sequence) Complexity and Entropy KU (x|`(x)) ≤ `(x) + c. Prefix Code Kolmogorov Complexity and Entropy

Algorithmically Random and Proof. A program for printing x is Incompressible Random and Incompressible Print the following `-bit sequence: x1x2 ... x`(x). Asymptotic Equipartition Property A Random Bernoulli Note that no bits are required to describe ` since ` is given. Sequence Ω The program is self-delimiting because `(x) is provided and the Kolmogorov end of the program is thus clearly deﬁned. The length of this Complexity and Universal program is `(x) + c. Probability Summary

References Kolmogorov Upper Bound on Conditional Complexity Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy Remark Preﬁx Code Kolmogorov Complexity and Without knowledge of the length of the string, we will Entropy Algorithmically need an additional stop symbol or we can use a Random and Incompressible self-punctuating scheme like the one described in the Random and Incompressible proof of the next theorem. Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Upper Bound on Kolmogorov Complexity Complexity Pei Wang

Definition Models of Computation Definitions Theorem Examples Kolmogorov Complexity and KU (x) ≤ KU (x|`(x)) + 2 log `(x) + c. Entropy Prefix Code Kolmogorov Complexity and Entropy

Algorithmically Proof. Random and Incompressible Random and I If the computer does not know `(x),we must have some Incompressible Asymptotic Equipartition way of informing the computer when it has come to the Property A Random Bernoulli end of the string of bits that describes the sequence. Sequence

Ω I We describe a simple but inefﬁcient method that uses a Kolmogorov sequence 01 as a "comma." Complexity and Universal Probability

Summary

References Kolmogorov Upper Bound on Kolmogorov Complexity Complexity Pei Wang

Definition Models of Computation Definitions Theorem Examples Kolmogorov Complexity and KU (x) ≤ KU (x|`(x)) + 2 log `(x) + c. Entropy Prefix Code Kolmogorov Complexity and Entropy

Ω I We describe a simple but inefﬁcient method that uses a Kolmogorov sequence 01 as a "comma." Complexity and Universal Probability

Summary

References Kolmogorov Upper Bound on Kolmogorov Complexity Complexity Pei Wang

Definition Models of Computation Definitions Theorem Examples Kolmogorov Complexity and KU (x) ≤ KU (x|`(x)) + 2 log `(x) + c. Entropy Prefix Code Kolmogorov Complexity and Entropy

Ω I We describe a simple but inefﬁcient method that uses a Kolmogorov sequence 01 as a "comma." Complexity and Universal Probability

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Definition Models of Computation Definitions I Suppose that `(x) = n. To describe `(x), repeat every bit Examples of the binary expansion of n twice; then end the Kolmogorov Complexity and description with a 01 so that the computer knows that it Entropy Prefix Code has come to the end of the description of n. Kolmogorov Complexity and Entropy

Algorithmically I For example, the number 5 (binary 101) will be described Random and as 11001101. This description requires 2dlog ne + 2 bits. Incompressible Random and Incompressible Asymptotic Equipartition I Thus, inclusion of the binary representation of `(x) does Property A Random Bernoulli not add more than 2dlog ne + 2 bits to the length of the Sequence program, and we have the bound in the theorem. Ω Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Summary

References Kolmogorov Lower Bound on Kolmogorov Complexity Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy Remark Preﬁx Code Kolmogorov Complexity and A more efﬁcient method for describing n is to do so Entropy recursively. The theorem above can be improved to Algorithmically Random and Incompressible ∗ Random and KU (x) ≤ KU (x|`(x)) + log `(x) + c. Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Lower Bound on Kolmogorov Complexity Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy Preﬁx Code Theorem Kolmogorov Complexity and The number of strings x with complexity K (x) < k Entropy Algorithmically satisﬁes Random and ∗ k Incompressible |{x ∈ {0, 1} : K (x) < k}| < 2 . (1) Random and Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Lower Bound on Kolmogorov Complexity Complexity Pei Wang

Definition Models of Computation Proof. We list all the programs of length < k, we have Definitions Examples k−1 Kolmogorov Complexity and z }| { Entropy Λ, 0, 1, 00, 01, 10, 11, ...,..., 11 ... 1 Prefix Code |{z} |{z} | {z } | {z } Kolmogorov Complexity and 1 2 4 2k−1 Entropy Algorithmically Random and and the total number of such programs is Incompressible Random and Incompressible k−1 k k Asymptotic Equipartition 1 + 2 + 4 + ... + 2 = 2 − 1 ≤ 2 . Property A Random Bernoulli Sequence Since each program can produce only one possible Ω output sequence, the number of sequences with Kolmogorov k Complexity and complexity < k is less than 2 . Universal Probability

Summary

References Kolmogorov Outline Complexity Pei Wang Definition Models of Computation Definition Models of Computation Definitions Definitions Examples

Examples Kolmogorov Complexity and Kolmogorov Complexity and Entropy Entropy Preﬁx Code Kolmogorov Complexity and Preﬁx Code Entropy

A Random Bernoulli Sequence Ω

Kolmogorov Ω Complexity and Universal Kolmogorov Complexity and Universal Probability Probability Summary Summary References Kolmogorov Complexity

Pei Wang

Deﬁnition Models of Computation Deﬁnitions I The Kolmogorov complexity will depend on the Examples

computer, but only up to an additive constant. Kolmogorov Complexity and Entropy I We consider a computer that can accept Preﬁx Code Kolmogorov Complexity and unambiguous commands in English (with numbers Entropy Algorithmically given in binary notation). Random and Incompressible Random and Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Complexity

Pei Wang

Deﬁnition Models of Computation Deﬁnitions I The Kolmogorov complexity will depend on the Examples

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Sequences of Zeros Complexity Pei Wang

Definition Models of Computation Definitions Example Examples Kolmogorov (A sequence of n zeros) If we assume that the computer Complexity and Entropy knows n, a short program to print this string is Prefix Code Kolmogorov Complexity and Entropy

Print the speciﬁed number of zeros. Algorithmically Random and Incompressible The length of this program is a constant number of bits. Random and Incompressible This program length does not depend on n. Hence, the Asymptotic Equipartition Property Kolmogorov complexity of this sequence is c, and A Random Bernoulli Sequence

Ω ( ... | ) = . K 000 0 n c Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Integer Complexity Pei Wang

Definition Models of Computation Example Definitions Examples (Integer n) If the computer knows the number of bits in Kolmogorov Complexity and the binary representation of the integer, we need only Entropy Prefix Code provide the values of these bits. This program will have Kolmogorov Complexity and Entropy length (c + log n). Algorithmically Random and In general, the computer will not know the length of the Incompressible Random and binary representation of the integer. By informing the Incompressible Asymptotic Equipartition Property computer in a recursive way when the description ends, A Random Bernoulli we see that the Kolmogorov complexity of an integer is Sequence Ω bounded by ∗ Kolmogorov K (n) ≤ log n + c. Complexity and Universal Probability

Summary

References Kolmogorov A Notation Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov We introduce a notation for the binary entropy function Complexity and Entropy Preﬁx Code Kolmogorov Complexity and H0(p) = −(1 − p) log(1 − p) − p log p. Entropy

Algorithmically H ( 1 Pn x ) Random and Thus, when we write 0 n i=1 i , we will mean Incompressible ¯ ¯ ¯ ¯ Random and −Xn log Xn − (1 − Xn) log(1 − Xn), and not the entropy of Incompressible ¯ Asymptotic Equipartition random variable Xn. When there is no confusion, we shall Property A Random Bernoulli simply write H(p) for H0(p). Sequence Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov An Inequality Complexity Lemma Pei Wang For k 6= 0, n, we have Definition r Models of Computation n n Definitions ≤ 2nH(k/n). (2) k πk(n − k) Examples Kolmogorov Complexity and Proof. Applying a strong form of Stirling’s approximation Entropy Prefix Code [Feller, 2008], which states that Kolmogorov Complexity and Entropy

√ nn √ nn 1 Algorithmically 2πn ≤ n! ≤ 2πn e 12n (3) Random and e e Incompressible Random and We obtain Incompressible Asymptotic Equipartition Property √ n 1 n 12n A Random Bernoulli n 2πn e e Sequence ≤ √ k k k p n−k (n−k) Ω 2πk e 2π(n − k) e r −k −(n−k) Kolmogorov n k n − k 1 Complexity and 12n ≤ e Universal πk(n − k) n n Probability r n ≤ 2nH(k/n), Summary πk(n − k) References

1 1 √ since e 12n < e 12 = 1.087 < 2. Kolmogorov Sequence Compress Complexity Pei Wang

Definition Models of Computation Definitions Example Examples Kolmogorov (Sequence of n bits with k ones) Can we compress a Complexity and Entropy sequence of n bits with k ones? Prefix Code Kolmogorov Complexity and Entropy

Consider the following program: Algorithmically Random and Incompressible Generate, in lexicographic order, all sequences with k ones; Random and Incompressible Of these sequences, print the ith sequence. Asymptotic Equipartition Property A Random Bernoulli This program will print out the required sequence. The Sequence only variables in the program are k ( with known range {0, Ω n Kolmogorov 1,. . . , n} )and i ( with conditional range {1, 2,. . . , k } ). Complexity and Universal Probability

Summary

References Kolmogorov Sequence Compress Complexity Pei Wang Example Definition The total length of this program is Models of Computation Definitions Examples n Kolmogorov `(p) = c + log n + log (4) Complexity and | {z } k Entropy to express k Prefix Code | {z } Kolmogorov Complexity and to express i Entropy Algorithmically 0 k 1 Random and ≤ c + log n + nH − log n, (5) Incompressible n 2 Random and Incompressible Asymptotic Equipartition n q n nH(k/n) k Property Since k ≤ πk(n−k) 2 by (2) for p = and A Random Bernoulli n Sequence

q = 1 − p and k 6= 0 and k 6= n. We have used log n bits Ω Pn to represent k. Thus, if i=1 xi = k,then Kolmogorov Complexity and Universal k 1 Probability K (x , x ,..., x |n) ≤ nH + log n + c. (6) 1 2 n n 2 Summary References Kolmogorov Sequence Compress Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Theorem Entropy The Kolmogorov complexity of a binary string x is Preﬁx Code Kolmogorov Complexity and bounded by Entropy Algorithmically Random and n ! Incompressible 1 X 1 Random and K (x1, x2,..., xn|n) ≤ nH xi + log n + c. (7) Incompressible n 2 Asymptotic Equipartition i=1 Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Sequence Compress Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Remark Kolmogorov Complexity and I In general, when the length `(x) of the sequence x is Entropy Preﬁx Code Kolmogorov Complexity and small, the constants that appear in the expressions Entropy

for the Kolmogorov complexity will overwhelm the Algorithmically Random and contributions due to `(x). Incompressible Random and Incompressible Asymptotic Equipartition I Hence, the theory is useful primarily when `(x) is Property A Random Bernoulli very large. In such cases we can safely neglect the Sequence terms that do not depend on `(x). Ω Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Sequence Compress Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Remark Kolmogorov Complexity and I In general, when the length `(x) of the sequence x is Entropy Preﬁx Code Kolmogorov Complexity and small, the constants that appear in the expressions Entropy

Summary

References Kolmogorov Sequence Compress Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Remark Kolmogorov Complexity and I In general, when the length `(x) of the sequence x is Entropy Preﬁx Code Kolmogorov Complexity and small, the constants that appear in the expressions Entropy

Summary

References Kolmogorov Outline Complexity Pei Wang Definition Models of Computation Definition Models of Computation Definitions Definitions Examples

Examples Kolmogorov Complexity and Kolmogorov Complexity and Entropy Entropy Preﬁx Code Kolmogorov Complexity and Preﬁx Code Entropy

A Random Bernoulli Sequence Ω

Kolmogorov Ω Complexity and Universal Kolmogorov Complexity and Universal Probability Probability Summary Summary References Kolmogorov Preﬁx Code Complexity Pei Wang

Definition Definition Models of Computation Definitions A code is called a prefix code or an instantaneous Examples code if no codeword is a prefix of any other codeword Kolmogorov Complexity and Entropy Theorem Prefix Code Kolmogorov Complexity and (Kraft inequality) For any instantaneous code (prefix Entropy Algorithmically code) over an alphabet of size D, the codeword lengths Random and Incompressible `1, `2, . . . , `m must satisfy the inequality Random and Incompressible Asymptotic Equipartition X −ì Property D ≤ 1. (8) A Random Bernoulli Sequence i Ω

Kolmogorov Conversely, given a set of codeword lengths that satisfy Complexity and Universal this inequality,there exists an instantaneous code with Probability

these word lengths. Summary

References Kolmogorov Preﬁx Code Complexity Pei Wang

Kolmogorov Conversely, given a set of codeword lengths that satisfy Complexity and Universal this inequality,there exists an instantaneous code with Probability

these word lengths. Summary

References Kolmogorov Kraft Inequality Complexity Proof. Consider a D-ary tree. Pei Wang I Let the branches of the tree represent the symbols of the Deﬁnition codeword. Models of Computation Deﬁnitions I Then each codeword is represented by a leaf on the tree. The Examples

path from the root traces out the symbols of the codeword. Kolmogorov Complexity and I A binary example of such a tree is shown in Figure (2). Entropy Preﬁx Code I The preﬁx condition on the codewords implies that each Kolmogorov Complexity and codeword eliminates its descendants as possible codewords. Entropy Algorithmically Random and Incompressible Random and Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

Figure: Code tree for the Kraft inequality References Kolmogorov Kraft Inequality Complexity Proof. Consider a D-ary tree. Pei Wang I Let the branches of the tree represent the symbols of the Deﬁnition codeword. Models of Computation Deﬁnitions I Then each codeword is represented by a leaf on the tree. The Examples

Ω

Kolmogorov Complexity and Universal Probability

Summary

Ω

Kolmogorov Complexity and Universal Probability

Summary

Ω

Kolmogorov Complexity and Universal Probability

Summary

Figure: Code tree for the Kraft inequality References Kolmogorov Proof Cont. Complexity Pei Wang

Definition Models of Computation Definitions Examples I Consider all nodes of the tree at level `max . Some of them are Kolmogorov codewords, some are descendants of codewords, and some are Complexity and neither. Entropy Prefix Code `max −ì I A codeword at level ì has D descendants at level `max . Kolmogorov Complexity and Entropy Each of these descendant sets must be disjoint. Algorithmically Also, the total number of nodes in these sets must be less than Random and I Incompressible `max or equal to D . Random and Incompressible Asymptotic Equipartition Hence, Property X ì −lmax `max A Random Bernoulli D ≤ D (9) Sequence Thus, Ω X ` D i ≤ 1. (10) Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov I Conversely, given any set of codeword lengths `1, `2, . . . , `m that Complexity and Entropy satisfy the Kraft inequality,we can always construct a tree like the Preﬁx Code one in Figure (2) Kolmogorov Complexity and Entropy

I Label the first node (lexicographically) of depth `1 as codeword Algorithmically Random and 1, and remove its descendants from the tree. Incompressible Random and I Then label the first remaining node of depth `2 as codeword 2, Incompressible Asymptotic Equipartition and so on. Property A Random Bernoulli I Proceeding this way, we construct a prefix code with the Sequence specified `1, `2, . . . , `m. Ω Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Summary

References Kolmogorov Extended Kraft Inequality Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Theorem Kolmogorov For any countably infinite set of codewords that form a Complexity and Entropy prefix code, the codeword lengths satisfy the extended Prefix Code Kolmogorov Complexity and Kraft inequality, Entropy ∞ Algorithmically Random and X −ì D ≤ 1. (11) Incompressible Random and i Incompressible Asymptotic Equipartition Property Conversely, given any `1, `2,... satisfying the extended A Random Bernoulli Kraft inequality, we can construct a prefix code with these Sequence Ω

codeword lengths. Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Extended Kraft Inequality Complexity Pei Wang , ,..., − Proof. Let the D-ary alphabet be 0 1 D 1. Consider the ith Definition codeword y1y2 ... yì . Let 0.y1y2 ... yì be the real number given by the Models of Computation Definitions D-ary expansion Examples ì X −j Kolmogorov 0.y1y2 ... yì = yj D . (12) Complexity and j=1 Entropy Prefix Code This codeword corresponds to the interval Kolmogorov Complexity and Entropy 1 Algorithmically 0.y1y2 ... y` , 0.y1y2 ... y` + Random and i i D−ì Incompressible Random and Incompressible the set of all real numbers whose D-ary expansion begins with Asymptotic Equipartition Property 0.y1y2 ... yì . This is a subinterval of the unit interval [0, 1]. By the A Random Bernoulli prefix condition, these intervals are disjoint. Hence, the sum of their Sequence lengths has to be less than or equal to 1. This proves that Ω Kolmogorov ∞ Complexity and X −` D i ≤ 1. (13) Universal Probability i Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Just as in the finite case, we can reverse the proof to construct Complexity and I Entropy the code for a given `1, `2,... that satisfies the Kraft inequality. Prefix Code Kolmogorov Complexity and Entropy

I First, reorder the indexing so that `1 ≤ `2 ≤ ... Then simply Algorithmically assign the intervals in order from the low end of the unit interval. Random and Incompressible Random and I For example, if we wish to construct a binary code with Incompressible 1 1 1 Asymptotic Equipartition `1 = 1, `2 = 2,..., we assign the intervals [0, ), [ , ), ... to the Property 2 2 4 A Random Bernoulli symbols, with corresponding codewords 0, 10,... Sequence Ω Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Summary

References Kolmogorov Expected Code Length Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Theorem Entropy Preﬁx Code The expected length L of any instantaneous D-ary code Kolmogorov Complexity and Entropy for a random variable X is greater than or equal to the Algorithmically entropy H (X); that is, Random and D Incompressible Random and Incompressible L ≥ H (X). (14) Asymptotic Equipartition D Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Expected Code Length Complexity Pei Wang

Proof. We can write the difference between the expected length and Definition the entropy as Models of Computation Definitions X X 1 Examples L − HD(X) = pi ì − pi logD (15) Kolmogorov pi Complexity and X −ì X Entropy = − pi logD D + pi logD pi . (16) Prefix Code Kolmogorov Complexity and Entropy −ì P −`j P −`j Letting ri = D / j D and c = j D ,we obtain Algorithmically Random and X pi Incompressible L − H = − pi logD − logD c (17) Random and ri Incompressible Asymptotic Equipartition 1 Property = D(pkr) + logD (18) A Random Bernoulli c Sequence ≥ 0 (19) Ω Kolmogorov by the nonnegativity of relative entropy and the fact (Kraft inequality) Complexity and Universal −ì Probability that c ≤ 1. Hence, L ≤ H with equality iff pi = D (i.e., iff − logD pi is an integer for all i). Summary References Kolmogorov Outline Complexity Pei Wang Definition Models of Computation Definition Models of Computation Definitions Definitions Examples

Examples Kolmogorov Complexity and Kolmogorov Complexity and Entropy Entropy Preﬁx Code Kolmogorov Complexity and Preﬁx Code Entropy

A Random Bernoulli Sequence Ω

Kolmogorov Ω Complexity and Universal Kolmogorov Complexity and Universal Probability Probability Summary Summary References Kolmogorov Kraft Inequality Complexity Pei Wang

Definition Models of Computation Definitions Lemma Examples Kolmogorov For any computer U, Complexity and Entropy X −`(p) Prefix Code 2 ≤ 1. (20) Kolmogorov Complexity and Entropy

p:U(p)halts Algorithmically Random and Incompressible Random and Proof. If the computer halts on any program, it does not Incompressible Asymptotic Equipartition Property look any further for input. Hence, there cannot be any A Random Bernoulli other halting program with this program as a preﬁx. Thus, Sequence Ω the halting programs form a preﬁx-free set, and their Kolmogorov lengths satisfy the Kraft inequality. Complexity and Universal Probability

Summary

References Kolmogorov Kolmogorov Complexity and Entropy Complexity Pei Wang

Definition Models of Computation Theorem Definitions Examples Let the stochastic process {Xi } be drawn i.i.d. according Kolmogorov to the probability mass function f (x), x ∈ X , where X is a Complexity and n Qn Entropy finite alphabet. Let f (x ) = i=1 f (xi ). Then there exists Prefix Code Kolmogorov Complexity and a constant c such that Entropy Algorithmically 1 X (|X | − 1) log n c Random and H(X) ≤ f (xn)K (xn|n) ≤ H(X) + + Incompressible n n n Random and xn Incompressible Asymptotic Equipartition (21) Property A Random Bernoulli for all n. Thus Sequence Ω 1 n Kolmogorov E K (x |n) → H(X). (22) Complexity and n Universal Probability

Summary

References Kolmogorov Kolmogorov Complexity and Entropy Complexity Pei Wang Proof. Definition I Consider the lower bound. The allowed programs satisfy Models of Computation Definitions the prefix property, and thus their lengths satisfy the Kraft Examples

inequality. Kolmogorov Complexity and n I We assign to each x the length of the shortest program p Entropy n Preﬁx Code such that U(p, n) = x .These shortest programs also Kolmogorov Complexity and Entropy satisfy the Kraft inequality. Algorithmically Random and I We know from the theory of source coding (9) that the Incompressible Random and expected codeword length must be greater than the Incompressible Asymptotic Equipartition entropy. Property A Random Bernoulli Hence, Sequence Ω X ( n) ( n| ) ≥ ( , ,..., ) = ( ). Kolmogorov f x K x n H X1 X2 Xn nH x (23) Complexity and x n Universal Probability We ﬁrst prove the upper bound when X is binary Summary

(i.e.,X1, X2,..., Xn are i.i.d. Bernoulli(θ)). References Kolmogorov Kolmogorov Complexity and Entropy Complexity Pei Wang Proof. Definition I Consider the lower bound. The allowed programs satisfy Models of Computation Definitions the prefix property, and thus their lengths satisfy the Kraft Examples

(i.e.,X1, X2,..., Xn are i.i.d. Bernoulli(θ)). References Kolmogorov Proof Cont. Complexity Pei Wang Using result (7), we can bound the complexity of a binary string by Definition Models of Computation Definitions n ! 1 X 1 Examples K (x1, x2,..., xn|n) ≤ nH0 xi + log n + c. (24) Kolmogorov n 2 Complexity and i=1 Entropy Prefix Code Kolmogorov Complexity and Entropy n ! 1 X 1 Algorithmically EK (X , X ,..., X |n) ≤ nEH X + log n + c (25) 1 2 n 0 n i 2 Random and i=1 Incompressible Random and Incompressible (a) n ! 1 X 1 Asymptotic Equipartition ≤ nH0 EXi + log n + c (26) Property n 2 A Random Bernoulli i=1 Sequence 1 Ω = nH0 (θ) + log n + c, (27) Kolmogorov 2 Complexity and Universal where (a) follows from Jensen’s inequality and the concavity of Probability the entropy. Thus, we have proved the upper bound in the Summary theorem for binary processes. References Kolmogorov Proof Cont. Complexity Pei Wang

Definition Models of Computation Definitions I We can use the same technique for the case of a Examples Kolmogorov nonbinary finite alphabet. Complexity and Entropy Prefix Code I We first describe the type of the sequence (the empirical Kolmogorov Complexity and Entropy frequency of occurrence of each of the alphabet symbols Algorithmically as defined later) using (|X | − 1) log n bits (the frequency Random and Incompressible of the last symbol can be calculated from the frequencies Random and Incompressible of the rest). Asymptotic Equipartition Property A Random Bernoulli Sequence I Then we describe the index of the sequence within the set of all sequences having the same type. The type class Ω nH(Pxn ) Kolmogorov has less than 2 elements (where Px n is the type of Complexity and the sequence x n) as shown later. Universal Probability

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Definition Models of Computation Definitions Examples n Therefore the two-stage description of a string x has length Kolmogorov Complexity and Entropy K (xn|n) ≤ nH (Px n )) + (|X − 1|) log n + c (28) Prefix Code Kolmogorov Complexity and Entropy

Again, taking the expectation and applying Jensen’s inequality Algorithmically Random and as in the binary case, we obtain Incompressible Random and Incompressible EK (Xn|n) ≤ nH (X) + (|X − 1|) log n + c (29) Asymptotic Equipartition Property A Random Bernoulli Sequence Dividing this by n yields the upper bound of the theorem. Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Type of Sequence Complexity Pei Wang

Definition Let X1, X2,..., Xn be a sequence of n symbols from an Models of Computation n Definitions alphabet X = {a1, a2,..., a|X |}. We use the notation x Examples and x interchangeably to denote a sequence Kolmogorov Complexity and x1, x2,..., xn. Entropy Prefix Code Kolmogorov Complexity and Entropy

Algorithmically Random and Deﬁnition Incompressible Random and The type P (or empirical probability distribution) of a Incompressible x Asymptotic Equipartition Property sequence x1, x2,..., xn is the relative proportion of A Random Bernoulli Sequence occurrences of each symbol of X (i.e., Px(a) = N(a|x)/n Ω for all a ∈ X , where N(a|x) is the number of times the Kolmogorov symbol a occurs in the sequence x ∈ X n).It is a probability Complexity and Universal mass function on X . Probability Summary

References Kolmogorov Type of Sequence Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Deﬁnition Complexity and Entropy Let Pn denote the set of types with denominator n. Preﬁx Code Kolmogorov Complexity and For example, if X = {0, 1}, the set of possible types with Entropy Algorithmically denominator n is Random and Incompressible Random and 0 n 1 n − 1 n 0 Incompressible Asymptotic Equipartition Pn = (P(0), P(1)) : , , , , ,..., , . Property n n n n n n A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Type Class Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Deﬁnition Complexity and Entropy If P ∈ Pn, the set of sequences of length n and type P is Preﬁx Code Kolmogorov Complexity and called the type class of P, denoted T (P): Entropy Algorithmically n Random and T (P) = {x ∈ X : Px = P} . Incompressible Random and Incompressible Asymptotic Equipartition Property The type class is sometimes called the composition class A Random Bernoulli Sequence

of P. Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Type Class Complexity Pei Wang

Deﬁnition Example Models of Computation Deﬁnitions Let X = {1, 2, 3}, a ternary alphabet. Let x = 11321.Then the Examples

type Px is: Kolmogorov Complexity and 3 1 1 Entropy P (1) = , P (2) = , P (3) = . Preﬁx Code x x x Kolmogorov Complexity and 5 5 5 Entropy

Algorithmically The type class of Px is the set of all sequences of length 5 with Random and three 1’s, one 2, and one 3. There are 20 such sequences, and Incompressible Random and Incompressible Asymptotic Equipartition T (Px) = {11123, 11132, 11213,..., 32111}. Property A Random Bernoulli Sequence

The number of elements in T (P)is Ω

Kolmogorov 5 5! Complexity and |T (P)| = = = 20. Universal 3, 1, 1 3!1!1! Probability Summary

References Kolmogorov Probability to Same Type Class Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy Theorem Preﬁx Code Kolmogorov Complexity and If X1, X2, ..., Xn are drawn i.i.d. according to Q(x), the Entropy probability of x depends only on its type and is given by Algorithmically Random and Incompressible n −n(H(Px)+D(PxkQ)) Random and Q (x) = 2 . Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Probability to Same Type Class Complexity Proof. Pei Wang

n Definition n Y Models of Computation Q (x) = Q(xi ) Definitions Examples i=1 Kolmogorov Y N(a|x) Complexity and = Q(a) Entropy Prefix Code a∈X Kolmogorov Complexity and Entropy Y nPx = Q(a) Algorithmically Random and a∈X Incompressible Y nP log Q(a) Random and = 2 x Incompressible Asymptotic Equipartition a∈X Property A Random Bernoulli Sequence Y n(Px log Q(a)−Px log Px+Px log Px) = 2 Ω

a∈X Kolmogorov P Px Complexity and n a∈X (−Px log Q(a) +Px log Px) Universal = 2 Probability = 2−n(H(Px)+D(PxkQ)). Summary References Kolmogorov Probability to Same Type Class Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy Corollary Preﬁx Code Kolmogorov Complexity and If x is in the type class of Q, then Entropy Algorithmically Random and n −nH(Q) Incompressible Q (x) = 2 . Random and Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Size of a Type Class Complexity Pei Wang Theorem Definition For any type P ∈ Pn, Models of Computation Definitions nH(P) Examples |T (P)| ≤ 2 . Kolmogorov Complexity and Entropy Prefix Code Proof. Since a type class must have probability ≤ 1, we Kolmogorov Complexity and Entropy have Algorithmically Random and n Incompressible 1 ≥ P (T (P)) Random and Incompressible X n Asymptotic Equipartition = P (x) Property A Random Bernoulli x∈T (P) Sequence Ω X −nH(P) = 2 Kolmogorov Complexity and x∈T (P) Universal Probability = | ( )| −nH(P), T P 2 Summary

References nH(P) Thus, |T (P)| ≤ 2 . Kolmogorov Outline Complexity Pei Wang Definition Models of Computation Definition Models of Computation Definitions Definitions Examples

Examples Kolmogorov Complexity and Kolmogorov Complexity and Entropy Entropy Preﬁx Code Kolmogorov Complexity and Preﬁx Code Entropy

A Random Bernoulli Sequence Ω

Kolmogorov Ω Complexity and Universal Kolmogorov Complexity and Universal Probability Probability Summary Summary References Kolmogorov Compression Complexity Pei Wang

Definition Models of Computation Definitions Examples I We now show that although there are some simple Kolmogorov Complexity and sequences, most sequences do not have simple Entropy descriptions. Prefix Code Kolmogorov Complexity and Entropy

I Hence, if we draw a sequence at random, we are likely to Algorithmically Random and draw a complex sequence. Incompressible Random and Incompressible Asymptotic Equipartition I The next theorem shows that the probability that a Property A Random Bernoulli sequence can be compressed by more than k bits is no Sequence −k greater than 2 . Ω Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Compression Complexity Pei Wang

Summary

References Kolmogorov Compression Complexity Pei Wang

Summary

References Kolmogorov Compression Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy Theorem Preﬁx Code 1 Kolmogorov Complexity and Let X1, X2, ..., Xn be drawn according to a Bernoulli 2 Entropy process. Then Algorithmically Random and Incompressible −k Random and P(K (X1, X2,..., Xn|n) < n − k) < 2 . (30) Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Compression Complexity Pei Wang

Definition Proof. Models of Computation Definitions Examples P(K (X1, X2,..., Xn|n) < n − k) Kolmogorov Complexity and X Entropy = p(x1, x2,..., xn) Prefix Code Kolmogorov Complexity and (x1,x2,...,xn:K (x1,x2,...,xn|n)

References Kolmogorov Compression Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy Preﬁx Code Remark Kolmogorov Complexity and Entropy

Thus, most sequences have a complexity close to their Algorithmically length. For example, the fraction of sequences of length n Random and Incompressible Random and that have complexity less than n − 5 is less than 1/32. Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Algorithmically Random Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Deﬁnition Kolmogorov Complexity and A sequence x1, x2,..., xn is said to be Entropy algorithmically random Preﬁx Code if Kolmogorov Complexity and Entropy

Algorithmically K (x1, x2,..., xn|n) ≥ n. (31) Random and Incompressible Random and Incompressible Asymptotic Equipartition Remark Property A Random Bernoulli Note that by the counting argument, there exists, for each Sequence n, at least one sequence xn such that K (xn|n) ≥ n. Ω Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Algorithmically Random Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Deﬁnition Kolmogorov Complexity and A sequence x1, x2,..., xn is said to be Entropy algorithmically random Preﬁx Code if Kolmogorov Complexity and Entropy

Summary

References Kolmogorov Incompressible String Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy Definition Prefix Code Kolmogorov Complexity and We call an infinite string x incompressible if Entropy Algorithmically Random and K (x1, x2, x3,..., xn|n) Incompressible lim = 1. (32) Random and n→∞ n Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Incompressible String Complexity Pei Wang

Definition Models of Computation Definitions Theorem Examples Kolmogorov (Strong law of large numbers for incompressible Complexity and Entropy sequences) If a string x1, x2,... is incompressible, it Prefix Code Kolmogorov Complexity and satisfies the law of large numbers in the sense that Entropy Algorithmically n Random and 1 1 Incompressible X Random and xi → . (33) Incompressible n 2 Asymptotic Equipartition i=1 Property A Random Bernoulli Sequence

Hence the proportions of 0’s and 1’s in any Ω

incompressible string are almost equal. Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Incompressible String Complexity Pei Wang

Definition Models of Computation Proof. Definitions Examples 1 Pn I Letθn = i=1 xi denote the proportion of 1’s in Kolmogorov n Complexity and x1, x2,..., xn. Entropy Prefix Code Kolmogorov Complexity and I Then using the method of Example in section 1 one Entropy can write a program of length nH(θn) + 2log(nθn) + c Algorithmically n Random and to print x . Incompressible Random and I Thus, Incompressible Asymptotic Equipartition Property n 0 A Random Bernoulli K (x |n) log n c Sequence < H (θ ) + 2 + . (34) n 0 n n n Ω Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Incompressible String Complexity Pei Wang

Summary

References Kolmogorov Incompressible String Complexity Pei Wang

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

By the incompressibility assumption, we also have the Definition Models of Computation lower bound for large enough n, Definitions Examples K (xn|n) log n c0 Kolmogorov 1 − ε ≤ < H (θ ) + 2 + . (35) Complexity and n 0 n n n Entropy Prefix Code Kolmogorov Complexity and Thus, Entropy 0 Algorithmically log n + c Random and H0(θn) > 1 − 2 − . (36) Incompressible n Random and Incompressible Inspection of the graph of H0(p) (Figure (3)) shows that Asymptotic Equipartition 1 Property θ is close to for large n. Specifically, the inequality A Random Bernoulli n 2 Sequence above implies that Ω

Kolmogorov 1 1 Complexity and θn ∈ − δn, + δn , (37) Universal 2 2 Probability Summary

References Kolmogorov Proof Cont. Complexity Pei Wang where δn is chosen so that

0 Definition 1 log n + cn + c Models of Computation H0( − δn) = 1 − 2 . (38) Definitions 2 n Examples Kolmogorov which implies that δn → 0 as n → ∞. Thus, Complexity and Entropy 1 Pn 1 θn = n i=1 xi → 2 as n → ∞. Prefix Code Kolmogorov Complexity and Entropy

Algorithmically Random and Incompressible Random and Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary Figure: H0(P) vs. p References Kolmogorov Incompressible String Complexity Pei Wang

Deﬁnition Models of Computation Remark Deﬁnitions Examples

I We have now proved that incompressible sequences look Kolmogorov Complexity and random in the sense that the proportion of 0’s and 1’s are Entropy almost equal. Preﬁx Code Kolmogorov Complexity and Entropy I In general, we can show that if a sequence is Algorithmically incompressible, it will satisfy all computable statistical Random and Incompressible tests for randomness. Random and Incompressible Asymptotic Equipartition I Otherwise, identiﬁcation of the test that x fails will reduce Property A Random Bernoulli the descriptive complexity of x, yielding a contradiction. Sequence Ω I In this sense, the algorithmic test for randomness is the Kolmogorov ultimate test, including within it all other computable tests Complexity and Universal for randomness. Probability

Summary

References Kolmogorov Incompressible String Complexity Pei Wang

Deﬁnition Models of Computation Remark Deﬁnitions Examples

Summary

References Kolmogorov Incompressible String Complexity Pei Wang

Deﬁnition Models of Computation Remark Deﬁnitions Examples

Summary

References Kolmogorov Incompressible String Complexity Pei Wang

Deﬁnition Models of Computation Remark Deﬁnitions Examples

Summary

References Kolmogorov Incompressible String Complexity Pei Wang

Deﬁnition Models of Computation Remark Deﬁnitions Examples

Summary

References Kolmogorov Outline Complexity Pei Wang Definition Models of Computation Definition Models of Computation Definitions Definitions Examples

Examples Kolmogorov Complexity and Kolmogorov Complexity and Entropy Entropy Preﬁx Code Kolmogorov Complexity and Preﬁx Code Entropy

A Random Bernoulli Sequence Ω

Kolmogorov Ω Complexity and Universal Kolmogorov Complexity and Universal Probability Probability Summary Summary References Kolmogorov AEP Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

I In information theory, the analog of the law of large Kolmogorov numbers is the asymptotic equipartition property (AEP). Complexity and Entropy Preﬁx Code I The law of large numbers states that for independent, Kolmogorov Complexity and 1 Pn Entropy identically distributed (i.i.d.) random variables, i=1 Xi n Algorithmically is close to its expected value EX for large values of n. Random and Incompressible 1 1 I The AEP states that log is close to the Random and n p(X1,X2,...,Xn) Incompressible Asymptotic Equipartition entropy H,where X1, X2,..., Xn are i.i.d. random variables. Property A Random Bernoulli Sequence I Thus, the probability p(X1, X2,..., Xn) assigned to an observed sequence will be close to 2−nH . Ω Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov AEP Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Summary

References Kolmogorov AEP Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Summary

References Kolmogorov AEP Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Summary

References Kolmogorov AEP Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov I This enables us to divide the set of all sequences into two Complexity and Entropy sets, the typical set, where the sample entropy is close to Preﬁx Code Kolmogorov Complexity and the true entropy, and the nontypical set, which contains Entropy the other sequences. Algorithmically Random and Incompressible Random and I Any property that is proved for the typical sequences will Incompressible Asymptotic Equipartition then be true with high probability and will determine the Property A Random Bernoulli average behavior of a large sample. Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov AEP Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov AEP Complexity Pei Wang Theorem Definition If X1, X2, ... are i.i.d.∼ p(x), then Models of Computation Definitions 1 Examples − log p(X , X , ..., X ) → H(X) in probability. (39) Kolmogorov 1 2 n Complexity and n Entropy Prefix Code Kolmogorov Complexity and Proof. Functions of independent random variables are Entropy Algorithmically also independent random variables. Thus, since the Xi Random and Incompressible are i.i.d., so are log p(Xi ).Hence,by the weak law of large Random and Incompressible numbers, Asymptotic Equipartition Property A Random Bernoulli 1 1 X Sequence − log p(X , X , ..., X ) = − log p(X ) Ω n 1 2 n n i i Kolmogorov Complexity and → −E log p(X) in probability Universal Probability

= H(x). Summary

References Kolmogorov Typical Set Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy Deﬁnition Preﬁx Code (n) Kolmogorov Complexity and The typical set A with respect to p(x) is the set of Entropy n sequences (X1, X2,..., Xn) ∈ X with the property Algorithmically Random and Incompressible −n(H(X)+) −n(H(X)−) Random and 2 ≤ p(X1, X2,..., Xn) ≤ 2 . (40) Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Typical Set Complexity Pei Wang

Definition Models of Computation Definitions As a consequence of the AEP, we can show that the set Examples (n) Kolmogorov A has the following properties: Complexity and Entropy Theorem Prefix Code Kolmogorov Complexity and Entropy (n) 1. If (X1, X2, ..., Xn) ∈ A , then Algorithmically 1 Random and H(X) − ≤ − n log p(X1, X2, ..., Xn) ≤ H(X) + . Incompressible Random and (n) Incompressible 2. Pr(A ) > 1 − for n sufficiently large. Asymptotic Equipartition Property (n) n(H(x)+) A Random Bernoulli 3. |A | ≤ 2 . Sequence (n) n(H(x)−) 4. |A | ≥ (1 − )2 for n sufficiently large. Ω Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Typical Set Complexity Pei Wang

Summary

References Kolmogorov Typical Set Complexity Pei Wang

Summary

References Kolmogorov Typical Set Complexity Pei Wang

Summary

References Kolmogorov Typical Set Complexity Pei Wang

Summary

References Kolmogorov Typical Set Complexity Pei Wang

Definition Models of Computation Definitions Proof. Property (3) follows from: Examples Kolmogorov X Complexity and 1 = p(x) Entropy n Prefix Code x∈X Kolmogorov Complexity and X Entropy ≥ p(x) Algorithmically Random and (n) x∈A Incompressible Random and X −n(H(X)+) Incompressible ≥ 2 Asymptotic Equipartition Property (n) A Random Bernoulli x∈A Sequence −n(H(X)+) (n) Ω = 2 |A |. Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Proof Cont. Complexity Pei Wang

Definition Models of Computation For sufficiently large n, Definitions Examples (n) Kolmogorov Pr(A ) > 1 − , Complexity and Entropy Prefix Code Kolmogorov Complexity and so that Entropy

Algorithmically (n) Random and 1 − < Pr(A ) Incompressible Random and X −n(H(X)−) Incompressible ≤ 2 Asymptotic Equipartition Property (n) ∈ A Random Bernoulli x A Sequence

−n(H(X)−) (n) Ω = 2 |A |. Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Outline Complexity Pei Wang Definition Models of Computation Definition Models of Computation Definitions Definitions Examples

Examples Kolmogorov Complexity and Kolmogorov Complexity and Entropy Entropy Preﬁx Code Kolmogorov Complexity and Preﬁx Code Entropy

A Random Bernoulli Sequence Ω

Kolmogorov Ω Complexity and Universal Kolmogorov Complexity and Universal Probability Probability Summary Summary References Kolmogorov A Random Bernoulli Sequence Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy Theorem Preﬁx Code Kolmogorov Complexity and Let X1, X2,..., Xn be drawn i.i.d.∼ Bernoulli(θ). Then Entropy Algorithmically Random and 1 Incompressible K (X1, X2, ..., Xn|n) → H0(θ)in probability. (41) Random and n Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov A Random Bernoulli Sequence Complexity Pei Wang

Definition Models of Computation ¯ 1 Pn Definitions Proof. LetXn = n i=1 Xi denote the proportion of 1’s in Examples X1, X2,..., Xn. Then using the method described in Kolmogorov Complexity and section 1, we have Entropy Prefix Code Kolmogorov Complexity and ¯ Entropy K (X1, X2,..., Xn|n) < nH0(Xn) + 2 log n + c, (42) Algorithmically Random and ¯ Incompressible and since by the weak law of large numbers, Xn → θ in Random and Incompressible probability, we have Asymptotic Equipartition Property A Random Bernoulli 1 Sequence Pr( K (X , X ,..., X |n) − H (θ) ≥ ) → 0. (43) Ω n 1 2 n 0 Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Proof cont. Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

I Conversely, we can bound the number of sequences Kolmogorov with complexity signiﬁcantly lower than the entropy. Complexity and Entropy Preﬁx Code I From the AEP, we can divide the set of sequences Kolmogorov Complexity and into the typical set and the nontypical set. Entropy Algorithmically n(H0(θ)−) Random and I There are at least (1 − )2 sequences in the Incompressible Random and typical set. Incompressible Asymptotic Equipartition At most 2n(H0(θ)−c) sequences in the typical set. of Property I A Random Bernoulli these typical sequences can have a complexity less Sequence Ω than n(H0(θ) − c). Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Proof cont. Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Summary

References Kolmogorov Proof cont. Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Summary

References Kolmogorov Proof cont. Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Summary

References Kolmogorov Proof cont. Complexity Pei Wang The probability that the complexity of the random Definition sequence is less than n(H0(θ) − c)is Models of Computation Definitions Examples n Pr(K (X |n) < n(H0(θ) − c)) Kolmogorov Complexity and n (n) n (n) n Entropy ≤ Pr(X 6∈ A ) + Pr(X ∈ A , K (K |n) < n(H0(θ) − c)) Prefix Code Kolmogorov Complexity and X Entropy ≤ + p(xn) Algorithmically n (n) n Random and X ∈A ,K (K |n)

which is arbitrarily small for appropriate choice of , n, Summary

and c. References Kolmogorov Proof cont. Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Hence with high probability, the Kolmogorov complexity of Entropy the random sequence is close to the entropy, and we Preﬁx Code Kolmogorov Complexity and have Entropy Algorithmically Random and 1 Incompressible K (X1, X2,..., Xn|n) → H0(θ)in probability. (44) Random and n Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Epimenides Liar Paradox Complexity Pei Wang

Definition Models of Computation Definitions Consider the following paradoxical statement: Examples Kolmogorov Complexity and This statement is false. Entropy Prefix Code Kolmogorov Complexity and Entropy

This paradox is sometimes stated in a two-statement Algorithmically form: Random and Incompressible Random and Incompressible Asymptotic Equipartition The next statement is false. Property A Random Bernoulli The preceding statement is true. Sequence Ω It illustrates the pitfalls involved in self-reference. Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Gödel’s Incompleteness Theorem Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

I In 1931, Gödel used this idea of self-reference to Kolmogorov Complexity and show that any interesting system of mathematics is Entropy Preﬁx Code not complete; there are statements in the system that Kolmogorov Complexity and Entropy

are true but that cannot be proved within the system. Algorithmically Random and Incompressible I To accomplish this, he translated theorems and Random and Incompressible proofs into integers and constructed a statement of Asymptotic Equipartition Property A Random Bernoulli the above form, which can therefore not be proved Sequence true or false. Ω Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Gödel’s Incompleteness Theorem Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Summary

References Kolmogorov The Halting Problem Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

I The halting problem is an essential example of Kolmogorov Complexity and Gödel’s incompleteness theorem. Entropy Preﬁx Code Kolmogorov Complexity and I In essence, it states that for any computational Entropy model, there is no general algorithm to decide Algorithmically Random and whether a program will halt or not (go on forever). Incompressible Random and Incompressible Asymptotic Equipartition I Note that it is not a statement about any speciﬁc Property A Random Bernoulli program. The halting problem says that we cannot Sequence answer this question for all programs. Ω Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov The Halting Problem Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Summary

References Kolmogorov The Halting Problem Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Summary

References Kolmogorov The Noncomputability of Kolmogorov Complexity Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov I One of the consequences of the nonexistence of an Complexity and algorithm for the halting problem is the Entropy Preﬁx Code Kolmogorov Complexity and noncomputability of Kolmogorov complexity. Entropy

Algorithmically I The only way to ﬁnd the shortest program in general Random and Incompressible is to try all short programs and see which of them Random and Incompressible Asymptotic Equipartition can do the job. Property A Random Bernoulli Sequence

I There is no effective (ﬁnite mechanical) way to tell Ω

whether or not they will halt and what they will print Kolmogorov Complexity and out. Universal Probability

Summary

References Kolmogorov The Noncomputability of Kolmogorov Complexity Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

I There is no effective (ﬁnite mechanical) way to tell Ω

whether or not they will halt and what they will print Kolmogorov Complexity and out. Universal Probability

Summary

References Kolmogorov The Noncomputability of Kolmogorov Complexity Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

I There is no effective (ﬁnite mechanical) way to tell Ω

whether or not they will halt and what they will print Kolmogorov Complexity and out. Universal Probability

Summary

References Kolmogorov Chaitin’s Mystical, Magical Number, Ω Complexity Pei Wang

Definition Models of Computation Definitions Examples Definition Kolmogorov Complexity and Entropy X −`(p) Prefix Code Ω = 2 . (45) Kolmogorov Complexity and Entropy

p:U(p)halts Algorithmically Random and Incompressible Random and Note that Ω = Pr(U(p)halts), the probability that the given Incompressible Asymptotic Equipartition universal computer halts when the input to the computer Property A Random Bernoulli 1 Sequence is a binary string drawn according to a Bernoulli( 2 ) process. Ω Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Chaitin’s Mystical, Magical Number, Ω Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy Prefix Code Since the programs that halt are prefix-free, their lengths Kolmogorov Complexity and satisfy the Kraft inequality, and hence the sum above is Entropy Algorithmically always between 0 and 1. Let Ωn = .ω1ω2 . . . ωn denote the Random and Incompressible first n bits of Ω. Random and Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Chaitin’s Mystical, Magical Number, Ω Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy The properties of Ω are as follows: Preﬁx Code Kolmogorov Complexity and 1. Ω is noncomputable. Entropy Algorithmically 2. Ω is a "philosopher’s stone". Random and Incompressible Random and 3. Ω is algorithmically random. Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Chaitin’s Mystical, Magical Number, Ω Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Chaitin’s Mystical, Magical Number, Ω Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Ω : noncomputable Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy Preﬁx Code Kolmogorov Complexity and There is no effective (ﬁnite, mechanical) way to check Entropy whether arbitrary programs halt (the halting problem), so Algorithmically Random and there is no effective way to compute Ω. Incompressible Random and Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Ω : a "philosopher’s stone" Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov I Knowing Ω to an accuracy of n bits will enable us to decide the Complexity and truth of any provable or finitely refutable mathematical theorem Entropy Prefix Code that can be written in less than n bits. Kolmogorov Complexity and Entropy I Actually, all that this means is that given n bits of Ω, there is an Algorithmically effective procedure to decide the truth of n-bit theorems; the Random and procedure may take an arbitrarily long (but finite) time. Incompressible Random and Incompressible I Without knowing Ω, it is not possible to check the truth or falsity Asymptotic Equipartition Property of every theorem by an effective procedure (Gödel’s A Random Bernoulli incompleteness theorem). Sequence Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Ω : a "philosopher’s stone" Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Ω : a "philosopher’s stone" Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Ω : a "philosopher’s stone" Complexity Pei Wang

Definition Models of Computation Definitions Examples −`(p) We run all programs until the sum of the masses 2 contributed by Kolmogorov Complexity and programs that halt equals or exceeds Ωn = .ω1ω2 . . . ωn. Then, since Entropy −n Prefix Code Ω − Ωn < 2 , (46) Kolmogorov Complexity and Entropy

−`(p) Algorithmically we know that the sum of all further contributions of the form 2 to Random and −n Incompressible Ω from programs that halt must also be less than 2 .This implies that Random and Incompressible no program of length ≤ n that has not yet halted will ever halt, which Asymptotic Equipartition Property enables us to decide the halting or nonhalting of all programs of length A Random Bernoulli Sequence

≤ n. Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Ω :algorithmically random Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy Theorem Preﬁx Code Kolmogorov Complexity and Ω cannot be compressed by more than a constant; that Entropy is, there exists a constant c such that Algorithmically Random and Incompressible Random and K (ω1ω2 . . . ωn) ≥ n − c, for all n. (47) Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Ω :algorithmically random Complexity Pei Wang

Proof. Definition Models of Computation I If we are given n bits of Ω, we can determine whether or not any Definitions Examples program of length ≤ n halts. Kolmogorov Complexity and I Using K (Ωn) bits, we can calculate n bits of Ω, then we can Entropy generate a list of all programs of length ≤ n that halt, together Prefix Code Kolmogorov Complexity and with their corresponding outputs. Entropy

I We ﬁnd the ﬁrst string x0 that is not on this list. The string x0 is Algorithmically Random and then the shortest string with Kolmogorov complexity K (x0) ≥ n. Incompressible Random and I The complexity of this program to print x0 is K (Ωn) + c, which Incompressible Asymptotic Equipartition must be at least as long as the shortest program for Property A Random Bernoulli x0.Consequently, for all n, Sequence Ω K (Ωn) + c ≥ K (x0) > n. Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Ω :algorithmically random Complexity Pei Wang

Summary

References Kolmogorov Ω :algorithmically random Complexity Pei Wang

Summary

References Kolmogorov Ω :algorithmically random Complexity Pei Wang

Summary

References Kolmogorov Universal Probability Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

I From our earlier discussions, it is clear that most sequences of Kolmogorov length n have complexity close to n. Complexity and Entropy −`(p) Preﬁx Code I Since the probability of an input program p is 2 ,shorter Kolmogorov Complexity and programs are much more probable than longer ones; and when Entropy they produce long strings, shorter programs do not produce Algorithmically Random and random strings; they produce strings with simply described Incompressible Random and structure. Incompressible Asymptotic Equipartition I The probability distribution on the output strings is far from Property A Random Bernoulli uniform. Under the computer-induced distribution, simple strings Sequence are more likely than complicated strings of the same length. Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universal Probability Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universal Probability Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universal Probability Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Definition Complexity and The universal probability of a string x is Entropy Prefix Code Kolmogorov Complexity and X −`(p) Entropy PU (x) = 2 = Pr(U(p) = x), (48) Algorithmically Random and p:U(p)=x Incompressible Random and Incompressible which is the probability that a program randomly drawn as Asymptotic Equipartition Property A Random Bernoulli a sequence of fair coin flips p1, p2,... will print out the Sequence string x. Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universal Probability Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

This probability mass function is called universal because Kolmogorov Complexity and of the following theorem. Entropy Preﬁx Code Kolmogorov Complexity and Theorem Entropy

For every computer A, Algorithmically Random and 0 Incompressible ( ) ≥ ( ) Random and PU x cAPA x (49) Incompressible Asymptotic Equipartition Property ∗ 0 A Random Bernoulli for every string x ∈ {0, 1} , where the constant cA Sequence depends only on U and A. Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universal Probability Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Proof. From the discussion of Section 2, we recall that for Complexity and 0 Entropy every program p for A that prints x, there exists a program p Preﬁx Code 0 0 Kolmogorov Complexity and for U of length not more than `(p ) + cA produced by preﬁxing Entropy A a simulation program for . Hence, Algorithmically Random and 0 0 X −`(p) X −`(p )−cA 0 Incompressible PU (x) = 2 = 2 = cAPA(x), Random and p:U(p)=x p0:A(p0)=x Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universal Probability Complexity Pei Wang

Definition Models of Computation Definitions Examples Remark Kolmogorov Complexity and Entropy I Any sequence drawn according to a computable Prefix Code Kolmogorov Complexity and probability mass function on binary strings can be Entropy considered to be produced by some computer A acting on Algorithmically Random and a random input (via the probability inverse transformation Incompressible Random and acting on a random input). Incompressible Asymptotic Equipartition Property I Hence, the universal probability distribution includes a A Random Bernoulli mixture of all computable probability distributions. Sequence Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universal Probability Complexity Pei Wang

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universal Probability Complexity Pei Wang

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Universal Probability Complexity Pei Wang

Definition Models of Computation Remark Definitions (Bounded likelihood ratio) Examples Kolmogorov Complexity and I In particular, Theorem above guarantees that a likelihood Entropy ratio test of the hypothesis that X is drawn according to U Prefix Code Kolmogorov Complexity and versus the hypothesis that it is drawn according to A will Entropy have bounded likelihood ratio. Algorithmically Random and Incompressible I If U and A are universal, then PU (x)/PA(x) is bounded Random and Incompressible away from 0 and infinity for all x. Asymptotic Equipartition Property A Random Bernoulli I Apparently, U, which is a mixture of all computable Sequence distributions, can never be rejected completely as the true Ω distribution of any data drawn according to some Kolmogorov Complexity and computable probability distribution. Universal Probability

Summary

References Kolmogorov Universal Probability Complexity Pei Wang

Summary

References Kolmogorov Universal Probability Complexity Pei Wang

Summary

References Kolmogorov Universal Probability Complexity Pei Wang

Summary

References Kolmogorov Kolmogorov Complexity and Universal Complexity Probability Pei Wang

Deﬁnition Models of Computation Deﬁnitions Examples

Kolmogorov Complexity and Entropy Theorem Preﬁx Code 1 Kolmogorov Complexity and (Equivalence of K (x) and log ) There exists a constant c, Entropy PU (x) independent of x, such that Algorithmically Random and Incompressible −K (x) −K (x) Random and 2 ≤ PU (x) ≤ c2 (50) Incompressible Asymptotic Equipartition Property for all strings x. Thus, the universal probability of a string x is A Random Bernoulli Sequence determined essentially by its Kolmogorov complexity. Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Complexity

Pei Wang

Definition Models of Computation Definitions Examples Remark Kolmogorov Complexity and 1 Entropy This implies that K (x) and log P (x) have equal status as U Prefix Code universal complexity measures, since Kolmogorov Complexity and Entropy

Algorithmically 0 1 Random and K (x) − c ≤ log ≤ K (x) (51) Incompressible PU (x) Random and Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Complexity

Pei Wang

Definition Remark Models of Computation Definitions Examples I Notice the striking similarity between the relationship of K (x) 1 Kolmogorov and log in Kolmogorov complexity and the relationship of Complexity and PU (x) 1 Entropy H(x) and log p(x) in information theory. Prefix Code Kolmogorov Complexity and Entropy I The ideal Shannon code length assignment 1 Algorithmically `(x) = log p(x) achieves an average description length Random and H(x),while in Kolmogorov complexity theory, the ideal Incompressible Random and description length log 1 is almost equal to K (x). Incompressible PU (x) Asymptotic Equipartition Property 1 A Random Bernoulli I Thus, log p(x) is the natural notion of descriptive complexity of x Sequence in algorithmic as well as probabilistic settings. Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Complexity

Pei Wang

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Complexity

Pei Wang

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Complexity

Pei Wang

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Complexity

Pei Wang

Definition Proof. Our objective in the proof is to find a short program to Models of Computation describe the strings that have high PU (x). Definitions Examples

I We want to construct a code tree in such a way that strings with Kolmogorov high probability have low depth. Complexity and Entropy I Since we cannot calculate the probability of a string, we do not Preﬁx Code Kolmogorov Complexity and know a priori the depth of the string on the tree. Entropy Algorithmically I Instead, we assign x successively to the nodes of the tree, Random and assigning x to nodes closer and closer to the root as our Incompressible Random and estimate of PU (x) improves. Incompressible Asymptotic Equipartition We want the computer to be able to recreate the tree and use Property I A Random Bernoulli the lowest depth node corresponding to the string x to Sequence reconstruct the string. Ω Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov Complexity

Pei Wang

Definition Proof. Our objective in the proof is to find a short program to Models of Computation describe the strings that have high PU (x). Definitions Examples

Summary

References Kolmogorov Complexity

Pei Wang

Definition Proof. Our objective in the proof is to find a short program to Models of Computation describe the strings that have high PU (x). Definitions Examples

Summary

References Kolmogorov Complexity

Pei Wang

Definition Proof. Our objective in the proof is to find a short program to Models of Computation describe the strings that have high PU (x). Definitions Examples

Summary

References Kolmogorov Complexity

Deﬁnition. The Kolmogorov complexity KU (x) of a string x is Pei Wang

Definition KU (x) = min `(p), p:U(p)=x Models of Computation Definitions Examples KU (x|`(x)) = min `(p), Kolmogorov p:U(p,`(x))=x Complexity and Entropy Universality of Kolmogorov complexity. There exists a Prefix Code Kolmogorov Complexity and universal computer U such that for any other computer A, Entropy Algorithmically Random and KU (x) ≤ KA(x) + cA. Incompressible Random and Incompressible Asymptotic Equipartition for any string x, where the constant cA does not depend on x. Property A Random Bernoulli If U and A are universal, |KU (x) − KA(x)| ≤ c for all x. Sequence

Upper bound on Kolmogorov complexity. Ω

Kolmogorov KU (x|`(x)) ≤ `(x) + c. Complexity and Universal Probability

KU (x) ≤ KU (x|`(x)) + 2 log `(x) + c. Summary

References Kolmogorov Kolmogorov complexity and entropy. If X1, X2,...are i.i.d. Complexity integer valued random variables with entropy H, there exists a Pei Wang constant c such that for all n, Definition 1 (|X | − 1) log n c Models of Computation H ≤ E K (x n|n) ≤ H + + Definitions n n n Examples Kolmogorov Complexity and Lower bound on Kolmogorov complexity. There are no Entropy k more than 2 strings x with complexity K (x) < k. If Prefix Code Kolmogorov Complexity and 1 Entropy X1, X2,... Xn are drawn according to a Bernoulli 2 process, Algorithmically −k Random and P(K (X1, X2,..., Xn|n) < n − k) < 2 . Incompressible Random and Incompressible Asymptotic Equipartition Definition. A sequence x is said to be incompressible if Property K (x1, x2, x3,..., xn|n) A Random Bernoulli → 1. Sequence n Ω Strong law of large numbers for incompressible Kolmogorov sequences. Complexity and Universal n Probability K (x1, x2, x3,..., xn|n) 1 X 1 → 1 ⇒ x → . Summary n n i 2 i=1 References Kolmogorov Complexity Definition. Ω = P 2−`(p) = Pr(U(p)halts) is the p:U(p)halts Pei Wang probability that the computer halts when the input p to the 1 computer is a binary string drawn according to a Bernoulli( 2 ) Definition Models of Computation process. Definitions Properties of Ω. Examples Ω Kolmogorov is noncomputable. Complexity and Ω is a "philosopher’s stone". Entropy Prefix Code Ω is algorithmically random. Kolmogorov Complexity and Definition. The universal probability of a string x is Entropy Algorithmically X Random and ( ) = −`(p) = (U( ) = ), Incompressible PU x 2 Pr p x Random and :U( )= Incompressible p p x Asymptotic Equipartition Property A Random Bernoulli Universality of PU (x). For every computer A, Sequence Ω 0 PU (x) ≥ cAPA(x) Kolmogorov Complexity and Universal ∗ 0 Probability for every string x ∈ {0, 1} , where the constant cA depends only on U and A. Summary References Kolmogorov Complexity

Pei Wang

Definition Models of Computation Definitions Equivalence of K (x) and log 1 . There exists a constant c Examples PU (x) Kolmogorov independent of x such that Complexity and Entropy Prefix Code 1 Kolmogorov Complexity and log − K (x) ≤ c Entropy PU (x) Algorithmically Random and Incompressible for all strings x. Thus, the universal probability of a string x is Random and Incompressible essentially determined by its Kolmogorov complexity. Asymptotic Equipartition Property A Random Bernoulli Sequence

Ω

Kolmogorov Complexity and Universal Probability

Summary

References Kolmogorov References I Complexity Pei Wang

Deﬁnition Models of Computation Deﬁnitions Cover, T., Thomas, J., et al. (1991). Examples

Elements of information theory. Kolmogorov Complexity and Wiley Online Library. Entropy Preﬁx Code Kolmogorov Complexity and Feller, W. (2008). Entropy An introduction to probability theory and its Algorithmically Random and applications, volume 2. Incompressible Random and John Wiley & Sons. Incompressible Asymptotic Equipartition Property A Random Bernoulli Li, M. and Vitányi, P. (2008). Sequence

An introduction to Kolmogorov complexity and its Ω applications. Kolmogorov Complexity and Springer-Verlag New York Inc. Universal Probability

Summary

References