Kolmogorov Complexity Pei Wang Kolmogorov Complexity Pei Wang School of Mathematical Sciences Peking University [email protected] May 15, 2012 Kolmogorov Outline Complexity Pei Wang Definition Models of Computation Definition Models of Computation Definitions Definitions Examples Examples Kolmogorov Complexity and Kolmogorov Complexity and Entropy Entropy Prefix Code Kolmogorov Complexity and Prefix Code Entropy Kolmogorov Complexity and Entropy Algorithmically Random and Incompressible Algorithmically Random and Incompressible Random and Incompressible Random and Incompressible Asymptotic Equipartition Property A Random Bernoulli Asymptotic Equipartition Property Sequence A Random Bernoulli Sequence Ω Kolmogorov Ω Complexity and Universal Kolmogorov Complexity and Universal Probability Probability Summary Summary References Kolmogorov A Turing Machine Complexity Pei Wang I a finite-state machine operating on a finite symbol set, Definition Models of Computation I at each unit of time, the machine inspects the Definitions program tape, writes some symbols on a work tape, Examples Kolmogorov changes its state according to its transition table, and Complexity and Entropy calls for more program. Prefix Code Kolmogorov Complexity and I reads from left to right only. Entropy Algorithmically Random and Incompressible Random and Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence Ω Kolmogorov Complexity and Universal Probability Figure: A Turing Machine Summary References Most of the materials is copy-edited from[Cover et al., 1991]. Kolmogorov A Turing Machine Complexity Pei Wang I a finite-state machine operating on a finite symbol set, Definition Models of Computation I at each unit of time, the machine inspects the Definitions program tape, writes some symbols on a work tape, Examples Kolmogorov changes its state according to its transition table, and Complexity and Entropy calls for more program. Prefix Code Kolmogorov Complexity and I reads from left to right only. Entropy Algorithmically Random and Incompressible Random and Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence Ω Kolmogorov Complexity and Universal Probability Figure: A Turing Machine Summary References Most of the materials is copy-edited from[Cover et al., 1991]. Kolmogorov A Turing Machine Complexity Pei Wang I a finite-state machine operating on a finite symbol set, Definition Models of Computation I at each unit of time, the machine inspects the Definitions program tape, writes some symbols on a work tape, Examples Kolmogorov changes its state according to its transition table, and Complexity and Entropy calls for more program. Prefix Code Kolmogorov Complexity and I reads from left to right only. Entropy Algorithmically Random and Incompressible Random and Incompressible Asymptotic Equipartition Property A Random Bernoulli Sequence Ω Kolmogorov Complexity and Universal Probability Figure: A Turing Machine Summary References Most of the materials is copy-edited from[Cover et al., 1991]. Kolmogorov A Turing Machine Complexity Pei Wang Definition I The programs this Turing Machine reads form a Models of Computation Definitions prefix-free set: Examples I no program leading to a halting computation can be Kolmogorov Complexity and the prefix of another such program. Entropy Prefix Code Kolmogorov Complexity and I We can view the Turing machine as a map: Entropy Algorithmically I from a set of finite-length binary strings to the set of Random and finite- or infinite-length binary strings, Incompressible Random and Incompressible I The computation may halt or not. Asymptotic Equipartition Property A Random Bernoulli Sequence Definition Ω ∗ ∗ 1 The set of functions f : f0; 1g ! f0; 1g [ f0; 1g Kolmogorov Complexity and computable by Turing machines is called the set of Universal partial recursive functions . Probability Summary References Kolmogorov A Turing Machine Complexity Pei Wang Definition I The programs this Turing Machine reads form a Models of Computation Definitions prefix-free set: Examples I no program leading to a halting computation can be Kolmogorov Complexity and the prefix of another such program. Entropy Prefix Code Kolmogorov Complexity and I We can view the Turing machine as a map: Entropy Algorithmically I from a set of finite-length binary strings to the set of Random and finite- or infinite-length binary strings, Incompressible Random and Incompressible I The computation may halt or not. Asymptotic Equipartition Property A Random Bernoulli Sequence Definition Ω ∗ ∗ 1 The set of functions f : f0; 1g ! f0; 1g [ f0; 1g Kolmogorov Complexity and computable by Turing machines is called the set of Universal partial recursive functions . Probability Summary References Kolmogorov A Turing Machine Complexity Pei Wang Definition I The programs this Turing Machine reads form a Models of Computation Definitions prefix-free set: Examples I no program leading to a halting computation can be Kolmogorov Complexity and the prefix of another such program. Entropy Prefix Code Kolmogorov Complexity and I We can view the Turing machine as a map: Entropy Algorithmically I from a set of finite-length binary strings to the set of Random and finite- or infinite-length binary strings, Incompressible Random and Incompressible I The computation may halt or not. Asymptotic Equipartition Property A Random Bernoulli Sequence Definition Ω ∗ ∗ 1 The set of functions f : f0; 1g ! f0; 1g [ f0; 1g Kolmogorov Complexity and computable by Turing machines is called the set of Universal partial recursive functions . Probability Summary References Kolmogorov A Turing Machine Complexity Pei Wang Definition I The programs this Turing Machine reads form a Models of Computation Definitions prefix-free set: Examples I no program leading to a halting computation can be Kolmogorov Complexity and the prefix of another such program. Entropy Prefix Code Kolmogorov Complexity and I We can view the Turing machine as a map: Entropy Algorithmically I from a set of finite-length binary strings to the set of Random and finite- or infinite-length binary strings, Incompressible Random and Incompressible I The computation may halt or not. Asymptotic Equipartition Property A Random Bernoulli Sequence Definition Ω ∗ ∗ 1 The set of functions f : f0; 1g ! f0; 1g [ f0; 1g Kolmogorov Complexity and computable by Turing machines is called the set of Universal partial recursive functions . Probability Summary References Kolmogorov A Turing Machine Complexity Pei Wang Definition I The programs this Turing Machine reads form a Models of Computation Definitions prefix-free set: Examples I no program leading to a halting computation can be Kolmogorov Complexity and the prefix of another such program. Entropy Prefix Code Kolmogorov Complexity and I We can view the Turing machine as a map: Entropy Algorithmically I from a set of finite-length binary strings to the set of Random and finite- or infinite-length binary strings, Incompressible Random and Incompressible I The computation may halt or not. Asymptotic Equipartition Property A Random Bernoulli Sequence Definition Ω ∗ ∗ 1 The set of functions f : f0; 1g ! f0; 1g [ f0; 1g Kolmogorov Complexity and computable by Turing machines is called the set of Universal partial recursive functions . Probability Summary References Kolmogorov A Turing Machine Complexity Pei Wang Definition I The programs this Turing Machine reads form a Models of Computation Definitions prefix-free set: Examples I no program leading to a halting computation can be Kolmogorov Complexity and the prefix of another such program. Entropy Prefix Code Kolmogorov Complexity and I We can view the Turing machine as a map: Entropy Algorithmically I from a set of finite-length binary strings to the set of Random and finite- or infinite-length binary strings, Incompressible Random and Incompressible I The computation may halt or not. Asymptotic Equipartition Property A Random Bernoulli Sequence Definition Ω ∗ ∗ 1 The set of functions f : f0; 1g ! f0; 1g [ f0; 1g Kolmogorov Complexity and computable by Turing machines is called the set of Universal partial recursive functions . Probability Summary References Kolmogorov Turing Machine Complexity Pei Wang Definition Models of Computation Definitions I Turing argued that this machine could mimic the Examples computational ability of a human being. Kolmogorov Complexity and Entropy I After Turing’s work, it turned out that every new Prefix Code Kolmogorov Complexity and computational system could be reduced to a Turing Entropy Algorithmically machine, and conversely. Random and Incompressible Random and I In particular,the familiar digital computer with its Incompressible Asymptotic Equipartition CPU, memory, and input output devices could be Property A Random Bernoulli simulated by and could simulate a Turing machine. Sequence Ω Kolmogorov Complexity and Universal Probability Summary References Kolmogorov Turing Machine Complexity Pei Wang Definition Models of Computation Definitions I Turing argued that this machine could mimic the Examples computational ability of a human being. Kolmogorov Complexity and Entropy I After Turing’s work, it turned out that every new Prefix Code Kolmogorov Complexity and computational system could be reduced to a Turing Entropy Algorithmically machine, and conversely. Random and Incompressible Random and I In particular,the familiar digital computer with its Incompressible Asymptotic Equipartition CPU, memory, and input output devices could be Property A Random Bernoulli simulated by and could simulate a Turing machine. Sequence Ω Kolmogorov Complexity