Lecture notes on descriptional complexity and randomness Peter Gács Boston University [email protected] A didactical survey of the foundations of Algorithmic Information The- ory. These notes introduce some of the main techniques and concepts of the field. They have evolved over many years and have been used and ref- erenced widely. “Version history” below gives some information on what has changed when. Contents Contents iii 1 Complexity1 1.1 Introduction ........................... 1 1.1.1 Formal results ...................... 3 1.1.2 Applications ....................... 6 1.1.3 History of the problem.................. 8 1.2 Notation ............................. 10 1.3 Kolmogorov complexity ..................... 11 1.3.1 Invariance ........................ 11 1.3.2 Simple quantitative estimates .............. 14 1.4 Simple properties of information ................ 16 1.5 Algorithmic properties of complexity .............. 19 1.6 The coding theorem....................... 24 1.6.1 Self-delimiting complexity................ 24 1.6.2 Universal semimeasure.................. 27 1.6.3 Prefix codes ....................... 28 1.6.4 The coding theorem for ¹Fº . 30 1.6.5 Algorithmic probability.................. 31 1.7 The statistics of description length ............... 32 2 Randomness 37 2.1 Uniform distribution....................... 37 2.2 Computable distributions..................... 40 2.2.1 Two kinds of test..................... 40 2.2.2 Randomness via complexity............... 41 2.2.3 Conservation of randomness............... 43 2.3 Infinite sequences ........................ 46 iii Contents 2.3.1 Null sets ......................... 47 2.3.2 Probability space..................... 52 2.3.3 Computability ...................... 54 2.3.4 Integral.......................... 54 2.3.5 Randomness tests .................... 55 2.3.6 Randomness and complexity............... 56 2.3.7 Universal semimeasure, algorithmic probability . 58 2.3.8 Randomness via algorithmic probability......... 60 3 Information 63 3.1 Information-theoretic relations.................. 63 3.1.1 The information-theoretic identity............ 63 3.1.2 Information non-increase................. 72 3.2 The complexity of decidable and enumerable sets . 74 3.3 The complexity of complexity.................. 78 3.3.1 Complexity is sometimes complex............ 78 3.3.2 Complexity is rarely complex .............. 79 4 Generalizations 83 4.1 Continuous spaces, noncomputable measures.......... 83 4.1.1 Introduction ....................... 83 4.1.2 Uniform tests....................... 85 4.1.3 Sequences ........................ 87 4.1.4 Conservation of randomness............... 88 4.2 Test for a class of measures ................... 90 4.2.1 From a uniform test ................... 90 4.2.2 Typicality and class tests................. 91 4.2.3 Martin-Löf’s approach.................. 94 4.3 Neutral measure......................... 97 4.4 Monotonicity, quasi-convexity/concavity . 101 4.5 Algorithmic entropy ....................... 103 4.5.1 Entropy.......................... 104 4.5.2 Algorithmic entropy ................... 105 4.5.3 Addition theorem..................... 106 4.5.4 Information........................ 110 4.6 Randomness and complexity................... 111 4.6.1 Discrete space ...................... 111 4.6.2 Non-discrete spaces ................... 113 iv Contents 4.6.3 Infinite sequences .................... 114 4.6.4 Bernoulli tests ...................... 116 4.7 Cells ............................... 118 4.7.1 Partitions......................... 119 4.7.2 Computable probability spaces . 122 5 Exercises and problems 125 A Background from mathematics 131 A.1 Topology............................. 131 A.1.1 Topological spaces.................... 131 A.1.2 Continuity ........................ 133 A.1.3 Semicontinuity...................... 134 A.1.4 Compactness....................... 135 A.1.5 Metric spaces....................... 136 A.2 Measures............................. 141 A.2.1 Set algebras ....................... 141 A.2.2 Measures......................... 141 A.2.3 Integral.......................... 144 A.2.4 Density.......................... 145 A.2.5 Random transitions.................... 146 A.2.6 Probability measures over a metric space . 147 B Constructivity 155 B.1 Computable topology ...................... 155 B.1.1 Representations ..................... 155 B.1.2 Constructive topological space . 156 B.1.3 Computable functions .................. 158 B.1.4 Computable elements and sequences . 159 B.1.5 Semicomputability.................... 160 B.1.6 Effective compactness .................. 161 B.1.7 Computable metric space . 163 B.2 Constructive measure theory................... 166 B.2.1 Space of measures .................... 166 B.2.2 Computable and semicomputable measures . 168 B.2.3 Random transitions.................... 169 Bibliography 171 v Contents Version history May 2021: changed the notation to the now widely accepted one: Kolmogorov’s original complexity is denoted 퐶¹Fº (instead of the earlier ¹Fº), while the pre- fix complexity is denoted ¹Fº instead of the earlier 퐻¹Fº. Corrected a remark on the notation 0=. June 2013: corrected a slight mistake in the section on the section on ran- domness via algorithmic probability. February 2010: chapters introduced, and recast using the memoir class. April 2009: besides various corrections, a section is added on infinite sequen- ces. This is not new material, just places the most classical results on randomness of infinite sequences before the more abstract theory. January 2008: major rewrite. • Added formal definitions throughout. • Made corrections in the part on uniform tests and generalized complexity, based on remarks of Hoyrup, Rojas and Shen. • Rearranged material. • Incorporated material on uniform tests from the work of Hoyrup-Rojas. • Added material on class tests. vi 1 Complexity Je n’ai fait celle-ci plus longue que parce que je n’ai pas eu le loisir de la faire plus courte. Pascal Ainsi, au jeu de croix ou pile, l’arrivée de croix cent fois de suite, nous paraît extraordinaire; parce que le nombre presque infini des combinaisons qui peuvent arriver en cent coups étant partagé en séries regulières, ou dans lesquelles nous voyons régner un ordre facile à saisir, et en séries irregulières; celles-ci sont incomparablement plus nombreuses. Laplace 1.1 Introduction The present section can be read as an independent survey on the problems of randomness. It serves as some motivation for the dryer stuff to follow. If we toss a coin 100 times and it shows each time Head, we feel lifted to a land of wonders, like Rosencrantz and Guildenstern in [49]. The argu- ment that 100 heads are just as probable as any other outcome convinces us only that the axioms of Probability Theory, as developed in [28], do not solve 1 1. Complexity all mysteries they are sometimes supposed to. We feel that the sequence con- sisting of 100 heads is not random, though others with the same probability are. Due to the somewhat philosophical character of this paradox, its history is marked by an amount of controversy unusual for ordinary mathematics. Before the reader hastes to propose a solution, let us consider a less trivial example, due to L. A. Levin. Suppose that in some country, the share of votes for the ruling party in 30 consecutive elections formed a sequence 0.99F7 where for every even 7, the num- ber F7 is the 7-th digit of c = 3.1415 .... Though many of us would feel that the election results were manipulated, it turns out to be surprisingly difficult to prove this by referring to some general principle. In a sequence of < fair elections, every sequence l of < digits has approxi- −< mately the probability &< ¹lº = 10 to appear as the actual sequence of third digits. Let us fix <. We are given a particular sequence l and want to test the validity of the government’s claim that the elections were fair. We interpret the assertion “l is random with respect to &<” as a synonym for “there is no reason to challenge the claim that l arose from the distribution &<”. How can such a claim be challenged at all? The government, just like the weather forecaster who announces 30% chance of rain, does not guarantee any particular set of outcomes. However, to stand behind its claim, it must agree to any bet based on the announced distribution. Let us call a payoff function with Í respect the distribution % any nonnegative function B¹lº with l %¹lºB¹lº ≤ 1. If a “nonprofit” gambling casino asks 1 dollar for a game and claims that each outcome has probability %¹lº then it must agree to pay B¹lº dollars on outcome l. We would propose to the government the following payoff function B0 with </2 respect to &<: let B0 ¹lº = 10 for all sequences l whose even digits are given by c, and 0 otherwise. This bet would cost the government 10</2 − 1 dollars. Unfortunately, we must propose the bet before the elections take place and it is unlikely that we would have come up exactly with the payoff function B0. Is then the suspicion unjustifiable? No. Though the function B0 is not as natural as to guess it in advance, it is still highly “regular”. And already Laplace assumed in [15] that the number of “regular” bets is so small we can afford to make them all in advance and still win by a wide margin. Kolmogorov discovered in [27] and [29] (probably without knowing about [15] but following a four decades long controversy on von Mises’ con- cept of randomness, see [53]) that to make this approach work we must define “regular” or “simple” as “having a short description”