Lecture notes on descriptional complexity and randomness Peter Gács Boston University [email protected] A didactical survey of the foundations of Algorithmic Information The- ory. These notes are short on motivation, history and background but in- troduce some of the main techniques and concepts of the field. The “manuscript” has been evolving over the years. Please, look at “Version history” below to see what has changed when. arXiv:2105.04704v1 [cs.IT] 10 May 2021 Contents Contents iii 1 Complexity 1 1.1 Introduction ........................... 1 1.1.1 Formalresults ...................... 3 1.1.2 Applications ....................... 6 1.1.3 Historyoftheproblem . 8 1.2 Notation ............................. 10 1.3 Kolmogorovcomplexity . 11 1.3.1 Invariance ........................ 11 1.3.2 Simplequantitativeestimates . 14 1.4 Simple properties of information . 16 1.5 Algorithmic properties of complexity . 19 1.6 Thecodingtheorem .. ... .. .. ... .. ... .. ... 24 1.6.1 Self-delimitingcomplexity . 24 1.6.2 Universalsemimeasure . 27 1.6.3 Prefixcodes ....................... 28 1.6.4 The coding theorem for F .............. 30 ( ) 1.6.5 Algorithmicprobability. 31 1.7 Thestatisticsofdescriptionlength . 32 2 Randomness 37 2.1 Uniformdistribution . 37 2.2 Computabledistributions. 40 2.2.1 Twokindsoftest. 40 2.2.2 Randomnessviacomplexity . 41 2.2.3 Conservationofrandomness . 43 2.3 Infinitesequences . .. ... .. .. ... .. ... .. ... 46 iii Contents 2.3.1 Nullsets ......................... 47 2.3.2 Probabilityspace . 52 2.3.3 Computability . 54 2.3.4 Integral.......................... 54 2.3.5 Randomnesstests . 55 2.3.6 Randomnessandcomplexity . 56 2.3.7 Universal semimeasure, algorithmic probability . .. 58 2.3.8 Randomness via algorithmic probability . 60 3 Information 63 3.1 Information-theoreticrelations. 63 3.1.1 Theinformation-theoreticidentity . 63 3.1.2 Informationnon-increase. 72 3.2 The complexity of decidable and enumerable sets . 74 3.3 Thecomplexityofcomplexity . 78 3.3.1 Complexityissometimescomplex . 78 3.3.2 Complexityisrarelycomplex . 79 4 Generalizations 83 4.1 Continuous spaces, noncomputable measures . 83 4.1.1 Introduction .. ... .. .. ... .. ... .. ... 83 4.1.2 Uniformtests.. ... .. .. ... .. ... .. ... 85 4.1.3 Sequences ........................ 87 4.1.4 Conservationofrandomness . 88 4.2 Testforaclassofmeasures . 90 4.2.1 Fromauniformtest . 90 4.2.2 Typicalityandclasstests . 91 4.2.3 Martin-Löf’sapproach . 94 4.3 Neutralmeasure ......................... 97 4.4 Monotonicity, quasi-convexity/concavity . 101 4.5 Algorithmicentropy . 103 4.5.1 Entropy.......................... 104 4.5.2 Algorithmicentropy . 105 4.5.3 Additiontheorem. 106 4.5.4 Information. 110 4.6 Randomnessandcomplexity. 111 4.6.1 Discretespace . 111 4.6.2 Non-discretespaces . 113 iv Contents 4.6.3 Infinitesequences . 114 4.6.4 Bernoullitests . 116 4.7 Cells ............................... 118 4.7.1 Partitions .. .. ... .. .. ... .. ... .. ... 119 4.7.2 Computableprobabilityspaces . 122 5 Exercises and problems 125 A Background from mathematics 131 A.1 Topology ............................. 131 A.1.1 Topologicalspaces . 131 A.1.2 Continuity . .. ... .. .. ... .. ... .. ... 133 A.1.3 Semicontinuity . 134 A.1.4 Compactness . 135 A.1.5 Metricspaces. 136 A.2 Measures............................. 141 A.2.1 Setalgebras .. ... .. .. ... .. ... .. ... 141 A.2.2 Measures ......................... 141 A.2.3 Integral.......................... 144 A.2.4 Density.......................... 145 A.2.5 Randomtransitions. 146 A.2.6 Probability measures over a metric space . 147 B Constructivity 155 B.1 Computabletopology . 155 B.1.1 Representations . 155 B.1.2 Constructivetopologicalspace . 156 B.1.3 Computablefunctions . 158 B.1.4 Computableelementsandsequences . 159 B.1.5 Semicomputability . 160 B.1.6 Effectivecompactness . 161 B.1.7 Computablemetricspace. 163 B.2 Constructivemeasuretheory . 166 B.2.1 Spaceofmeasures . 166 B.2.2 Computable and semicomputable measures . 168 B.2.3 Randomtransitions. 169 Bibliography 171 v Contents Version history May 2021: changed the notation to the now widely accepted one: Kolmogorov’s original complexity is denoted F (instead of the earlier F ), while the pre- ( ) ( ) fix complexity is denoted F instead of the earlier F . ( ) ( ) June 2013: corrected a slight mistake in the section on the section on ran- domness via algorithmic probability. February 2010: chapters introduced, and recast using the memoir class. April 2009: besides various corrections, a section is added on infinite sequen- ces. This is not new material, just places the most classical results on randomness of infinite sequences before the more abstract theory. January 2008: major rewrite. • Added formal definitions throughout. • Made corrections in the part on uniform tests and generalized complexity, based on remarks of Hoyrup, Rojas and Shen. • Rearranged material. • Incorporated material on uniform tests from the work of Hoyrup-Rojas. • Added material on class tests. vi 1 Complexity Je n’ai fait celle-ci plus longue que parce que je n’ai pas eu le loisir de la faire plus courte. Pascal Ainsi, au jeu de croix ou pile, l’arrivée de croix cent fois de suite, nous paraît extraordinaire; parce que le nombre presque infini des combinaisons qui peuvent arriver en cent coups étant partagé en séries regulières, ou dans lesquelles nous voyons régner un ordre facile à saisir, et en séries irregulières; celles-ci sont incomparablement plus nombreuses. Laplace 1.1 Introduction The present section can be read as an independent survey on the problems of randomness. It serves as some motivation for the dryer stuff to follow. If we toss a coin 100 times and it shows each time Head, we feel lifted to a land of wonders, like Rosencrantz and Guildenstern in [49]. The argu- ment that 100 heads are just as probable as any other outcome convinces us only that the axioms of Probability Theory, as developed in [28], do not solve 1 1. Complexity all mysteries they are sometimes supposed to. We feel that the sequence con- sisting of 100 heads is not random, though others with the same probability are. Due to the somewhat philosophical character of this paradox, its history is marked by an amount of controversy unusual for ordinary mathematics. Before the reader hastes to propose a solution, let us consider a less trivial example, due to L. A. Levin. Suppose that in some country, the share of votes for the ruling party in 30 consecutive elections formed a sequence 0.99F7 where for every even 7, the num- ber F7 is the 7-th digit of c = 3.1415 .... Though many of us would feel that the election results were manipulated, it turns out to be surprisingly difficult to prove this by referring to some general principle. In a sequence of < fair elections, every sequence l of < digits has approxi- mately the probability & l = 10 < to appear as the actual sequence of third < ( ) − digits. Let us fix <. We are given a particular sequence l and want to test the validity of the government’s claim that the elections were fair. We interpret the assertion “l is random with respect to &<” as a synonym for “there is no reason to challenge the claim that l arose from the distribution &<”. How can such a claim be challenged at all? The government, just like the weather forecaster who announces 30% chance of rain, does not guarantee any particular set of outcomes. However, to stand behind its claim, it must agree to any bet based on the announced distribution. Let us call a payoff function with respect the distribution % any nonnegative function B l with % l B l ( ) l ( ) ( ) ≤ 1. If a “nonprofit” gambling casino asks 1 dollar for a game and claims that each outcome has probability % l then it must agree to pay B l dollarsÍ on outcome ( ) ( ) l. We would propose to the government the following payoff function B0 with respect to & : let B l = 10< 2 for all sequences l whose even digits are given < 0 ( ) / by c, and 0 otherwise. This bet would cost the government 10< 2 1 dollars. / − Unfortunately, we must propose the bet before the elections take place and it is unlikely that we would have come up exactly with the payoff function B0. Is then the suspicion unjustifiable? No. Though the function B0 is not as natural as to guess it in advance, it is still highly “regular”. And already Laplace assumed in [15] that the number of “regular” bets is so small we can afford to make them all in advance and still win by a wide margin. Kolmogorov discovered in [27] and [29] (probably without knowing about [15] but following a four decades long controversy on von Mises’ con- cept of randomness, see [53]) that to make this approach work we must define “regular” or “simple” as “having a short description” (in some formal sense to 2 1.1. Introduction be specified below). There cannot be many objects having a short description because there are not many short strings of symbols to be used as descriptions. We thus come to the principle saying that on a random outcome, all suffi- ciently simple payoff functions must take small values. It turns out below that this can be replaced by the more elegant principle saying that a random out- come itself is not too simple. If descriptions are written in a 2-letter alphabet then a typical sequence of < digits takes < log 10 letters to describe (if not stated otherwise, all logarithms in these notes are to the base 2). The digits of c can be generated by an algorithm whose description takes up only some constant length. Therefore the sequence F1 ...F< above can be described with approx- imately < 2 log 10 letters, since every other digit comes from c. It is thus ( / ) significantly simpler than a typical sequence and can be declared nonrandom. 1.1.1 Formal results The theory of randomness is more impressive for infinite sequences than for finite ones, since sharp distinction can be made between random and nonrandom in- finite sequences. For technical simplicity, first we will confine ourselves to finite sequences, especially a discrete sample space Ω, which we identify with the set of natural numbers.