Lecture Notes on Descriptional Complexity and Randomness

Home , Leonid Levin

arXiv:2105.04704v1 [cs.IT] 10 May 2021 etr oe ndsrpinlcmlxt and complexity descriptional on notes Lecture rdc oeo h antcnqe n ocpso h ﬁeld. the of concepts and backgr techniques and main history the motivation, of on some short troduce are notes These ory. Vrinhsoy eo osewa a hne when. changed has what see to below history” “Version iatclsre ftefudtoso loihi Infor Algorithmic of foundations the of survey didactical A h mnsrp”hsbe vligoe h er.Pes,l Please, years. the over evolving been has “manuscript” The otnUniversity Boston randomness [email protected] ee Gács Peter udbtin- but ound ainThe- mation o at ook

Contents

Contents iii

1 Complexity 1 1.1 Introduction ...... 1 1.1.1 Formalresults ...... 3 1.1.2 Applications ...... 6 1.1.3 Historyoftheproblem ...... 8 1.2 Notation ...... 10 1.3 Kolmogorovcomplexity ...... 11 1.3.1 Invariance ...... 11 1.3.2 Simplequantitativeestimates ...... 14 1.4 Simple properties of information ...... 16 1.5 Algorithmic properties of complexity ...... 19 1.6 Thecodingtheorem ...... 24 1.6.1 Self-delimitingcomplexity ...... 24 1.6.2 Universalsemimeasure ...... 27 1.6.3 Preﬁxcodes ...... 28 1.6.4 The coding theorem for F ...... 30 ( ) 1.6.5 Algorithmicprobability...... 31 1.7 Thestatisticsofdescriptionlength ...... 32

2 Randomness 37 2.1 Uniformdistribution ...... 37 2.2 Computabledistributions...... 40 2.2.1 Twokindsoftest...... 40 2.2.2 Randomnessviacomplexity ...... 41 2.2.3 Conservationofrandomness ...... 43 2.3 Inﬁnitesequences ...... 46

iii Contents

2.3.1 Nullsets ...... 47 2.3.2 Probabilityspace ...... 52 2.3.3 Computability ...... 54 2.3.4 Integral...... 54 2.3.5 Randomnesstests ...... 55 2.3.6 Randomnessandcomplexity ...... 56 2.3.7 Universal semimeasure, algorithmic probability . . . .. 58 2.3.8 Randomness via algorithmic probability ...... 60

3 Information 63 3.1 Information-theoreticrelations...... 63 3.1.1 Theinformation-theoreticidentity ...... 63 3.1.2 Informationnon-increase...... 72 3.2 The complexity of decidable and enumerable sets ...... 74 3.3 Thecomplexityofcomplexity ...... 78 3.3.1 Complexityissometimescomplex ...... 78 3.3.2 Complexityisrarelycomplex ...... 79

4 Generalizations 83 4.1 Continuous spaces, noncomputable measures ...... 83 4.1.1 Introduction ...... 83 4.1.2 Uniformtests...... 85 4.1.3 Sequences ...... 87 4.1.4 Conservationofrandomness ...... 88 4.2 Testforaclassofmeasures ...... 90 4.2.1 Fromauniformtest ...... 90 4.2.2 Typicalityandclasstests ...... 91 4.2.3 Martin-Löf’sapproach ...... 94 4.3 Neutralmeasure ...... 97 4.4 Monotonicity, quasi-convexity/concavity ...... 101 4.5 Algorithmicentropy ...... 103 4.5.1 Entropy...... 104 4.5.2 Algorithmicentropy ...... 105 4.5.3 Additiontheorem...... 106 4.5.4 Information...... 110 4.6 Randomnessandcomplexity...... 111 4.6.1 Discretespace ...... 111 4.6.2 Non-discretespaces ...... 113

iv Contents

4.6.3 Inﬁnitesequences ...... 114 4.6.4 Bernoullitests ...... 116 4.7 Cells ...... 118 4.7.1 Partitions ...... 119 4.7.2 Computableprobabilityspaces ...... 122

5 Exercises and problems 125

A Background from mathematics 131 A.1 Topology ...... 131 A.1.1 Topologicalspaces ...... 131 A.1.2 Continuity ...... 133 A.1.3 Semicontinuity ...... 134 A.1.4 Compactness ...... 135 A.1.5 Metricspaces...... 136 A.2 Measures...... 141 A.2.1 Setalgebras ...... 141 A.2.2 Measures ...... 141 A.2.3 Integral...... 144 A.2.4 Density...... 145 A.2.5 Randomtransitions...... 146 A.2.6 Probability measures over a metric space ...... 147

B Constructivity 155 B.1 Computabletopology ...... 155 B.1.1 Representations ...... 155 B.1.2 Constructivetopologicalspace ...... 156 B.1.3 Computablefunctions ...... 158 B.1.4 Computableelementsandsequences ...... 159 B.1.5 Semicomputability ...... 160 B.1.6 Eﬀectivecompactness ...... 161 B.1.7 Computablemetricspace...... 163 B.2 Constructivemeasuretheory ...... 166 B.2.1 Spaceofmeasures ...... 166 B.2.2 Computable and semicomputable measures ...... 168 B.2.3 Randomtransitions...... 169

Bibliography 171

v Contents

Version history

May 2021: changed the notation to the now widely accepted one: Kolmogorov’s original complexity is denoted F (instead of the earlier F ), while the pre- ( ) ( ) fix complexity is denoted F instead of the earlier F . ( ) ( ) June 2013: corrected a slight mistake in the section on the section on randomness via algorithmic probability. February 2010: chapters introduced, and recast using the memoir class. April 2009: besides various corrections, a section is added on infinite sequences. This is not new material, just places the most classical results on randomness of infinite sequences before the more abstract theory. January 2008: major rewrite. • Added formal definitions throughout. • Made corrections in the part on uniform tests and generalized complexity, based on remarks of Hoyrup, Rojas and Shen. • Rearranged material. • Incorporated material on uniform tests from the work of Hoyrup-Rojas. • Added material on class tests.

vi 1 Complexity

Je n’ai fait celle-ci plus longue que parce que je n’ai pas eu le loisir de la faire plus courte.

Pascal

Ainsi, au jeu de croix ou pile, l’arrivée de croix cent fois de suite, nous paraît extraordinaire; parce que le nombre presque inﬁni des combinaisons qui peuvent arriver en cent coups étant partagé en séries regulières, ou dans lesquelles nous voyons régner un ordre facile à saisir, et en séries irregulières; celles-ci sont incomparablement plus nombreuses.

Laplace

1.1 Introduction

The present section can be read as an independent survey on the problems of randomness. It serves as some motivation for the dryer stuﬀ to follow. If we toss a coin 100 times and it shows each time Head, we feel lifted to a land of wonders, like Rosencrantz and Guildenstern in [49]. The argument that 100 heads are just as probable as any other outcome convinces us only that the axioms of Probability Theory, as developed in [28], do not solve

1 1. Complexity all mysteries they are sometimes supposed to. We feel that the sequence consisting of 100 heads is not random, though others with the same probability are. Due to the somewhat philosophical character of this paradox, its history is marked by an amount of controversy unusual for ordinary mathematics. Before the reader hastes to propose a solution, let us consider a less trivial example, due to L. A. Levin. Suppose that in some country, the share of votes for the ruling party in 30 consecutive elections formed a sequence 0.99F7 where for every even 7, the number F7 is the 7-th digit of c = 3.1415 .... Though many of us would feel that the election results were manipulated, it turns out to be surprisingly difficult to prove this by referring to some general principle. In a sequence of < fair elections, every sequence l of < digits has approximately the probability & l = 10 < to appear as the actual sequence of third < ( ) − digits. Let us fix <. We are given a particular sequence l and want to test the validity of the government’s claim that the elections were fair. We interpret the assertion “l is random with respect to &<” as a synonym for “there is no reason to challenge the claim that l arose from the distribution &<”. How can such a claim be challenged at all? The government, just like the weather forecaster who announces 30% chance of rain, does not guarantee any particular set of outcomes. However, to stand behind its claim, it must agree to any bet based on the announced distribution. Let us call a payoff function with respect the distribution % any nonnegative function B l with % l B l ( ) l ( ) ( ) ≤ 1. If a “nonprofit” gambling casino asks 1 dollar for a game and claims that each outcome has probability % l then it must agree to pay B l dollarsÍ on outcome ( ) ( ) l. We would propose to the government the following payoff function B0 with respect to & : let B l = 10< 2 for all sequences l whose even digits are given < 0 ( ) / by c, and 0 otherwise. This bet would cost the government 10< 2 1 dollars. / − Unfortunately, we must propose the bet before the elections take place and it is unlikely that we would have come up exactly with the payoff function B0. Is then the suspicion unjustifiable? No. Though the function B0 is not as natural as to guess it in advance, it is still highly “regular”. And already Laplace assumed in [15] that the number of “regular” bets is so small we can afford to make them all in advance and still win by a wide margin. Kolmogorov discovered in [27] and [29] (probably without knowing about [15] but following a four decades long controversy on von Mises’ concept of randomness, see [53]) that to make this approach work we must define “regular” or “simple” as “having a short description” (in some formal sense to

2 1.1. Introduction be specified below). There cannot be many objects having a short description because there are not many short strings of symbols to be used as descriptions. We thus come to the principle saying that on a random outcome, all sufficiently simple payoff functions must take small values. It turns out below that this can be replaced by the more elegant principle saying that a random outcome itself is not too simple. If descriptions are written in a 2-letter alphabet then a typical sequence of < digits takes < log 10 letters to describe (if not stated otherwise, all logarithms in these notes are to the base 2). The digits of c can be generated by an algorithm whose description takes up only some constant length. Therefore the sequence F1 ...F< above can be described with approximately < 2 log 10 letters, since every other digit comes from c. It is thus ( / ) significantly simpler than a typical sequence and can be declared nonrandom.

1.1.1 Formal results The theory of randomness is more impressive for infinite sequences than for finite ones, since sharp distinction can be made between random and nonrandom infinite sequences. For technical simplicity, first we will confine ourselves to finite sequences, especially a discrete sample space Ω, which we identify with the set of natural numbers. Binary strings will be identified with the natural numbers they denote. Definition 1.1.1 A Turing machine is an imaginary computing device consisting of the following. A control state belonging to a finite set of possible control states. A fixed number of infinite (or infinitely extendable) strings of cells called tapes. Each cell contains a symbol belonging to a finite tape alphabet . On each tape, there is a read-write head observing one of the tape cells. The machine’s configuration (global state) is determined at each instant by giving its control state, the contents of the tapes and the positions of the read-write heads. The “hardware program” of the machine determines its configurationin the next step as a functon of the control state and the tape symbols observed. It can change the control state, the content of the observed cells and the position of the read- write heads (the latter by one step to the left or right). Turing machines can emulate computers of any known design (see for example [60]). The tapes are used for storing programs, input data, output and as memory. y The Turing machines that we will use to interpret descriptions will be somewhat restricted. Definition 1.1.2 Consider a Turing machine which from a binary string >

3 1. Complexity and a natural number F computes the output >, F (if anything at all). We ( ) will say that interprets > as a description of >, F in the presence of the side ( ) information F. We suppose that if >, F is defined then ?,F is not defined ( ) ( ) for any prefix ? of >. Such machines are called self-delimiting (s.d.). The conditional complexity F G of the number F with respect to the ( | ) number G is the length of the shortest description > for which >, G = F. y ( ) Kolmogorov and Solomonoff observed that the function F G depends ( | ) only weakly on the machine , because there are universal Turing machines capable of simulating the work of any other Turing machine whose description is supplied. More formally, the following theorem holds: Theorem 1.1.1 (Invariance Theorem) There is a s.d. Turnig machine ) such that for any s.d. machine a constant 2 exists such that for all F,G we have F G F G 2 . ) ( | ) ≤ ( | )+ This theorem motivates the following definition. Definition 1.1.3 Let us fix ) and define F G = F G and F = ( | ) ) ( | ) ( ) F 0 . y ( | ) The function F G is not computable. We can compute a nonincreas- ( | ) ing, convergent sequence of approximations to F (it is semicomputable from ( ) above), but will not know how far to go in this sequence for some prescribed accuracy. If F is a binary string of length < then F < 2 log < 2 for some ( ) ≤ + + constant 2. The description of F with this length gives F bit-for-bit, along with some information of length 2 log < which helps the s.d. machine find the end of the description. For most binary strings of lenght <, no significantly shorter description exists, since the number of short descriptions is small. Below, this example is generalized and sharpened for the case when instead of counting the number of simple sequences, we measure their probability.

Definition 1.1.4 Denote by F∗ the first one among the shortest descriptions of F. y The correspondence F F is a code in which no codeword is the prefix → ∗ of another one. This implies by an argument well-known in Information Theory the inequality F G 2− ( | ) 1, (1.1.1) F ≤ Õ hence only a few objects F can have small complexity. The converse of the same argument goes as follows. Let ` be a computable probability distribution, one for

4 1.1. Introduction which there is a binary program computing ` l for each l to any given degree ( ) of accuracy. Let ` be the length of the shortest one of these programs. Then ( ) l log ` l ` 2. (1.1.2) ( )≤− ( )+ ( )+ Here 2 is a universal constant. These two inequalities are the key to the estimates of complexity and the characterization of randomness by complexity. Denote 3 l = log ` l l . ` ( ) − ( )− ( ) Inequality (1.1.1) implies 3 l B l = 2 ` ( ) ` ( ) can be viewed as a payoﬀ function. Now we are in a position to solve the election paradox. We propose the payoﬀ function

log & l l < 2− < ( )− ( | ) to the government. (We used the conditional complexity l < because the ( | ) uniform distribution &< depends on <.) If every other digit of the outcome F B F comes from c then F < < 2 log 10 20 hence we win a sum 2 ( ) < 2 ( | ) ≤ ( / ) + ≥ 2110 / from the government (for some constants 20, 21 > 0), even though the bet does not contain any reference to the number c. The fact that B l is a payoﬀ function implies by Markov’s Inequality for ` ( ) any 9 > 0 9 ` l : l < log ` l 9 < 2− . (1.1.3) { ( ) − ( )− } Inequalities (1.1.2) and (1.1.3) say that with large probability, the complexity l of a random outcome l is close to its upper bound log ` l ` . This ( ) − ( )+ ( ) law occupies distinguished place among the “laws of probability”, because if the outcome l violates any such law, the complexity falls far below the upper bound. Indeed, a proof of some “law of probability” (like the law of large numbers, the law of iterated logarithm, etc.) always gives rise to some simple computable payoﬀ function B l taking large values on the outcomes violating the law, just ( ) as in the election example. Let ; be some large number, suppose that B l has ( ) complexity < ; 2, and that B l > 2;. Then inequality (1.1.2) can be applied / ( 0) to a l = ` l B l , and we get ( ) ( ) ( ) l log ` l ; a 2 ( )≤− ( )− + ( )+ 0 log ` l ; 2 ` 2 ≤ − ( )− / + ( )+ 1 for some constants 20, 21.

5 1. Complexity

More generally, the payoff function B l is maximal (up to a multiplicative ` ( ) constant) among all payoff functions that are semicomputable (from below). Hence the quantity log ` l l is a universal test of randomness. Its − ( ) − ( ) value measures the deficiency of randomness in the outcome l with respect to the distribution `, or the extent of justified suspicion against the hypothesis ` given the outcome l.

1.1.2 Applications Algorithmic Information Theory (AIT) justifies the intuition of random sequences as nonstandard analysis justifies infinitely small quantities. Any statement of classical probability theory is provable without the notion of randomness, but some of them are easier to find using this notion. Due to the incomputability of the universal randomness test, only its approximations can be used in practice. Pseudorandom sequences are sequences generated by some algorithm, with some randomness properties with respect to the coin-tossing distribution. They have very low complexity (depending on the strength of the tests they withstand, see for example [14]), hence are not random. Useful pseudorandom sequences can be defined using the notion of computational complexity, for example the number of steps needed by a Turing machine to compute a certain function. The existence of such sequences can be proved using some difficult unproven (but plausible) assumptions of computation theory. See [6], [59], [24], [34].

Inductive inference The incomputable “distribution” m l = 2 l has the remarkable property ( ) − ( ) that, the test 3 l m , shows all outcomes l “random” with respect to it. Rela- ( | ) tions (1.1.2) and (1.1.3) can be read as saying that if the real distribution is ` then ` l and m l are close to each other with large probability. Therefore if ( ) ( ) we know that l comes from some unknown simple distribution ` then we can use m l as an estimate of ` l . This suggests to call m the “apriori proba- ( ) ( ) bility” (but we will not use this term much). The randomness test 3 l can ` ( ) be interpreted in the framework of hypothesis testing: it is the likelihood ratio between the hypothesis ` and the ﬁxed alternative hypothesis m. In ordinary statistical hypothesis testing, some properties of the unknown distribution ` are taken for granted. The sample l is generally a large independent sample: ` is supposed to be a product distribution. Under these conditions, the universal test could probably be reduced to some of the tests used in statistical practice. However, these conditions do not hold in other applications: for

6 1.1. Introduction example testing for independence in a proposed random sequence, predicting some time series of economics, or pattern recognition. If the apriori probability m is a good estimate of the actual probability then we can use the conditional apriori probability for prediction, without reference to the unknown distribution `. For this purpose, one first has to define apriori probability for the set of infinite sequences of natural numbers, as done in [61]. We denote this function by ". For any finite sequence F, the number " F is ( ) the apriori probability that the outcome is some extension of F. Let F,G be finite sequences. The formula " FG ( ) (1.1.4) " F ( ) is an estimate of the conditional probability that the next terms of the outcome will be given by G provided that the first terms are given by F. It converges to the actual conditional probability ` FG ` F with `-probability 1 for any ( )/ ( ) computable distribution ` (see for example [46]). To understand the surprising generality of the formula (1.1.4), suppose that some infinite sequence H 7 is ( ) given in the following way. Its even terms are the subsequent digits of c, its odd terms are uniformly distributed, independently drawn random digits. Let H 1 : < = H 1 H < . ( ) ( )··· ( ) Then " H 1 : 27 0 " H 1: 27 converges to 0.1 for 0 = 0,..., 9, while ( ( ) / ( ( )) " H 1:27 1 0 " H 1:27 1 converges to 1 if 0 is the 7-th digit of c, and ( ( + ) )/ ( ( + )) to 0 otherwise. The inductive inference formula using conditional apriori probability can be viewed as a mathematical form of “Occam’s Razor”: the advice to predict by the simplest rule fitting the data. It can also viewed as a realization of Bayes’Rule, with a universally applicable apriori distribution. Since the distribution " is incomputable, we view the main open problem of inductive inference to find maximally efficient approximations to it. Sometimes, even a simple approximation gives nontrivial results (see [3]).

Information theory Since with large probability, l is close to log ` l , the entropy ( ) − ( ) ` l log ` l of the distribution ` is close to the average complexity − l ( ) ( ) ` l l . The complexity F of an object F can indeed be interpreted l ( ) ( ) ( ) asÍ the distribution-free deﬁnition of information content. The correspondence ÍF F is a sort of universal code: its average (even individual) “rate”, or code- 7→ ∗ word length is almost equally good for any simple computable distribution.

7 1. Complexity

It is of great conceptual help to students of statistical physics that entropy can be defined now not only for ensembles (probability distributions), but for individual states of a physical system. The notion of an individual incompressible sequence, a sequence whose complexity is maximal, proved also extremely useful in finding information-theoretic lower bounds on the computing speed of certain Turing machines (see [39]). The conditional complexity F G obeys identities analogous to the ( | ) information-theoretical identities for conditional entropy, but these identities are less trivial to prove in AIT. The information F : G = F G F,G has ( ) ( )+ ( )− ( ) several interpretations. It is known that F : G is equal, to within an additive ( ) constant, to G G F , the amount by which the object F (as defined in ( )− ( | ∗) ∗ the previous section) decreases our uncertainty about G. But it can be written as log m2 F,G F,G = 3 F,G m2 where m2 = m m is the product − ( )− ( ) (( ) | ) × distribution of m with itself. It is thus the deficiency of randomness of the pair F,G with respect to this product distribution. Since any object is random with ( ) respect to m, we can view the randomness of the pair F,G with respect to the ( ) product m2 as the independence of F and G from each other. Thus “information” measures the “deficiency of independence”.

Logic Some theorems of Mathematical Logic (in particular, Gödel’s theorem) have a strong quantitative form in AIT, with new philosophical implications (see [8], [9], [31], [33]). Levin based a new system of intuitionistic analysis on his Inde- pendence Principle (see below) in [33].

1.1.3 History of the problem P. S. Laplace thought that the number of “regular” sequences (whatever “regular” means) is much smaller than the number of irregular ones (see [15]). In the first attempt at formalization hundred years later, R. von Mises defined an infinite binary sequence as random (a “Kollektiv”) if the relative frequencies converge in any subsequence selected according to some (non-anticipating) “rule” (whatever “rule” means, see [53]). As pointed out by A. Wald and others, Mises’s definitions are sound only if a countable set of possible rules is fixed. The logician A. Church, in accordance with his famous thesis, proposed to understand “rule” here as “recursive (computable) function”. The Mises selection rules can be considered as special randomness tests. In the ingenious work [52], J. Ville proved that they do not capture all relevant

8 1.1. Introduction properties of random sequences. In particular, a Kollektiv can violate the law of iterated logarithm. He proposed to consider arbitrary payoff functions (a countable set of them), as defined on the set of infinite sequences—these are more commonly known as martingales. For the solution of the problem of inductive inference, R. Solomonoff introduced complexity and apriori probability in [45] and proved the Invariance The- orem. A. N. Kolmogorov independently introduced complexity as a measure of individual information content and randomness, and proved the Invariance The- orem (see [27] and [29]). The incomputability properties of complexity have noteworthy philosophical implications (see [4], [8], [9]. P. Martin-Löf defined in [38] randomness for infinite sequences. His concept is essentially the synthesis of Ville and Church (as noticed in [41]). He recognized the existence of a universal test, and pointed out the close connection between the randomness of an infinite sequence and the complexity of its initial segments. L. A. Levin defined the apriori probability " as a maximal (to within a multiplicative constant) semicomputable measure. With the help of a modi- fied complexity definition, he gave a simple and general characterization of random sequences by the behavior of the complexity of their initial segments (see [61], [30]). In [17] and [31], the information-theoretical properties of the self-delimiting complexity (as defined above) are exactly described. See also [10], [42] and [58]. In [33], Levin defined a deficiency of randomness 3 l ` in a uniform man- ( | ) ner for all (computable or incomputable) measures `. He proved that all outcomes are random with respect to the apriori probability ". In this and earlier papers, he also proved the Law of Information Conservation, stating that the information U; V in a sequence U about a sequence V cannot be significantly ( ) increased by any algorithmic processing of U (even using random number generators). He derived this law from a so-called Law of Randomness Conservation via the definition of information U; V as deficiency of randomness with respect to ( ) the product distribution "2. Levin suggested the Independence Principle saying that any sequence U arising in nature contains only finite information U; V ( ) about any sequence V defined by mathematical means. With this principle, he showed that the use of more powerful notions of definability in randomness tests (or the notion of complexity) does not lead to fewer random sequences among those arising in nature. The monograph [16] is very useful as background information on the various attempts in this century at solving the paradoxes of probability theory. The

9 1. Complexity work [33] is quite comprehensive but very dense; it can be recommended only to devoted readers. The work [61] is comprehensive and readable but not quite up-to-date. The surveys [12], [41] and [43] can be used to complement it. The most up-to-date and complete survey, which subsumes most of these notes, is [36]. AIT created many interesting problems of its own. See for example [10], [11], [17], [18], [19], [20], [35], [33], [37], [41], [44], [47], and the technically diﬃcult results [48] and [55].

Acknowledgment The form of exposition of the results in these notes and the general point of view represented in them were greatly inﬂuenced by Leonid Levin. More recent com- munication with Paul Vitányi, Mathieu Hoyrup, Cristóbal Rojas and Alexander Shen has also been very important.

1.2 Notation

When not stated otherwise, log means base 2 logarithm. The cardinality of a set will be denoted by . (Occasionally there will be inconsistency, sometimes | | denoting it by , sometimes by #.) If is a set then 1 F is its indicator | | ( ) function:

1 if F , 1 F = ∈ ( ) (0 otherwise.

The empty string is denoted by Λ. The set ∗ is the set of all finite strings of elements of , including the empty string. Let : F denote the length of ( ) string F. (Occasionally there will be inconsistency, sometimes denoting it by F , sometimes by : F ). For sequences F and G, let F G denote that F is a | | ( ) ⊑ prefix of G. For a string F and a (finite or infinite) sequence G, we denote their concatenation by FG. For a sequence F and < : F , the <-th element of F is ≤ ( ) F < , and ( ) F 7 : 8 = F 7 F 8 . ( ) ( )··· ( ) Sometimes we will also write

< F ≤ = F 1 : < . ( )

10 1.3. Kolmogorov complexity

The string F F will sometimes be written also as F ,...,F . For natural 1 · · · < ( 1 <) number ; let V ; be the binary sequence denoting ; in the binary notation. ( ) We denote by - N the set of all inﬁnite sequences of elements of -. The sets of natural numbers, integers, rational numbers, real numbers and complex numbers will be denoted respectively by N, Z, Q, R, C. The set of nonnegative real numbers will be denoted by R . The set of real numbers with + , added (with the appropriate topology making it compact) will be de- −∞ ∞ noted by R. Let

S = 0,...,@ 1 ∗, S = N∗, B = 0, 1 . @ { − } { } We use and to denote min and max, further ∧ ∨

F + = F 0, F − = F + | | ∨ | | |− | for real numbers F. Let be some standard one-to-one encoding of N to N, with partial in- h·i ∗ verses where F = F 7 for 7 : F . For example, we can have [·]7 [h i]7 ( ) ≤ ( ) 1 7, 8 = 7 1 7 8 1 8, <1,...,<9 1 = <1,...,<9 ,<9 1 . h i 2 ( + )( + + )+ h + i hh i + i We will use , in particular as a pairing function over N. Similar pairing h· ·i functions will be assumed over other domains, and they will also be denoted by , . h· ·i Another use of the notation , may arise when the usual notation F,G (· ·) ( ) of an ordered pair of real numbers could be confused with the same notation of an open interval. In this case, we will use F,G to denote the pair. ( ) The relations 5 <6,+ 5 <∗ 6 mean inequality to within an additive constant and multiplicative constant respectively. The ﬁrst is equivalent to 5 6 $ 1 , the second to 5 = $ 6 . The ≤ + ( ) ( ) relation 5 =∗ 6 means 5 <∗ 6 and 5 >∗ 6.

1.3 Kolmogorov complexity

1.3.1 Invariance It is natural to try to deﬁne the information content of some text as the size of the smallest string (code) from which it can be reproduced by some decoder,

11 1. Complexity interpreter. We do not want too much information “to be hidden in the decoder” we want it to be a “simple” function. Therefore, unless stated otherwise, we require that our interpreters be computable, that is partial recursive functions. Definition 1.3.1 A partial recursive function from S S to S will be called 2 × a (binary) interpreter, or decoder. We will often refer to its first argument as the program, or code. y Partial recursive functions are relatively simple; on the other hand, the class of partial recursive functions has some convenient closure properties. Definition 1.3.2 For any binary interpreter and strings F,G S, the number ∈ F G = min : > : >, G = F ( | ) { ( ) ( ) } is called the conditional Kolmogorov-complexity of F given G, with respect to the interpreter . (If the set after the “min” is empty, then the minimum is ). We ∞ write F = F Λ . y ( ) ( | ) The number F measures the length of the shortest description for F when ( ) the algorithm is used to interpret the descriptions. The value F depends, of course, on the underlying function . But, ( ) as Theorem 1.3.1 shows, if we restrict ourselves to sufficiently powerful interpreters , then switching between them can change the complexity function only by amounts bounded by some additive constant. Therefore complexity can be considered an intrinsic characteristic of finite objects. Definition 1.3.3 A binary p.r. interpreter * is called optimal if for any binary p.r. interpreter there is a constant 2 < such that for all F,G we have ∞ F G F G 2. (1.3.1) * ( | ) ≤ ( | )+ y

Theorem 1.3.1 (Invariance Theorem) There is an optimal p.r.binary interpreter.

Proof. The idea is to use interpreters that come from universal partial recursive functions. However, not all such functions can be used for universal interpreters. Let us introduce an appropriate pairing function. For F B<, let ∈ F= = F 1 0F 2 0 ...F < 1 0F < 1 ( ) ( ) ( − ) ( ) where F= = F1 for : F = 1. Any binary sequence F can be uniquely represented ( ) in the form > = 0=1.

12 1.3. Kolmogorov complexity

We know there is a p.r. function + : S S S S that is universal: for 2 × 2 × → any p.r. binary interpreter , there is a string 0 such that for all >, F, we have >, F = + 0, >,F . Let us deﬁne the function * >, F as follows. We represent ( ) ( ) ( ) the string > in the form > = C={ and deﬁne * >, F = + C,{,F . Then * is a ( ) ( ) p.r. interpreter. Let us verify that it is optimal. Let be a p.r. binary interpreter, 0 a binary string such that >, F = * 0, >,F for all >, F. Let F,G be two strings. ( ) ( ) If F G = , then (1.3.1) holds trivially. Otherwise, let > be a binary string ( | ) ∞ of length F G with >, G = F. Then we have ( | ) ( ) * 0= >, G = + 0, >, G = >, G = F, ( ) ( ) ( ) and F G 2: 0 F G . * ( | ) ≤ ( )+ ( | )

The constant 2: 0 is in the above proof a bound on the complexity of de- ( ) scription of the interpreter for the optimal interpreter *. Let us note that for 1 2 any two optimal interpreters * ( ), * ( ), there is a constant 2 such that for all F,G, we have

1 F G 2 F G < 2. (1.3.2) | * ( ) ( | )− * ( ) ( | )| Hence the complexity F of description of an object F does not depend strongly * ( ) on the interpreter *. Still, for every string F, there is an optimal interpreter * with F = 0. Imposing a universal bound on the table size of the Turing * ( ) machines used to implement optimal interpreters, we can get a universal bound on the constants in (1.3.2). The theorem motivates the following definition. Definition 1.3.4 We fix an optimal binary p.r. interpreter * and write F G ( | ) for F G . y * ( | ) Theorem 1.3.1 (as well as other invariance theorems) is used in AIT for much more than just to show that F is a proper concept. It is the principal tool to ( ) find upper bounds on F ; this is why most such upper bounds are proved to ( ) hold only to within an additive constant. The optimal interpreter * >, F defined in the proof of Theorem 1.3.1 is obvi- ( ) ously a universal partial recursive function. Because of its convenient properties we will use it from now on as our standard universal p.r. function, and we will refer to an arbitrary p.r. function as * F = * >, F . > ( ) ( )

13 1. Complexity

Definition 1.3.5 We define * F ,...,F = * F ...,F . Similarly, we > ( 1 9) > (h 1 9i) will refer to an arbitrary computably enumerable set as the range ,> of some * . We often do not need the second argument of the function * >, F . We > ( ) therefore define * > = * >, Λ . ( ) ( ) y

It is of no consequence that we chose binary strings as as descriptions. It is rather easy to define, (Exercise) for any two natural numbers @, A, a standard @ encoding cnvA of base @ strings F into base A strings with the property log @ : cnv@ F : F 1. ( A ( )) ≤ ( ) log A + Now, with @-ary strings as descriptions, we must define F as the minimum ( ) of : > log @ over all programs > in S with > = F. The equivalence of the ( ) @ ( ) definitions for different values of @ is left as an exercise.

1.3.2 Simple quantitative estimates We found it meaningful to speak about the information content, complexity of a ﬁnite object. But how to compute this quantity? It turns out that complexity is not a computable function, but much is known about its statistical behavior. The following notation will be useful in the future.

Deﬁnition 1.3.6 The relation 5 <+ 6 means inequality to within an additive constant, that there is a constant 2 such that for all F, 5 F 6 F . We can ( ) ≤ ( ) write this also as 5 6 $ 1 . We also say that 6 additively dominates 5 . The ≤ + ( ) relation 5 =+ 6 means 5 <+ 6 and 5 >+ 6. The relation 5 <∗ 6 among nonnegative functions means inequality to within a multiplicative constant. It is equivalent to log 5 <+ log 6 and 5 = $ 6 . We also say that 6 multiplicatively dominates 5 . ( ) The relation 5 =∗ 6 means 5 <∗ 6 and 5 >∗ 6. y With the above notation, here are the simple properties. Theorem 1.3.2 The following simple upper bound holds. a) For any natural number ;, we have

; <+ log ;. (1.3.3) ( ) b) For any positive real number { and string G, every ﬁnite set of size ; has at least ; 1 2 { 1 elements F with F G log ; {. ( − − + ) ( | ) ≥ − 14 1.3. Kolmogorov complexity

Corollary 1.3.7 lim< < = . →∞ ( ) ∞ Theorem 1.3.2 suggests that if there are so few strings of low complexity, then < converges fast to . In fact this convergence is extremely slow, as ( ) ∞ shown in a later section. Proof of Theorem 1.3.2. First we prove (a). Let the interpreter be deﬁned such that if V ; = > then >, G = ;. Since V ; = log ; , we have ; <+ ( ) ( ) | ( )| ⌈ ⌉ ( ) ; < log ; 1. ( ) + Part (b) says that the trivial estimate (1.3.3) is quite sharp for most numbers under ;. The reason for this is that since there are only few short programs, there are only few objects of low complexity. For any string G, and any positive real natural number C, we have

C 1 F : F G C < 2 + . (1.3.4) |{ ( | ) ≤ }| To see this, let < = log C . The number 2< 1 1 of diﬀerent binary strings of ⌊ ⌋ + − length < is an upper bound on the number of diﬀerent shortest programs of ≤ length C. Now (b) follows immediately from (1.3.4). ≤ The three properties of complexity contained in Theorems 1.3.1 and 1.3.2 are the ones responsible for the applicabiliy of the complexity concept. Several important later results can be considered transformations, generalizations or analogons of these. Let us elaborate on the upper bound (1.3.3). For any string F S , we have ∈ @ F <+ < log @ 2 log @. (1.3.5) ( ) + In particular, for any binary string F, we have

F <+ : F . ( ) ( ) Indeed, let the interpreter be defined such that if > = V @ =cnv@ F then ( ) 2 ( ) >, G = F. We have F 2: V @ < log @ 1 < 2 log @ 3. Since ( ) ( ) ≤ ( ( )) + + ≤ ( + ) + F <+ F , we are done. ( ) ( ) We will have many more upper bound proofs of this form, and will not always explicitly write out the reference to F G <+ F G , needed for the last step. ( | ) ( | ) Apart from the conversion, inequality (1.3.5) says that since the beginning of a program can command the optimal interpreter to copy the rest, the complexity of a binary sequence F of length < is not larger than <. Inequality (1.3.5) contains the term 2 log @ instead of log @ since we have to apply the encoding |= to the string | = V @ ; otherwise the interpreter cannot ( ) 15 1. Complexity detach it from the program >. We could use the code V | =|, which is also a (| |) prefix code, since the first, detachable part of the codeword tells its length. For binary strings F, natural numbers < and real numbers C 1 we define ≥ C = C 2 log C, ( ) + ] F = V F =F, ( ) (| |) ] < = ] V < . ( ) ( ( )) We have : ] F <+ : F for binary sequences F, and : ] @ <+ log @ for ( ( )) ( ( )) ( ( )) ( ) numbers @. Therefore (1.3.5) is true with log @ in place of 2 log @. Of course, ( ) F could be replaced by still smaller functions, for example F log F . We ( ) + ( ) return to this topic later.

1.4 Simple properties of information

If we transform a string F by applying to it a p.r. function, then we cannot gain information over the amount contained in F plus the amount needed to describe the transformation. The string F becomes easier to describe, but it helps less in describing other strings G.

Theorem 1.4.1 For any partial recursive function *?, over strings, we have

* F G <+ F G : ? , ( ? ( ) | ) ( | )+ ( ( )) G * F >+ G F : ? . ( | ? ( )) ( | )− ( ( )) Proof. To define an interpreter with * F G <+ F G : ? , we ( ? ( ) | ) ( | ) + ( ( )) define ] ? >, G = * * >, G . To define an interpreter with G F <+ ( ( ) ) ? ( ( )) ( | ) G * F ? , we define ] ? >, F = * >, * F . ( | ? ( )) + ( ) ( ( ) ) ( ? ( )) The following notation is natural. Definition 1.4.1 The definition of conditional complexity is extended to pairs, etc. by

F ,...,F G ,...,G = F ,...,F G ,...,G . ( 1 ; | 1 <) (h 1 ;i | h 1

With the new notation, here are some new results.

16 1.4. Simple properties of information

Corollary 1.4.2 For any one-to-one p.r. function *>, we have

F * F <+ : > . | ( )− ( >( ))| ( ( )) Further,

F G, H <+ F * G , H : > , ( | ) ( | > ( ) )+ ( ( )) * F <+ F : > , ( > ( )) ( )+ ( ( )) F H <+ F,G H , (1.4.1) ( | ) ( | ) F G, H <+ F G , ( | ) ( | ) F,F =+ F , (1.4.2) ( ) ( ) F,G H =+ G,F H , ( | ) ( | ) F G, H =+ F H,G , ( | ) ( | ) F,G F, H =+ G F, H , ( | ) ( | ) F F, H =+ F F =+ 0. ( | ) ( | ) We made several times implicit use of a basic additivity property of complexity which makes it possible to estimate the joint complexity of a pair by the complexities of its constituents. As expressed in Theorem 1.4.2, it says essentially that to describe the pair of strings, it is enough to know the description of the first member and of a method to find the second member using our knowledge of the first one. Theorem 1.4.2 F,G <+ F G F . ( ) ( ( )) + ( | ) Proof. We define an interpreter as follows. We decompose any binary string > into the form > = ] | ?, and let >, H = * ?, * | . Then F,G is ( ) ( ) ( ( )) ( ) bounded, to within an additive constant, by the right-hand side of the above inequality.

Corollary 1.4.3 For any p.r. function *> over pairs of strings, we have

* F,G <+ F G F : > ( > ( )) ( ( )) + ( | )+ ( ( )) <+ F G : > , ( ( )) + ( )+ ( ( )) and in particular, F,G <+ F G . ( ) ( ( )) + ( )

17 1. Complexity

This corollary implies the following continuity property of the function < : ( ) for any natural numbers <, ℎ, we have

< ℎ < <+ log ℎ . (1.4.3) | ( + )− ( )| ( ) Indeed, < ℎ is a recursive function of < and ℎ. The term 2 log F making + ( ) up the diﬀerence between F and F in Theorem 1.4.2 is attributable to ( ) ( ( )) the fact that minimal descriptions cannot be concatenated without loosing an “endmarker”. It cannot be eliminated, since there is a constant 2 such that for all <, there are binary strings F,G of length < with ≤ F G log < < F,G 2. ( )+ ( )+ ( )+ Indeed, there are <2< pairs of binary strings whose sum of lengths is <. Hence by Theorem 1.3.2 b, there will be a pair F,G of binary strings whose sum of ( ) lengths is <, with F,G > < log < 1. For these strings, inequality (1.3.5) ( ) + − implies F G <+ : F : G . Hence F G log < <+ < log < < ( )+ ( ) ( )+ ( ) ( ) + ( )+ + F,G 1. ( )+ Regularities in a string will, in general, decrease its complexity radically. If the whole string F of length < is given by some rule, that is we have F 9 = * 9 , ( ) > ( ) for some recursive function *>, then

F <+ < : > . ( ) ( )+ ( ( )) Indeed, let us deﬁne the p.r. function + ?, 9 = * 1 ...* 9 . Then F = ( ) ? ( ) ? ( ) + >, < and the above estimate follows from Corollary 1.4.2. ( ) For another example, let F = G G G G G G , and G = G G G . Then 1 1 2 2 · · · < < 1 2 · · · < F =+ G even though the string F is twice longer. This follows from Corol- ( ) ( ) lary 1.4.2 since F and G can be obtained from each other by a simple recursive operation. Not all “regularities” decrease the complexity of a string, only those which distinguish it from the mass of all strings, putting it into some class which is both small and algorithmically deﬁnable. For a binary string of length <, it is nothing unusual to have its number of 0-s between < 2 √< and < 2. Therefore such / − / strings can have maximal or almost maximal complexity. If F has 9 zeroes then the inequality < F <+ log log < log 9 ( ) 9 + ( )+ ( ) follows from Theorems 1.3.2 and 1.4.3.

18 1.5. Algorithmic properties of complexity

Theorem 1.4.3 Let = ,> be an enumerable set of pairs of strings deﬁned enumerated with the help of the program > for example as

, = * >, F : F N . (1.4.4) > { ( ) ∈ } We define the section 0 = F : 0, F . (1.4.5) { h i ∈ } Then for all 0 and F 0, we have ∈ 0 F 0 <+ log : > . ( | ) | |+ ( ( )) Proof. We define an interpreter ?, 1 as follows. We decompose ? into ? = ( ) ] > V B . From a standard enumeration 0 9 ,F 9 9 = 1, 2,... of the set ( ) ( ) ( ( ) ( )) ( ) , , we produce an enumeration F 9, 1 9 = 1, 2,... , without repetition, of > ( ) ( ) the set ,1 for each 1, and define ?, 1 = F B, 1 . For this interpreter , we > ( ) ( ) get 9 F 0 <+ : > log 0 . ( | ) ( ( )) + | | It follows from Theorem 1.3.2 that whenever the set 0 is finite, the estimate of Theorem 1.4.3 is sharp for most of its elements F. The shortest descriptions of a string F seem to carry some extra information in them, above the description of F. This is suggested by the following strange identities.

Theorem 1.4.4 We have F, F =+ F . ( ( )) ( ) Proof. The inequality >+ follows from (1.4.1). To prove <+ , let > = * >, : > . ( ) h ( ( ))i Then F, F F . ( ( )) ≤ ( ) More generally, we have the following inequality. Theorem 1.4.5 G F, 7 G F,7 <+ G F,7 . ( | − ( | )) ( | ) The proof is exercise. The above inequalities are in some sense “pathological”, and do not necessarily hold for all “reasonable” deﬁnitions of descriptional complexity.

1.5 Algorithmic properties of complexity

The function F is not computable, as it will be shown in this section. However, ( ) it has a property closely related to computability. Let Q be the set of rational numbers.

19 1. Complexity

Deﬁnition 1.5.1 (Computability) Let 5 : S R be a function. It is computable → if there is a recursive function 6 F,< with rational values, and 5 F 6 F,< < ( ) | ( )− ( )| 1 <. y / Deﬁnition 1.5.2 (Semicomputability) A function 5 : S , is (lower) → (−∞ ∞] semicomputable if the set

F,@ : F S,@ Q,@<5 F { ( ) ∈ ∈ ( )} is recursively enumerable. A function 5 is called upper semicomputable if 5 is − lower semicomputable. y It is easy to show the following: • A function 5 : S R is lower semicomputable iﬀ there exists a recursive → function with rational values, (or, equivalently, a computable real function) 6 F,< nondecreasing in <, with 5 F = lim< 6 F,< . ( ) ( ) →∞ ( ) • A function 5 : S R is computable if it is both lower and upper semicom- → putable. The notion of semicomputibility is naturally extendable over other discrete domains, like S S. It is also useful to extend it to functions R R. Let I × → denote the set of open rational intervals of R, that is

V = >, ? : >, ? Q,>

1 5 − ?, = . (( ∞)) F Ø∈ y

20 1.5. Algorithmic properties of complexity

This says that if 5 F > @ then sooner or later we will find an interval F ( ) ∈ with the property that 5 H > ? for all H . ( ) ∈ The following facts are useful and simple to verify: Proposition 1.5.5 a) The function : R R is lower semicomputable. ⌈·⌉ → b) If 5 , 6 : R R are computable then their composition 5 6 is, too. → ( (·)) c) If 5 : R R is lower semicomputable and 6 : R R (or S R) is → → → computable then 5 6 is lower semicomputable. ( (·)) d) If 5 : R R is lower semicomputable and monotonic, and 6 : R R (or → → S R) is lower semicomputable then 5 6 is lower semicomputable. → ( (·)) Definition 1.5.6 We introduce a universal lower semicomputable function. For every binary string >, and F S, let ∈ ( F = sup G H : G, H /, H ≠ 0, F ,G,H , > ( ) { / ∈ hh i i ∈ > } where ,> was defined in (1.4.4). y The function ( F is lower semicomputable as a function of the pair >, F , > ( ) ( ) and for different values of >, it enumerates all semicomputable functions of F. Definition 1.5.7 Let ( F ,...,F = ( F ,...,F . For any lower semi- >( 1 9) >(h 1 9i) computable function 5 , we call any binary string > with (> = 5 a Gödel number of 5 . There is no universal computable function, but for any computable function 6, we call any number > , ? a Gödel number of 6 if 6 = ( = ( . y hh i h ii > − ? With this notion, we claim the following. Theorem 1.5.1 The function F G is upper semicomputable. ( | ) Proof. By Theorem 1.3.2 there is a constant 2 such that for any strings F,G, we have F G < log F 2. Let some Turing machine compute our optimal binary ( | ) h i+ p.r. interpreter. Let *B >, G be defined as * >, G if this machine, when started ( ) ( ) on input >, G , gives an output in B steps, undefined otherwise. Let B F G be ( ) ( | ) the smaller of log F 2 and h i+ min : > B : *B >, G = F . { ( ) ≤ ( ) } B B Then the function F G is computable, monotonic in B and limB F G = ( | ) →∞ ( | ) F G . ( | ) The function < is not computable. Moreover, it has no nontrivial partial ( ) recursive lower bound.

21 1. Complexity

Theorem 1.5.2 Let * < be a partial recursive function whose values are numbers > ( ) smaller than < whenever defined. Then ( ) * < <+ : > . > ( ) ( ( )) Proof. The proof of this theorem resembles “Berry’s paradox ”, which says: “The least number that cannot be defined in less than 100 words” is a definition of that number in 12 words. The paradox can be used to prove, whenever we agree on a formal notion of “definition”, that the sentence in quotes is not a definition. Let F >, 9 for 9 = 1, 2,... be an enumeration of the domain of * . For any ( ) > natural number ; in the range of * , let 5 >, ; be equal to the first F >, 9 > ( ) ( ) with * F >, 9 = ;. Then by definition, ; < 5 >, ; . On the other hand, > ( ( )) ( ( )) 5 >, ; is a p.r. function, hence applying Corollary 1.4.3 and (1.3.5) we get ( ) ; < 5 >, ; <+ > ; ( ( )) ( )+ ( ( )) <+ : > 1.5 log ;. ( )+

Church’s first example of an undecidable recursively enumerable set used universal partial recursive functions and the diagonal technique of Gödel and Cantor, which is in close connection to Russel’s paradox. The set he constructed is complete in the sense of “many-one reducibility”. The first example of sets not complete in this sense were Post’s simple sets. Definition 1.5.8 A computably enumerable set is simple if its complement is infinite but does not contain any infinite computably enumerable subsets. y The paradox used in the above proof is not equivalent in any trivial way to Russell’s paradox, since the undecidable computably enumerable sets we get from Theorem 1.5.2 are simple. Let 5 < log < be any recursive function ( ) ≤ with lim< 5 < = . We show that the set = < : < 5 < is →∞ ( ) ∞ { ( ) ≤ ( )} simple. It follows from Theorem 1.5.1 that is recursively enumerable, and from Theorem 1.3.2 that its complement is infinite. Let be any computably enumerable set disjoint from . The restriction 5 < of the function 5 < to is ( ) ( ) a p.r. lower bound for < . It follows from Theorem 1.5.2 that 5 is bounded. ( ) ( ) Since 5 < , this is possible only if is finite. ( )→∞ Gödel used the diagonal technique to prove the incompleteness of any sufficiently rich logical theory with a recursively enumerable axiom system. His technique provides for any sufficiently rich computably enumerable theory a concrete example of an undecidable sentence. Theorem 1.5.2 leads to a new proof of this

22 1.5. Algorithmic properties of complexity result, with essentially different examples of undecidable propositions. Let us consider any first-order language ! containing the language of standard first- order arithmetic. Let be a standard encoding of the formulas of this theory h·i into natural numbers. Let the computably enumerable theory )> be the theory for which the set of codes Φ of its axioms Φ is the computably enumerable set h i ,> of natural numbers. Corollary 1.5.9 There is a constant 2 < such that for any >, if any sentences ∞ with the meaning “; < < ” for ; > : > 2 are provable in theory ) then ( ) ( ( )) + > some of these sentences are false. Proof. For some >, suppose that all sentences “; < < ” provable in ) are ( ) > true. For a sentence Φ, let ) Φ denote that Φ is a theorem of the theory ) . > ⊢ > Let us define the function

>, < = max ; : ) “; < < ” . ( ) { > ⊢ ( ) } This function is semicomputable since the set >, Φ : ) Φ is recursively { ( h i) > ⊢ } enumerable. Therefore there is a binary string 0 such that >, < = ( 0, >, < . ( ) ( h i) It is easy to see that we have a constant string 1 such that

( 0, >, < = ( ?,< ( h i) ( ) where ? = 10>. (Formally, this statement follows for example from the so-called (< -theorem.) The function ( < is a lower bound on the complexity function ; ? ( ) < . By an immediate generalization of Theorem 1.5.2 for semicomputable ( ) functions), we have ( ?,< <+ : ? <+ : > . ( ) ( ( )) ( ( )) We have seen that the complexity function F G is not computable, and ( | ) does not have any nontrivial recursive lower bounds. It has many interesting upper bounds, however, for example the ones in Theorems 1.3.2 and 1.4.3. The following theorem characterizes all upper semicomputable upper bounds (majorants) of F G . ( | ) Theorem 1.5.3 (Levin) Let F,G be a function of strings semicomputable from ( ) above. The relation F G <+ F,G holds for all F,G if and only if we have ( | ) ( ) log F : F,G < ; <+ ; (1.5.1) |{ ( ) }| for all strings G and natural numbers ;.

Proof. Suppose that F G <+ F,G holds. Then (1.5.1) follows from b of ( | ) ( ) Theorem 1.3.2.

23 1. Complexity

Now suppose that (1.5.1) holds for all G,;. Then let be the computably enumerable set of triples F, G,; with F,G < ;. It follows from (1.5.1) ( ) ( ) and Theorem 1.4.3 that for all F, G,; we have F G,; <+ ;. By ( ) ∈ ( | ) Theorem 1.4.5 then F G <+ ;. ( | ) The minimum of any ﬁnite number of majorants is again a majorant. Thus, we can combine diﬀerent heuristics for recognizing patterns of sequences.

1.6 The coding theorem

1.6.1 Self-delimiting complexity Some theorems on the addition of complexity do not have as simple a form as desirable. For example, Theorem 1.4.2 implies

F,G <+ F G . ( ) ( ( )) + ( ) We wouldlike to see just F on the right-hand side, but the inequality F,G <+ ( ) ( ) F G does not always hold (see Exercise 5). The problem is that if we ( ) + ( ) compose a program for F,G from those of F and G then we have to separate ( ) them from each other somehow. In this section, we introduce a variant of Kol- mogorov’s complexity discovered by Levin and independently by Chaitin and Schnorr, which has the property that “programs” are self-delimiting: this will free us of the necessity to separate them from each other. Definition 1.6.1 A set of strings is called prefix-free if for any pair F,G of elements in it, F is not a prefix of G. A one-to-one function into a set of strings is a prefix code, or instantaneous code if its domain of definition is prefix free. An interpreter 5 >, F is self-delimiting (s.d.) if for each F, theset of strings ( ) F > for which 5 >, F is defined is a prefix-free set. y ( ) Example 1.6.2 The mapping > >= is a prefix code. y → A self-delimiting p.r. function 5 > can be imagined as a function computable ( ) on a special self-delimiting Turing machine. Definition 1.6.3 A Turing machine T is self-delimiting if it has no input tape, but can ask any time for a new input symbol. After some computation and a few such requests (if ever) the machine decides to write the output and stop. y The essential difference between a self-delimiting machine T and an ordinary Turing machine is that T does not know in advance how many input symbols

24 1.6. The coding theorem suﬃce; she must compute this information from the input symbols themselves, without the help of an “endmarker”. Requiring an interpreter >, F to be self-delimiting in > has many advan- ( ) tages. First of all, when we concatenate descriptions, the interpreter will be able to separate them without endmarkers. Lost endmarkers were responsible for the additional logarithmic terms and the use of the function < in Theorem 1.4.2 ( ) and several other formulas for . Deﬁnition 1.6.4 A self-delimiting partial recursive interpreter ) is called optimal if for any other s.d. p.r. interpreter , we have

) <+ . (1.6.1) y

Theorem 1.6.1 There is an optimal s.d. interpreter. Proof. The proof of this theorem is similar to the proof of Theorem 1.3.1. We take the universal partial recursive function + 0, >,F . We transform each function ( ) + >, F = + 0, >,F into a self-delimiting function , >, F = , 0, >,F , but 0 ( ) ( ) 0 ( ) ( ) so as not to change the functions +0 which are already self-delimiting. Then we form ) from , just as we formed * from +. It is easy to check that the function ) thus defined has the desired properties. Therefore we are done if we construct ,. Let some Turing machine M compute G = + >, F in B steps. We define 0 ( ) , >, F = G if : > B and if M does not compute in B or fewer steps any 0 ( ) ( ) ≤ output + ?, F from any extension or prefix ? of length B of >. If + is self- 0 ( ) ≤ 0 delimiting then ,0 = +0. But ,0 is always self-delimiting. Indeed, suppose that > > and that M computes + > ,F in B steps. The value , > ,F can be 0 ⊑ 1 0 ( 7 ) 7 0 ( 7 ) defined only if : > B . The definition of , guarantees that if B B then ( 7) ≤ 7 0 0 ≤ 1 , > ,F is undefined, otherwise , > ,F is undefined. 0 ( 1 ) 0 ( 0 ) Definition 1.6.5 We fix an optimal self-delimiting p.r. interpreter ) >, F and ( ) write F G = F G . ( | ) ) ( | ) We call G F the (self-delimiting) conditional complexity of F with respect to ( | ) G. y We will use the expression “self-delimiting” only if confusion may arise. Oth- erwise, under complexity, we generally understand the self-delimiting complexity. Let ) > = ) >, Λ . All definitions for unconditional complexity, joint com- ( ) ( ) plexity etc. are automatically in force for = ) .

25 1. Complexity

Let us remark that the s.d. interpreter deﬁned in Theorem 1.6.1 has the following stronger property. For any other s.d. interpreter there is a string 6 such that we have F = ) 6F (1.6.2) ( ) ( ) for all F. We could say that the interpreter ) is universal. Let us show that the functions and are asymptotically equal. Theorem 1.6.2 <+ <+ . (1.6.3) ( ) Proof. Obviously, <+ . We deﬁne a self-delimiting p.r. function >, F . The ( ) machine computing tries to decompose > into the form > = C={ such that C is the number : { in binary notation. If it succeeds, it outputs * {, F . We have ( ) ( ) G F <+ G F <+ G F 2 log G F . ( | ) ( | ) ( | )+ ( | )

Let us review some of the relations proved in Sections 1.3 and 1.4. Instead of the simple estimate F <+ < for a binary string F, of length <, only the obvious ( ) consequence of Theorem 1.6.2 holds that is

F <+ < . ( ) ( ) We must similarly change Theorem 1.4.3. We will use the universal p.r. function ) F = ) >, F . The r.e. set + is the range of the function ) . Let = + be an >( ) ( ) > > > enumerable set of pairs of strings. The section 0 is deﬁned as in (1.4.5). Then for all F 0, we have ∈ F 0 <+ log 0 : > . (1.6.4) ( | ) ( | |) + ( ) We do not need : > since the function ) is self-delimiting. However, the real ( ( )) analogon of Theorem 1.4.3 is the Coding Theorem, to be proved later in this section. Part b of Theorem 1.3.2 holds for , but is not the strongest what can be said. The counterpart of this theorem is the inequality (1.6.10) below. Theorem 1.4.1 and its corollary hold without change for . The additivity relations will be proved in a sharper form for in the next section.

26 1.6. The coding theorem

1.6.2 Universal semimeasure Self-delimiting complexity has an interesting characterization that is very useful in applications. Definition 1.6.6 A function | : ( 0, 1 is called a semimeasure over the → [ ] space ( if | F 1. (1.6.5) F ( ) ≤ Õ It is called a measure (a probability distribution) if equality holds here. A semimeasure is called constructive if it is lower semicomputable. A constructive semimeasure which multiplicatively dominates every other constructive semimeasure is called universal. y Remark 1.6.7 Just as in an earlier remark, we warn that semimeasures will later be defined over the space NN. They will be characterized by a nonnegative function | over S = N∗ but with a condition different from (1.6.5). y The following remark is useful. Proposition 1.6.8 If a constructive semimeasure | is also a measure then it is computable.

Proof. Every computable family |7 7 = 1, 2,... of semimeasures is dominated 7 ( ) by the semimeasure 7 2− |7. Since there are only countably many constructive semimeasures there exists a semimeasure dominating them all. The only point to prove is that the dominatingÍ semimeasure can be made constructive. Below, we construct a function ` F lower semicomputable in >, F such that >( ) ` > as a function of F is a constructive semimeasure for all > and all constructive semimeasures occur among the |>. Once we have ` > we are done because for any positive computable function X > with X > 1 the semimeasure ( ) > ( ) ≤ X > ` is obviously lower semicomputable and dominates all the ` . > ( ) > > Let ( F be the lower semicomputable functionÍ with Gödel number > and > ( ) letÍ (B F be a function recursive in >,B,F with rational values, dondecreasing in > ( ) B, such that lim (B F = max 0, ( F and for each B, >, the function (B F is B > ( ) { >( )} >( ) different from 0 only for F B. Let us define `B recursively in B as follows. Let ≤ > `0 F = 0. Suppose that `B is already defined. If (B 1 is a semimeasure then we > ( ) > >+

27 1. Complexity define `B 1 = (B 1, otherwise `B 1 = `B . Let ` = lim `B . The function ` F >+ >+ >+ > > B > >( ) is, by its definition, lower semicomputable in >, F and a semimeasure for each fixed >. It is equal to (> whenever (> is a semimeasure, hence it enumerates all constructive semimeasures. The above theorem justifies the following notation. Definition 1.6.9 We choose a fixed universal constructive semimeasure and call it m F . y ( ) With this notation, we can make it a little more precise in which sense this measure dominates all other constructive semimeasures. Definition 1.6.10 For an arbitrary constructive semimeasure a let us define

m a = m > : the (self-delimiting) program > computes a . (1.6.6) ( ) { ( ) } Õ Similar notation will be used for other objects: for example now m 5 make ( ) sense for a recursive function 5 . y Let us apply the above concept to some interesting cases. Theorem 1.6.4 For all constructive semimeasures a and for all strings F, we have

m a a F <∗ m F . (1.6.7) ( ) ( ) ( ) Proof. Let us repeat the proof of Theorem 1.6.3, using m > in place of X > . We ( ) ( ) obtain a new constructive semimeasure m F with the property m a a F ′( ) ( ) ( ) ≤ m F for all F. Noting m F =∗ m F ﬁnishes the proof. ′( ) ′( ) ( ) 1.6.3 Preﬁx codes

The main theorem of this section says F =+ log m F . Before proving it, we ( ) − ( ) relate descriptional complexity to the classical coding problem of information theory. Definition 1.6.11 A (binary) code is any mapping from a set of binary sequences to a set of objects. This mapping is the decoding function of the code. If the mapping is one-to-one, then sometimes the set itself is also called a code. A code is a prefix code if is a prefix-free set. y The interpreter * > is a decoding function. ( ) Definition 1.6.12 We call string > the first shortest description of F if it is the lexicographically first binary word ? with ? = F and * ? = F. y | | ( ) ( )

28 1.6. The coding theorem

The set of first shortest descriptions is a code. This imposes the implicit lower bound stated in (1.5.1) on F : ( ) F : F = < 2<. (1.6.8) |{ ( ) }| ≤ The least shortest descriptions in the definition of F form moreover a ( ) prefix code. We recall a classical result of information theory.

Lemma 1.6.13 (Kraft’s Inequality, see [13]) For any sequence :1,:2,... of natural numbers, there is a prefix code with exactly these numbers as codeword lengths, if and only if : 2− 7 1. (1.6.9) ≤ Õ7 Proof. Recall the standard correspondence F F between binary strings and ↔ [ ] binary subintervals of the interval 0, 1 . A prefix code corresponds to a set [ ] of disjoint intervals, and the length of the interval F is 2 : F . This proves [ ] − ( ) that (1.6.9) holds for the lengths of codewords of a prefix code. Suppose that :7 is given and (1.6.9) holds. We can assume that the sequence :7 is nondecreasing. Chop disjoint, adjacent intervals 1,2,... of length :1 :2 2− , 2− ,... from the left end of the interval 0, 1 . The right end of 9 is 9 : 8 [ ] 8=1 2− . Since the sequence : 8 is nondecreasing, all intervals 8 are binary intervals. Take the binary string corresponding to 8 as the 8-th codeword. Í Corollary 1.6.14 We have log m F <+ F . − ( ) ( ) Proof. The lemma implies G F 2− ( | ) 1. (1.6.10) G ≤ Õ Since F is an upper semicomputable function, the function 2 F is lower ( ) − ( ) semicomputable. Hence it is a constructive semimeasure, and we have log m F <+ − ( ) F . ( ) The construction in the second part of the proof of Lemma 1.6.13 has some disadvantages. We do not always want to rearrange the numbers : 8, for example because we want that the order of the codewords reflect the order of our original objects. Without rearrangement, we can still achieve a code with only slightly longer codewords.

29 1. Complexity

Lemma 1.6.15 (Shannon-Fano code) (see [13]) Let |1, |2,... be positive numbers with | 1. There is a binary prefix code > , > ,... where the codewords 8 8 ≤ 1 2 are in lexicographical order, such that Í > log | 2. (1.6.11) | 8|≤− 8 + Proof. We follow the construction above and cut off disjoint, adjacent (not necessarily binary) intervals of length | from the left end of 0, 1 . Let { be the 8 8 [ ] 8 length of the longest binary intervals contained in 8. Let >8 be the binary word corresponding to the first one of these. Four or fewer intervals of length {8 cover 8. Therefore (1.6.11) holds.

1.6.4 The coding theorem for F ( ) The above results suggest F log m F . But in the proof, we have to deal ( )≤− ( ) with the problem that m is not computable. Theorem 1.6.5 (Coding Theorem, see [31, 17, 10])

F =+ log m F (1.6.12) ( ) − ( ) Proof. We construct a self-delimiting p.r. function > with the property that ( ) F log m F 4. The function to be constructed is the decoding funct- ( )≤− ( )+ ion of a preﬁx code hence the code-construction of Lemma 1.6.13 proposes itself. But since the function m F is only lower semicomputable, it is given only by a ( ) sequence converging to it from below. Let H , 9 : B = 1, 2,... be a recursive enumeration of the set F, 9 : { ( B B) } { ( ) 9 < m F without repetition. Then ( )} 9 9 2− B = 2− B 2m F < 2. = ≤ ( ) ÕB ÕF ÕHB 1 ÕF

9B 1 Let us cut off consecutive adjacent, disjoint intervals B of length 2− − from the left side of the interval 0, 1 . We define as follows. If > is a largest binary [ ] [ ] subinterval of some then > = H . Otherwise > is undefined. B ( ) B ( ) The function is obviously self-delimiting and partial recursive. It follows from the construction that for every F there is a B with H = F and 0.5m F < B ( ) 2 9B . Therefore, for every F there is a > such that > = F and > log m F − ( ) | |≤− ( )+ 4.

30 1.6. The coding theorem

The Coding Theorem can be straightforwardly generalized as follows. Let 5 F,G be a lower semicomputable nonnegative function. Then we have ( ) G F <+ log 5 F,G (1.6.13) ( | ) − ( ) for all F with 5 F,G 1. The proof is the same. G ( ) ≤ 1.6.5 AlgorithmicÍ probability We can interpret the Coding Theorem as follows. It is known in classical information theory that for any probability distribution | F F ( a binary preﬁx ( ) ( ∈ ) code 5 F can be constructed such that : 5 F log | F 1. We learned | ( ) ( |( )) ≤− ( )+ that for computable, moreover, even for contstructive distributions | there is a universal code with a self-delimiting partial-recursive decoding function ) independent of | such that for the codeword length F we have ( ) F log | F 2 . ( )≤− ( )+ | Here, only the additive constant 2| depends on the distribution |. Let us imagine the self-delimiting Turing machine T computing our interpreter ) > . Every time the machine asks for a new bit of the description, we ( ) could toss a coin to decide whether to give 0 or 1. Deﬁnition 1.6.16 Let % F be the probability that the self-delimiting machine ) ( ) T gives out result F after receiving random bits as inputs. y We can write % F = , F where , F is the set 2 : > : ) > = F . ) ( ) ) ( ) ) ( ) { − ( ) ( ) } Since 2 F = max , F , we have log % F F . The semimeasure % − ( ) ) ( ) − ) ( ) ≤ ( ) ) is constructive, hence % <∗ Ím, hence

<+ log m <+ log % , − − ) ≤ hence log % =+ log m =+ . − ) − Hence the sum %) F of the set ,) F is at most a constant times larger than ( ) F ( ) its maximal element 2− ( ). This relation was not obvious in advance. The outcome F might have high probability because it has many long descriptions. But we found that then it must have a short description too. In what follows it will be convenient to fix the definition of m F as follows. ( ) Definition 1.6.17 From now on let us define m F = % F . (1.6.14) ( ) ) ( ) y

31 1. Complexity

1.7 The statistics of description length

We can informally summarize the relation =+ log % saying that if an object − ) has many long descriptions then it has a short one. But how many descriptions does an object really have? With the Coding Theorem, we can answer several questions of this kind. The reader can view this section as a series of exercises in the technique acquired up to now. Theorem 1.7.1 Let 5 F,< be the number of binary strings > of length < with ( ) ) > = F for the universal s.d. interpreter ) defined in Theorem 1.6.1. Then for ( ) every < F , we have ≥ ( ) log 5 F,< =+ < F,< . (1.7.1) ( ) − ( ) Using F <+ F,< , which holds just as (1.4.1), and substituting < = F ( ) ( ) ( ) we obtain log 5 F, F =+ F F, F <+ 0, implying the following. ( ( )) ( )− ( ( )) Corollary 1.7.1 The number of shortest descriptions of any object is bounded by a universal constant. Since the number log 5 F, F is nonnegative, we also derived the identity ( ( )) F, F =+ F (1.7.2) ( ( )) ( ) which could have been proven also in the same way as Theorem 1.4.4. Proof of Theorem 1.7.1. It is more convenient to prove equation (1.7.1) in the form < 2− 5 F,< =∗ m F,< (1.7.3) ( ) ( ) where m F,G = m F,G . First we prove <∗ . Let us define a p.r. self-delimiting ( ) (h i) function by > = ) > ,: > . Applying to a coin-tossing argument, ( ) h ( ) ( )i 2 < 5 F,< is the probability that the pair F,< is obtained. Therefore the left- − ( ) h i hand side of (1.7.3), as a function of F,< , is a constructive semimeasure, dom- h i inated by m F,< . ( ) Now we prove >∗ . We define a self-delimiting p.r. function as follows. The machine computing > tries to decompose > into three segments > = V 2 ={| ( ) ( ) in such a way that ) { is a pair F,: > 2 . If it succeeds then it outputs F. ( ) h ( )+ i By the universality of ), there is a binary string 6 such that ) 6> = > for all ( ) ( ) >. Let @ = : 6 . For an arbitrary pair F,<, let ? be a shortest binary string with ( ) ) ? = F,< , and | an arbitrary string of length ( ) h i : = < : ? @ : V @ = . − ( )− − ( ( ) ) 32 1.7. The statistics of description length

Then V @ =?| = ) 6V @ =?| = F. Since | is arbitrary here, there are 2: ( ( ) ) ( ( ) ) binary strings > of length < with ) > = F. ( ) How many objects are there with a given complexity of length < for which ) > is deﬁned. Let us deﬁne < ( ) the moving average 2 1 ℎ) <, 2 = 6) < 7 . ( ) 22 1 ( + ) + 7= 2 Õ− y

Here is an estimation of these numbers. Theorem 1.7.2 ([47]) There is a natural number 2 such that

log =+ < < , (1.7.4) | <| − ( ) log ℎ <, 2 =+ < < . (1.7.5) ) ( ) − ( ) Since we are interested in general only in accuracy up to additive constants, we could omit the normalizing factor 1 22 1 from the moving average ℎ . /( + ) ) We do not know whether this average can be replaced by 6 < . This might ) ( ) depend on the special universal partial recursive function we use to construct the optimal s.d. interpreter ) > . But equation (1.7.5) is true for any optimal ( ) s.d. interpreter ) (such that the inequality (1.6.1) holds for all ) while there is an optimal s.d. interpreter for which 6 < = 0 for every odd <. Indeed, ( ) let 00> = ) > if : > is even, 1> = ) > if : > is odd and let be ( ) ( ) ( ) ( ) ( ) ( ) undeﬁned in all other cases. Then is deﬁned only for inputs of even length, while 2. ≤ ) + Lemma 1.7.3 m F,G =∗ m F . G ( ) ( ) Õ Proof. The left-hand side is a constructive semimeasure therefore it is dominated by the right side. To show >∗ , note that by equation (1.4.2) as applied to , we have F,F =+ F . Therefore m F =∗ m F,F < m F,G . ( ) ( ) ( ) ( ) G ( ) Proof of Theorem 1.7.2: Let 3 = . Using Lemma 1.7.3 and Theorem 1.7.1 < | <| Í we have < < 6) < 3< = 5 F,< =∗ 2 m F,< =∗ 2 m < . ( ) ≤ F ( ) F ( ) ( ) Õ Õ 33 1. Complexity

Since the complexity has the same continuity property (1.4.3) as , we can derive from here ℎ <, 2 2∗ 3 for an ) ( ) ≤ ( ) ( ) ) ( ) < appropriate 2, we will prove that the complexity of at least half of the elements of < is near <. For some constant 2 , we have > < 2 for all > . Indeed, if the 0 ( ) ≤ + 0 ∈ < s.d. p.r. interpreter > is equal to > where ) > is deﬁned and is undeﬁned ( ) ( ) otherwise then > < for > . By >, : > =+ > , and a variant of ( ) ≤ ∈ < ( ( )) ( ) Lemma 1.7.3, we have

m > =∗ m >, < =∗ m < . ( ) ( ) ( ) > > Õ∈ < Õ∈ <

Using the expression (1.7.4) for 3<, we see that there is a constant 21 such that

1 > < 2 3− 2− ( ) 2− + 1 . < ≤ > Õ∈ < Using Markov’s Inequality, hence the number of elements > of with > < ( ) ≥ < 2 1 is at least 3 2. We ﬁnish the proof making 2 to be the maximium of − 1 −

< > < 2− 2− ( ) 22− m < . ≤ ( ) : > =< (Õ) From this, Markov’s Inequality gives that the number of strings > B< with ∈ > < < 9 is at most 22< 9. ( ) − − There is a more polished way to express this result. Deﬁnition 1.7.4 Let us introduce the following function of natural numbers:

+ < = max 9 . ( ) 9 < ( ) ≤ y

34 1.7. The statistics of description length

This is the smallest monotonic function above < . It is upper semicom- ( ) putable just as is, hence 2− + is universal among the monotonically decreasing constructive semimeasures. The following theorem expresses + more directly in terms of .

Theorem 1.7.3 We have < =+ log < log < . +( ) + (⌊ ⌋) Proof. To prove <+ , we construct a s.d. interpreter as follows. The machine computing > ﬁnds a decomposition C{ of > such that ) C = : { , then outputs ( ) ( ) ( ) the number whose binary representation (with possible leading zeros) is the binary string {. With this , we have

9 <+ 9 log < log < (1.7.7) ( ) ( ) ≤ + (⌊ ⌋) for all 9 <. The numbers between < 2 and < are the ones whose binary ≤ / representation has length log < 1. Therefore if the bound (1.7.6) is sharp for ⌊ ⌋+ most F then the bound (1.7.7) is sharp for most 9. The sharp monotonic estimate log < log < for < is less satisfactory + (⌊ ⌋) ( ) than the sharp estimate log < for Kolmogorov’s complexity < , because it is not ( ) a computable function. We can derive several computable upper bounds from it, for example log < 2 log log <, log < log log < 2 log log log <, etc. but none + + + of these is sharp for large <. We can still hope to find computable upper bounds of which are sharp infinitely often. The next theorem provides one. Theorem 1.7.4 (see [47]) There is a computable upper bound < of the funct- ( ) ion < with the property that < = < holds for infinitely many <. ( ) ( ) ( ) For the proof, we make use of a s.d. Turing machine T computing the function ). Similarly to the proof of Theorem 1.5.1, we introduce a time-restricted complexity. Definition 1.7.5 We know that for some constant 2, the function < is bounded ( ) by 2 log < 2. Let us fix such a 2. + For any number < and s.d. Turing machine M, let M <; B be the minimum ( ) of 2 log < 2 and the lengths of descriptions from which M computes < in B or + fewer steps. y At the time we proved the Invariance Theorem we were not interested in exactly how the universal p.r. function + was to be computed. Now we need to be more specific. Definition 1.7.6 Let us agree now that the universal p.r. function is computed by a universal Turing machine V with the property that for any Turing machine

35 1. Complexity

M there is a constant ; such that the number of steps needed to simulate B steps of M takes no more than ;B steps on V. Let T be the Turing machine constructed from V computing the optimal s.d. interpreter ), and let us write

<,B = T <,B . ( ) ( ) y

With these deﬁnitions, for each s.d. Turing machine M there are constants ;0,;1 with <,; B M <,B ; . (1.7.8) ( 0 ) ≤ ( )+ 1 The function <; B is nonincreasing in B. ( ) Proof of Theorem 1.7.4. First we prove that there are constants 20, 21 such that <; B < <; B 1 implies ( ) ( − ) <,B ; 2 B <; B 2 . (1.7.9) (h i 0 ) ≤ ( )+ 1 Indeed, we can construct a Turing machine M simulating the work of T such that if T outputs a number < in B steps then M outputs the pair <,B in 2B steps: h i

M <,B ;2B <,B . (h i ) ≤ ( ) Combining this with the inequality (1.7.8) gives the inequality (1.7.9). We define now a recursive function as follows. To compute F , the s.d. Turing ( ) machine first tries to find <,B such that F = <,B . (It stops even if it did not find h i them.) Then it outputs F; 2 B . Since F is the length of some description ( 0 ) ( ) of F, it is an upper bound for F . On the other hand, suppose that F is one of ( ) the infinitely many integers of the form <,B with h i < = <; B < <; B 1 . ( ) ( ) ( − ) Then the inequality (1.7.9) implies F < 2 while < <+ F is ( ) ≤ ( ) + 1 ( ) ( ) known (it is the equivalent of 1.4.4 for ), so F < 2 for some constant ( ) ≤ ( )+ 2. Now if would not be equal to at infinitely many places, only close to it (closer than 2) then we can decrease it by some additive constant less than 2 and modify it at finitely many places to obtain the desired function .

36 2 Randomness

2.1 Uniform distribution

Speaking of a “random binary string”, one generally understands randomness with respect to the coin-toss distribution (when the probability % F of a binary < ( ) sequence F of length < is 2− ). This is the only distribution considered in the present section. One can hope to distinguish sharply between random and nonrandom infinite sequences. Indeed, an infinite binary sequence whose elements are 0 with only finitely many exceptions, can be considered nonrandom without qualification. This section defines randomness for finite strings. For a sequence F of length <, it would be unnatural to fix some number 9, declare F nonrandom when its first 9 elements are 0, but permit it to be random if only the first 9 1 − elements are 0. So, we will just declare some finite strings less random than others. For this, we introduce a certain real-valued function 3 F 0 measur- ( ) ≥ ing the deficiency of randomness in the string F. The occurrence of a nonrandom sequence is considered an exceptional event, therefore the function 3 F can ( ) assume large values only with small probability. What is “large” and “small” depends only on our “unit of measurement”. We require for all <, 9

< 9 % F : F B , 3 F > 9 < 2− , (2.1.1) { ( ) ∈ ( ) } Õ saying that there be at most 2< 9 binary sequences F of length < with 3 F > 9. − ( ) Under this condition, we even allow 3 F to take the value . ( ) ∞ To avoid arbitrariness in the distinction between random and nonrandom, the function 3 F must be simple. We assume therefore that the set <,9,F : ( ) { ( ) : F = <, 3 F > 9 is recursively enumerable, or, which is the same, that the ( ) ( ) } function

d : Ω , → (−∞ ∞]

37 2. Randomness is lower semicomputable. Remark 2.1.1 We do not assume that the set of strings F B< with small defi- ∈ ciency of randomness is enumerable, because then it would be easy to construct a “random” sequence. y Definition 2.1.2 A lower semicomputable function 3 : S R (where R is the 2 → set of real numbers) is called a Martin-Löf test (ML-test), or a probability-bounded test if it satisfies (2.1.1). A ML-test 3 F is universal if it additively dominates all other ML-tests: for 0 ( ) any other ML-test 3 F there is a constant 2 < such that for all F we have ( ) ∞ 3 F < 3 F 2. y ( ) 0 ( )+ If a test 30 is universal then any other test 3 of randomness can discover at most by a constant amount more deficiency of randomness in any sequence F than 3 F . Obviously, the difference of any two universal ML-tests is bounded 0 ( ) by some constant. The following theorem reveals a simple connnection between descriptional complexity and a certain randomness property. Definition 2.1.3 Let us define the following function for binary strings F:

3 F = : F F : F . (2.1.2) 0 ( ) ( )− ( | ( )) y Theorem 2.1.1 (Martin-Löf) The function 3 F is a universal Martin-Löf test. 0 ( ) Proof. Since F G is semicomputable from above, 3 is semicomputable from ( | ) 0 below. The property of semicomputability holds for 30 as a straightforward consequence of Theorem 1.5.1 Therefore 30 is a ML-test. We must show that it is larger (to within an additive constant) than any other ML-test. Let 3 be a ML- test. Let us deﬁne the function F,G to be G 3 F for G = : F , and ( ) − ( ) ( ) ∞ otherwise. Then is upper semicomputable, and satisﬁes (1.5.1). Therefore by Theorem 1.5.3, we have F : F <+ : F 3 F . ( | ( )) ( )− ( ) Theorem 2.1.1 says that under very general assumptions about randomness, those strings F are random whose descriptional complexity is close to its maximum, : F . The more “regularities” are discoverable in a sequence, the less ( ) random it is. However, this is true only of regularities which decrease the descriptional complexity. “Laws of randomness” are regularities whose probability is high, for example the law of large numbers, the law of iterated logarithm, the arcsine law, etc. A random sequence will satisfy all such laws.

38 2.1. Uniform distribution

Example 2.1.4 The law of large numbers says that in most binary sequences of length <, the number of 0’s is close to the numer of 1’s. We prove this law of probability theory by constructing a ML-test 3 F taking large values on sequences ( ) in which the number of 0’s is far from the number of 1’s. Instead of requiring (2.1.1) we require a somewhat stronger property of 3 F : for all <, ( ) 3 F % F 2 ( ) 1. (2.1.3) ( ) ≤ F B< Õ∈ From this, the inequality (2.1.1) follows by the following well-known inequality called the Markov Inequality which says the following. Let % be any probability distribution, 5 any nonnegative function, with expected value 5 = % F 5 F . For all _ 0 we have % ( ) F ( ) ( ) ≥ Í % F : 5 F > _ 5 _. (2.1.4) { ( ) ( ) % ( )} ≤ Õ For any string F S, and natural number 7, let # 7 F denote the number ∈ ( | ) of occurrences of 7 in F. For a binary string F of length < deﬁne > = # 1 F <, F ( | )/ and

# 1 G # 0 G % G = > ( | ) 1 > ( | ) F ( ) F ( − F ) 3 F = log % F < log < 1 . ( ) F ( )+ − ( + ) We show that 3 F is a ML-test. It is obviously computable. We prove (2.1.3). ( ) We have

3 F < < 1 1 % F 2 ( ) = 2− 2 %F F = %F F ( ) ( ) < 1 < 1 ( ) F F + + F Õ Õ < Õ < 1 < 9 9 9 < 9 1 = 1 − < 1 = 1. < 1 = 9 ( <) ( − <) < 1 = + Õ9 0 + Õ9 0 The test 3 F expresses the (weak) law of large numbers in a rather strong ver- ( ) sion. We rewrite 3 F as ( ) 3 F = < 1 ℎ > log < 1 ( ) ( − ( F )) − ( + ) where ℎ > = > log > 1 > log 1 > . The entropy function ℎ > achieves ( ) − − ( − ) ( − ) ( ) its maximum 1 at > = 1 2. Therefore the test 3 F tells us that the probability / ( ) of sequences F with > < >< 1 2 for some constant > is bounded by the F /

39 2. Randomness exponentially decreasing quantitiy < 1 2 < 1 ℎ > . We also see that since for ( + ) − ( − ( )) some constant 2 we have

1 ℎ > > 2 > 1 2 2, − ( ) ( − / ) therefore if the diﬀerence > 1 2 is much larger than | F − / | log < r 2< then the sequence F is “not random”, and hence, by Theorem 2.1.1 its complexity is signiﬁcantly smaller than <. y

2.2 Computable distributions

Let us generalize the notion of randomness to arbitrary discrete computable probability distributions.

2.2.1 Two kinds of test Let % F be a probability distribution over some discrete countable space Ω, ( ) which we will identify for simplicity with the set S of ﬁnite strings of natural numbers. Thus, % is a nonnegative function with

% F = 1. F ( ) Õ Later we will consider probability distributions % over the space NN, and they will also be characterized by a nonnegative function % F over S. However, the ( ) above condition will be replaced by a different one. We assume % to be computable, with some Gödel number 4. We want to define a test 3 F of randomness with respect to %. It will measure how justified ( ) is the assumption that F is the outcome of an experiment with distribution %. Definition 2.2.1 A function 3 : S R is an integrable test, or expectation- → bounded test of randomness with respect to % if it is lower semicomputable and satisfies the condition 3 F % F 2 ( ) 1. (2.2.1) F ( ) ≤ Õ It is universal if it dominates all other integrable tests to within an additive constant. y

40 2.2. Computable distributions

(A similar terminology is used in [36] in order to distinguish tests satisfying condition (2.2.1) from Martin-Löf tests.) Remark 2.2.2 We must allow 3 F to take negative values, even if only, say val- ( ) ues 1, since otherwise condition (2.2.1) could only be satisfied with 3 F = 0 ≥ − ( ) for all F with % F > 0. y ( ) Proposition 2.2.3 Let 2 = log 2c2 6 . If a function 3 F satisfies (2.2.1) then it ( / ) ( ) satisfies (2.1.1). If 3 F satisfies (2.1.1) then 3 2 log 3 2 satisfies (2.2.1). ( ) − − Proof. The first statement follows by Markov’s Inequality (2.1.4). The second one can be checked by immediate computation.

Thus, condition 2.2.1 is nearly equivalent to condition (2.1.1): to each test 3 according to one of them, there is a test 3′ asymptotically equal to 3 satisfying the other. But condition (2.2.1) has the advantage of being just one inequality instead of the inﬁnitely many ones.

2.2.2 Randomness via complexity We want to find a universal integrable test. Let us introduce the function | F = ( ) % F 23 F . The above conditions on 3 F imply that | F is semicomputable ( ) ( ) ( ) ( ) from below and satisfies | F 1: so, it is a constructive semimeasure. F ( ) ≤ With the universal constructive semimeasure of Theorem 1.6.3, we also obtain a universal test for any computableÍ probability distribution %. Definition 2.2.4 For an arbitrary measure % over a discrete space Ω, let us denote m F d% F = log ( ) = log % F F . (2.2.2) ( ) % F − ( )− ( ) ( ) y

The following theorem shows that randomness can be tested by checking how close is the complexity F to its upper bound log % F . ( ) − ( ) Theorem 2.2.1 The function d F = log % F F is a universal integrable % ( ) − ( )− ( ) test for any ﬁxed computable probability distribution %. More exactly, it is lower semicomputable, satisﬁes (2.2.1) and for all integrable tests 3 F for %, we have ( ) 3 F <+ d F 3 % . ( ) % ( )+ ( )+ ( ) Proof. Let 3 F be an integrable test for %. Then a F = 23 F % F is a construc- ( ) ( ) ( ) ( ) tive semimeasure, and it has a self-delimiting program of length <+ 3 % . ( )+ ( )

41 2. Randomness

It follows (using the definition (1.6.6)) that ; a <∗ 2 3 % ; hence inequal- ( ) ( )+ ( ) ity (1.6.7) gives 3 % a F 2 ( )+ ( ) <∗ m F . ( ) ( ) Taking logarithms finishes the proof. The simple form of our universal test suggests remarkable interpretations. It says that the outcome F is random with respect to the distribution %, if the latter assigns to F a large enough probability—however, not in absolute terms, only relatively to the universal semimeasure m F . The relativization is essen- ( ) tial, since otherwise we could not distinguish between random and nonrandom outcomes for the uniform distribution %< defined in Section 2.1. For any (not necessarily computable) distribution %, the %-expected value of the function m F % F is at most 1, therefore the relation ( )/ ( ) m F 9% F ( ) ≤ ( ) holds with probability not smaller than 1 1 9. If % is computable then the − / inequality m % % F <∗ m F ( ) ( ) ( ) holds for all F. Therefore the following applications are obtained. • If we assume F to be the outcome of an experiment with some simple computable probability distribution % then m F is a good estimate of % F . The ( ) ( ) goodness depends on how simple % is to define and how random F is with respect to %: how justified is the assumption. Of course, we cannot compute m F but it is nevertheless well defined. ( ) • If we trust that a given object F is random with respect to a given distribution % then we can use % F as an estimate of m F . The degree of approximation ( ) ( ) depends on the same two factors. Let us illustrate the behavior of the universal semimeasure m F when F runs ( ) over the set of natural numbers. We define the semimeasures { and | as follows. 9 Let { < = 2 <2 for an appropriate constant 2, let | < = { F if < = 22 , and ( ) / ( ) ( ) 0 otherwise. Then m < dominates both { < and | < . The function m < ( ) ( ) ( ) ( ) dominates 1 < log2 <, but jumps at many places higher than that. Indeed, it / is easy to see that m < converges to 0 slower than any positive computable ( ) function converging to 0: in particular, it is not computable. Hence it cannot be a measure, that is m F < 1. We feel that we can make m F “large” F ( ) ( ) whenever F is “simple”. Í

42 2.2. Computable distributions

We cannot compare the universal test defined in the present section directly with the one defined in Section 2.1. There, we investigated tests for a whole < family %< of distributions, where %< is the uniform distribution on the set B . That the domain of these distributions is a different one for each < is not essential since we can identify the set B< with the set 2<,..., 2< 1 1 . Let us rewrite { + − } the function 3 F defined in (2.1.2) as 0 ( ) 3 F = < F < 0 ( ) − ( | ) for F B<, and set it for F outside B<. Now the similarity to the expression ∈ ∞ d F = log % F log m F % ( ) − ( )− ( ) is striking because < = log % F . Together with the remark made in the previous < ( ) paragraph, this observation suggests that the value of log m F is close to the − ( ) complexity F . ( ) 2.2.3 Conservation of randomness The idea of conservation of randomness has been expressed first by Levin, who has published some papers culminating in [33] developing a general theory. In this section, we address the issue in its simplest form only. Suppose that we want to talk about the randomness of integers. One and the same integer F can be represented in decimal, binary, or in some other notation: this way, every time a different string will represent the same number. Still, the form of the representation should not affect the question of randomness of F significantly. How to express this formally? Let % be a computable measure and let 5 : S S be a computable function. → We expect that under certain conditions, the randomness of F should imply the randomness of 5 F —but, for which distribution? For example, if F is a binary ( ) string and 5 F is the decimal notation for the number expressed by 1F then we ( ) expect 5 F to be random not with respect to %, but with respect to the measure ( ) 5 ∗% that is the image of % under 5 . This measure is defined by the formula

1 5 ∗% G = % 5 − G = % F . (2.2.3) ( )( ) ( ( )) ( ) 5 F =G Õ( ) Proposition 2.2.5 If 5 is a computable function and % is a computable measure then 5 ∗% is a computable measure.

43 2. Randomness

Proof. It is suﬃcient to show that 5 ∗% is lower semicomputable, since we have seen that a lower semicomputable measure is computable. But, the semicomputability can be seen immediately from the deﬁnition.

Let us see that randomness is indeed preserved for the image. Similarly to (1.6.6), let us deﬁne for an arbitrary computable function 5 :

m 5 = m > : program > computes 5 . (2.2.4) ( ) { ( ) } Theorem 2.2.2 Let 5 be aÕ computable function. Between the randomness test for % and the test for the image of %, the following relation holds, for all F:

d 5 F <+ d F 5 % . (2.2.5) 5 ∗% ( ( )) % ( )+ ( )+ ( ) Proof. Let us denote the function on the left-hand side of (2.2.5) by

m 5 F 3% F = d 5 % 5 F = log ( ( )) . ( ) ∗ ( ( )) 5 % 5 F ( ∗ )( ( )) It is lower semicomputable, with the help of a program of length 5 % , ( )+ ( ) by its very deﬁnition. Let us check that it is an integrable test, so it satisﬁes the inequality (2.2.1). We have

3 F d G % F 2 % ( ) = 5 ∗% G 2 5 ∗% ( ) = m G 1. F ( ) G ( )( ) G ( ) ≤ Õ Õ Õ

Hence 3 F isa test, andhenceit is <+ d F 5 % , by the universality of % ( ) % ( )+ ( )+ ( ) the test d F . (Strictly speaking, we must modify the proof of Theorem 2.2.1 % ( ) and see that a program of length 5 % is also suﬃcient to deﬁne the ( ) + ( ) semimeasure % F 3 F % .) ( ) ( | ) The appearance of the term 5 on the right-hand side is understandable: ( ) using some very complex function 5 , we can certainly turn any string into any other one, also a random string into a much less random one. On the other hand, the term % appears only due to the imperfection of our concepts. It ( ) will disappear in the more advanced theory presented in later sections, where we develop uniform tests. Example 2.2.6 For each string F of length < 1, let ≥ 2 < % F = − . ( ) < < 1 ( + ) 44 2.2. Computable distributions

This distribution is uniform on strings of equal length. Let 5 F be the function ( ) that erases the first 9 bits. Then for < 1 and for any string F of length < we ≥ have < 2− 5 ∗% F = . ( )( ) < 9 < 9 1 ( + )( + + ) (The empty string has the rest of the weight of 5 ∗%.) The new distribution is still uniform on strings of equal length, though the total probability of strings of length < has changed somewhat. We certainly expect randomnes conservation here: after erasing the first 9 bits of a random string of length < 9, we should + still get a random string of length <. y The computable function 5 applied to a string F can be viewed as some kind of transformation F 5 F . As we have seen, it does not make F less random. 7→ ( ) Suppose now that we introduce randomization into the transformation process itself: for example, the machine computing 5 F can also toss some coins. We ( ) want to say that randomness is also conserved under such more general transformations, but first of all, how to express such a transformation mathematically? We will describe it by a “matrix”: a computable probability transition function ) F,G 0 with the property that ( ) ≥ ) F,G = 1. G ( ) Õ Now, the image of the distribution % under this transformation can be written as )∗%, and defined as follows:

)∗% G = % F ) F,G . ( )( ) ( ) ( ) ÕF How to express now randomness conservation? It is certainly not true that every possible outcome G is as random with respect to )∗% as F is with respect to %. We can only expect that for each F, the conditional probability (in terms of the transition ) F,G ) of those G whose non-randomness is larger than that of F, is ( ) small. This will indeed be expressed by the corollary below. To get there, we d G upperbound the ) F, -expected value of 2 )∗% . Let us go over to exponential ( ·) ( ) notation: Deﬁnition 2.2.7 Denote m F d% F t% F = 2 ( ) = ( ) . (2.2.6) ( ) % F ( ) y

45 2. Randomness

Theorem 2.2.3 We have

d G log ) F,G 2 )∗% ( ) <+ d F ) % . (2.2.7) ( ) % ( )+ ( )+ ( ) ÕG Proof. Let us denote the function on the left-hand side of (2.2.7) by B F . It % ( ) is lower semicomputable by its very construction, using a program of length <+ ) % . Let us check that it satisﬁes % F B F 1 which, in this ( )+ ( ) F ( ) % ( ) ≤ notation, corresponds to inequality (2.2.1). We have Í % F B F = % F ) F,G B G ( ) % ( ) ( ) ( ) )∗% ( ) ÕF ÕF ÕG m G = % F ) F,G ( ) = m G 1. ) % G F ( ) G ( ) ∗ G ( ) ≤ Õ Õ ( )( ) Õ It follows that 3 F = log B F is an integrable test and hence 3 F <+ d F % ( ) % ( ) % ( ) % ( )+ ) % . (See the remark at the end of the proof of Theorem 2.2.2.) ( )+ ( ) Corollary 2.2.8 There is a constant 2 such that for every integer 9 0, for all F 0 ≥ we have

9 2 ) F,G : d G d F > 9 ) % 2− + 0 . { ( ) )∗% ( )− % ( ) + ( )+ ( )} ≤ Proof. TheÕ theorem says

∗ ) % ) F,G t)∗% G < B% F 2 ( )+ ( ). G ( ) ( ) ( ) Õ

d) % G Thus, it upperbounds the expected value of the function 2 ∗ ) ( ) according to the distribution ) F, . Applying Markov’s inequality (2.1.4) to this function ( ·) yields the desired result.

2.3 Inﬁnite sequences

These lecture notes treat the theory of randomness over continuous spaces mostly separately, starting in Section 4.1. But the case of computable measures over infinite sequences is particularly simple and appealing, so we give some of its results here, even if most follow from the more general results given later. In this section, we will fix a finite or countable alphabet Σ = A , A ,... , and { 1 2 } consider probability distributions over the set

- = ΣN

46 2.3. Infinite sequences of infinite sequences with members in Σ. An alternative way of speaking of this is to consider a sequence of random variables - ,- ,..., where - Σ, with a 1 2 7 ∈ joint distribution. The two interesting extreme special cases are Σ = B = 0, 1 , { } giving the set of infinite 0-1 sequences, and Σ = N, giving the set sequences of natural numbers.

2.3.1 Null sets Our goal is here to illustrate the measure theory developed in Section A.2, through Subsection A.2.3, on the concrete example of the set of infinite sequences, and then to develop the theory of randomness in it. We will distinguish a few simple kinds of subsets of the set ( of sequences, those for which probability can be defined especially easily. Definition 2.3.1 • For string H Σ , we will denote by H- the set of elements of - with prefix H. ∈ ∗ Such sets will be called cylinder sets. • A subset of - is open if it is the union of any number (not necessarily finite) of cylinder sets. It is called closed if its complement is open. • An open set is called constructive if it is the union of a recursively enumerable set of cylinder sets.

• A setis called X if it is the intersection of a sequence of open sets. It is called f if it is the union of a sequence of closed sets. • We say that a set - is ﬁnitely determined if there is an < and an E Σ< ⊆ ⊆ such that

= E- = A-. A E Ø∈ Let F be the class of all finitely determined subsets of -. • A class of subsets of - is called an algebra if it is closed with respect to finite intersections and complements (and then of course, also with respect to finite unions). It is called a f-algebra (sigma-algebra) when it is also closed with respect to countable intersections (and then of course, also with respect to countable unions). y Example 2.3.2 An example open set that is not finitely determined, is the set of all sequences that contain a substring 11. y

47 2. Randomness

The following observations are easy to prove. Proposition 2.3.3 a) Every open set can be represented as the union of a sequence of disjoint cylinder sets. b) The set of open sets is closed with respect to finite intersection and arbitrarily large union. c) Each finitely determined set is both open and closed. d) The class F of finitely determined sets forms an algebra. Probability is generally defined as a nonnegative, monotonic function (a “measure”, see later) on some subsets of the event space (in our case the set -) that has a certain additivity property. We do not need immediately the complete definition of measures, but let us say what we mean by an additive set function. Definition 2.3.4 Consider a nonnegative, function ` is defined over some subsets of -. that is also monotonic,that is implies ` ` . We say that ⊆ ( ) ⊆ ( ) < it is additive if, whenever it is defined on the sets 1,...,<, and on = 7=1 7 and the sets are mutually disjoint, then we have ` = ` ` . 7 ( ) ( 1)+···+ ( <) We say that ` is countably additive, or f-additive (sigma-additive), ifÐ in addition whenever it is defined on the sets 1,2,..., and on = ∞7=1 7 and the sets 7 are mutually disjoint, then we have ` = ∞= ` 7 . y ( ) 7 1 ( ) Ð First we consider only measures defined on cylinderÍ sets. N Definition 2.3.5 (Measure over Σ ) Let ` : Σ∗ R be a function assigning a → + nonnegative number to each string in Σ∗. We will call ` a measure if it satisfies the condition

` F = ` FA . (2.3.1) ( ) ( ) A Σ Õ∈ We will take the liberty to also write

` A- = ` A ( ) ( ) for all A Σ , and ` = 0. We call ` a probability measure if ` Λ = 1 and ∈ ∗ (∅) ( ) correspondingly, ` - = 1. ( ) A measure is called computable if it is computable as a function ` : Σ∗ R → + according to Deﬁnition 1.5.2. y The following observation lets us extend measures.

48 2.3. Inﬁnite sequences

Lemma 2.3.6 Let ` be a measure deﬁned as above. If a cylinder set A- is the disjoint union F- = F - F - ... of cylinder sets, then we have ` H = ` F . 1 ∪ 2 ∪ ( ) 7 ( 7) Σ Proof. Let us call an arbitrary G ∗ good if Í ∈ ` HG = ` HG- F - ( ) ( ∩ 7 ) Õ7 holds. We have to show that the empty string Λ is good. It is easy to see that G is good if HG- F - for some 7. Also, if GA is good ⊆ 7 for all A Σ then G is good. Now assume that Λ is not good. Then there is also ∈ an A Σ that is not good. But then there is also an A Σ such that A A is 1 ∈ 2 ∈ 1 2 not good. And so on, we obtain an inﬁnite sequence A A such that A A is 1 2 · · · 1 · · · < not good for any <. But then the sequence HA A is not contained in F -, 1 2 · · · 7 7 contrary to the assumption . Ð Corollary 2.3.7 Two disjoint union representations of the same open set

F7- = G7- Ø7 Ø7 imply ` F = ` G . 7 ( 7) 7 ( 7) Proof. ÍThe set 7ÍF7 - can also be written as

Ð F - = F - G -. 7 7 ∩ 8 Ø7 Ø7,8 An element F - G - is nonempty only if one of the strings F ,G is a continuation 7 ∩ 8 7 8 of the other. Calling this string H we have F - G - = H -. Now Lemma 2.3.6 78 7 ∩ 8 78 is applicable to the union F - = F - G - , giving 7 8 ( 7 ∩ 8 ) ` FÐ= ` F - G - , ( 7) ( 7 ∩ 8 ) Õ8 ` F = ` F - G - . ( 7) ( 7 ∩ 8 ) Õ7 Õ7,8 The right-hand side is also equal similarly to ` G . 8 ( 8) The above corollary allows the following deﬁnition:Í Deﬁnition 2.3.8 For an open set given as a disjoint union of cylinder sets F - let ` = ` F . For a closed set = - where is open, let 7 ( ) 7 ( 7) \ ` = ` - ` . y ( ) ( )− Í( ) 49 2. Randomness

The following is easy to check. Proposition 2.3.9 The measure as defined on open sets is monotonic and countably additive. In particular, it is countably additive on the algebra of finitely determined sets. Before extending measures to a wider range of sets, consider an important special case. With infinite sequences, there are certain events that are not impossible, but still have probability 0. Definition 2.3.10 Given a measure `, a set # - is called a null set with ⊆ respect to ` if there is a sequence 1, 2,... of open sets with the property # , and ` 2 ;. We will say that the sequence witnesses the ⊆ ; ; ( ;) ≤ − ; fact that # is a null set. y Ñ Examples 2.3.11 Let Σ = 0, 1 , let us define the measure _ by _ F = 2 < for { } ( ) − all < and for all F Σ<. ∈ 1. For every infinite sequence b -, the one-element set b is a null set with ∈ { } respect to _. Indeed, for each natural number <, let = b - : b 0 = < { ∈ ( ) b 0 , b 1 = b 1 ,...,b < = b < . Then _ = 2 < 1, and A = ( ) ( ) ( ) ( ) ( )} ( <) − − { } < <. 2. For a less trivial example, consider the set of those elements B - that are Ñ ∈ 1 in each positive even position, that is

= B - : B 2 = 1,B 4 = 1,B 6 = 1 ... . { ∈ ( ) ( ) ( ) } Then is a null set. Indeed, for each natural number <, let = B - : < { ∈ B 2 = 1,B 4 = 1,...,B 2< = 1 . This helps expressing as = < <, ( ) ( ) < ( ) } where _ < = 2− . ( ) Ñ y

Proposition 2.3.12 Let #1, #2,... be a sequence of null sets with respect to a measure `. Their union # = 7 #7 is also a null set.

Proof. Let 7,1, 7,2,... be theÐ inﬁnite sequence witnessing the fact that #7 is a null set. Let ; = ∞7=1 7,; 7. Then the sequence ; of open sets witnesses the + fact that # is a null set. Ð We would like to extend our measure to null sets # and say ` # = 0. The ( ) proposition we have just proved shows that such an extension would be countably additive on the null sets. But we do not need this extension for the moment,

50 2.3. Infinite sequences so we postpone it. Still, given a probability measure % over -, it becomes meaningful to say that a certain property holds with probability 1. What this means is that the set of those sequences that do not have this property is a null set. Following the idea of Martin-Löf, we would like to call a sequence nonrandom, if it is contained in some simple null set; that is it has some easily definable property with probability 0. As we have seen in part 1 of Example 2.3.11, it is important to insist on simplicity, otherwise (say with respect to the measure _) every sequence might be contained in a null set, namely the one-element set consisting of itself. But most of these sets are not defined simply at all. An example of a simply defined null set is given is part 2 of Example 2.3.11. These reflections justify the following definition, in which “simple” is specified as “constructive”. Definition 2.3.13 Let ` be a computable measure. A set # - is called a ⊆ constructive null set if there is recursively enumerable set Γ N Σ∗ with the Γ = Γ = ⊆ × property that denoting ; F : ;, F and ; F Γ; F- we have ;{ ( ) ∈ } ∈ # ;, and ` ; 2− . y ⊆ ; ( ) ≤ Ð InÑ words, the difference between the definition of null sets and constructive = null sets is that the sets ; F Γ; F- here are required to be constructive open, moreover, in such a way that∈ from ; one can compute the program gen- Ð erating ;. In even looser words, a set is a constructive null set if for any Y > 0 one can construct effectively a union of cylinder sets containing it, with total measure Y. ≤ Now we are in a position to define random infinite sequences. Definition 2.3.14 (Random sequence) An infinite sequence b is random with respect to a probability measure % if and only if b is not contained in any constructive null set with respect to %. y It is easy to see that the set of nonrandom sequences is a null set. Indeed, there is only a countable number or constructive null sets, so even their union is a null set. The following theorem strengthens this observation significantly. Theorem 2.3.1 Let us fix a computable probability measure %. The set of all nonrandom sequences is a constructive null set. Thus, there is a universal constructive null set, containing all other constructive null sets. A sequence is random when it does not belong to this set. Proof of Theorem 2.3.1. The proof uses the projection technique that has appeared several times in this book, for example in proving the existence of a universal lower semicomputable semimeasure.

51 2. Randomness

We know that it is possible to list all recursively enumerable subsets of the set N Σ into a sequence, namely that there is a recursively enumerable set × ∗ Δ N2 Σ with the property that for every recursively enumerable set Γ ⊆ × ∗ ⊆ N Σ there is an 4 with Γ = ;, F : 4,;,F Δ . We will write × ∗ { ( ) ( ) ∈ } Δ = F : 4,;,F Δ , 4,; { ( ) ∈ } 4,; = F-. F Δ Ø∈ 4,;

We transform the set Δ into another set Δ′ with the following property. ; 1 • For each 4,; we have 4,;,F Δ ` F 2− + . ( ) ∈ ′ ( ) ≤ ; • If for some 4 we have 4,;,F Δ ` F 2− for all ; then for all ; we have Í( ) ∈ ( ) ≤ F : 4,;,F Δ′ = F : 4,;,F Δ . { ( ) ∈ } Í{ ( ) ∈ } This transformation is routine, so we leave it to the reader. By the construction of Δ′, for every constructive null set # there is an 4 with

# 4,;′ . ⊆ ; Ù Deﬁne the recursively enumerable set

Γˆ = ;, F : 4 4,; 4 2,F Δ′ . { ( ) ∃ ( + + ) ∈ } ˆ = Then ; 4 4,;′ 4 2. For all ; we have + + Ð ; 4 1 ; ` F = ` F 2− − − = 2− . ( ) Δ ( ) ≤ F Γˆ 4 4,; 4 2,F ′ 4 Õ∈ ; Õ ( +Õ+ ) ∈ Õ This shows that Γˆ deﬁnes a constructive null set. Let Γ be any other recursively enumerable subset of N Σ that deﬁnes a constructive null set. Then there is × ∗ an 4 such that for all ; we have ; = 4,;′ . The universality follows now from

ˆ ; = 4,;′ 4,;′ 4 2 ;. ⊆ + + ⊆ Ù; Ù; Ù; Ù;

2.3.2 Probability space Now we are ready to extend measure to a much wider range of subsets of -.

52 2.3. Inﬁnite sequences

Definition 2.3.15 Elements of the smallest f-algebra A containing the cylinder sets of - are called the Borel sets of -. The pair -, A is an example of a ( ) measureable space, where the Borel sets are called the measureable sets. A nonnegative sigma-additive function ` over A is called a measure. It is called a probability measure if ` - = 1. If ` is fixed then the triple -, A,` ( ) ( ) is called a measure space. If it is a probability measure then the space is called a probability space. y In the Appendix we cited a central theorem of measure theory, Caratheodory’s extension theorem. It implies that if a measure is defined on an algebra L in a sigma-additive way then it can be extended uniquely to the f-algebra generated by L, that is the smallest f-algebra containing L. We defined a measure ` over - as a nonnegative function ` : Σ∗ R satisfying the equality (2.3.1). Then → + we defined ` F- = ` F , and further extended ` in a f-additive way to all ( ) ( ) elements of the algebra F. Now Caratheodory’s theorem allows us to extend it uniquely to all Borel sets, and thus to define a measureable space -, A,` . ( ) Of course, all null sets in A get measure 0. Open, closed and measureable sets can also be defined in the set of real numbers. Definition 2.3.16 A subset R is open if it is the union of a set of open ⊆ intervals 0 , 1 . It is closed if its complement is open. ( 7 7) The set B of Borel sets of B is defined as the smallest f-algebra containing all open sets. y The pair R, B is another example of a measureable space. ( ) Definition 2.3.17 (Lebesgue measure) Consider the set left-closed intervals of the line (including intervals of the form ,0 . Let L be the set of finite disjoint (−∞ ) unions of such intervals. This is an algebra. We define the function _ over L as follows: _ 0 , 1 = 1 0 . It is easy to check that this is a f-additive ( 7 [ 7 7)) 7 7 − 7 measure and therefore by Caratheodory’s theorem can be extended to the set B of all BorelÐ sets. This functionÍ is called the Lebesgue measure over R, giving us the measureable space R, B, _ . y ( ) Finally, we can define the notion of a measureable function over -. Definition 2.3.18 (Measureable functions) A function 5 : - R is called → measureable if and only if 5 1 A for all B. y − ( ) ∈ ∈ The following is easy to prove. Proposition 2.3.19 Function 5 : - R is measureable if and only if all sets of →

53 2. Randomness the form 5 1 @, = F : 5 F > @ are measureable, where @ is a rational − (( ∞)) { ( ) } number.

2.3.3 Computability Measureable functions are quite general; it is worth introducing some more resticted kinds of function over the set of sequences. Deﬁnition 2.3.20 A function 5 : - R is continuous if and only if for every → F -, for every Y > 0 there is a cylinder set F such that 5 G 5 F < Y ∈ ∋ | ( )− ( )| for all G . ∈ A function 5 : - R is lower semicontinuous if for every @ R the set F → ∈ { ∈ - : 5 F > @ is open. It is upper semicontinuous if 5 is lower semicontinuous. ( ) } − y

The following is easy to verify. Proposition 2.3.21 A function 5 : - R is continuous if and only if it is both → upper and lower semicontinuous. Definition 2.3.22 A function 5 : - R is computable if for every open rational → interval C, { the set 5 1 C, { is a constructive open set of -, uniformly in ( ) − (( )) C, {. y Informally this means that if for the infinite sequence b - we have C < ∈ 5 b < { then from C, { sooner or later we will find a prefix F of b with the ( ) property C < 5 F- < {. ( ) Definition 2.3.23 A function 5 : - R is lower semicomputable if for every → rational @ the set A - : 5 A > @ is a constructive open set of -, uniformly { ∈ ( ) } in @. It is upper semicomputable if 5 is lower semicomputable. y − The following is easy to verify. Proposition 2.3.24 A function - R is computable if and only if it is both lower → and upper semicomputable.

2.3.4 Integral The definition of integral over a measure space is given in the Appendix, in Sub- section A.2.3. Here, we give an exposition specialized to infinite sequences. Definition 2.3.25 A measurable function 5 : - R is called a step function if → its range is finite. The set of step functions will be called E.

54 2.3. Inﬁnite sequences

Given a step function 5 which takes values F7 on sets 7, and a ﬁnite measure `, we deﬁne

` 5 = ` 5 = 5 3` = 5 F ` 3F = F ` . ( ) ( ) ( ) 7 ( 7) ¹ ¹ Õ7 y

Proposition A.2.14, when specified to our situation here, says the following. Proposition 2.3.26 The functional ` defined above on step functions can be extended to the set E of monotonic limits of nonnegative elements of E, by continuity. + The set E is the set of all nonnegative measurable functions. + Now we extend the notion of integral to a wider class of functions. Definition 2.3.27 A measurable function 5 is called integrable with respect to a finite measure ` if ` 5 < and ` 5 < . In this case, we define ` 5 = | |+ ∞ | |− ∞ ` 5 ` 5 . y | |+ − | |− It is easy to see that the mapping 5 ` 5 is linear when 5 runs through the 7→ set of measureable functions with ` 5 < . | | ∞ The following is also easy to check. Proposition 2.3.28 Let ` be a computable measure. a) If 5 is computable then a program to compute ` 5 can be found from the program to compute 5 . b) If 5 is lower semicomputable then a program to lower semicompute ` 5 can be found from a program to lower semicompute 5 .

2.3.5 Randomness tests We can now define randomness tests similarly to Section 2.2. Definition 2.3.29 A function 3 : - R is an integrable test, or expectation- → bounded test of randomness with respect to the probability measure % if it is lower semicomputable and satisfies the condition

3 F 2 ( ) % 3F 1. (2.3.2) ( ) ≤ ¹ It is called a Martin-Löf test, or probability-bounded test, if instead of the latter condition only the weaker one is satisﬁed saying % 3 F > ; < 2 ; for each ( ( ) ) − positive integer ;.

55 2. Randomness

It is universal if it dominates all other integrable tests to within an additive constant. Universal Martin-Löf tests are deﬁned in the same way. y Randomness tests and constructive null sets are closely related.

Theorem 2.3.2 Let % be a computable probability measure over the set ( = Σ∞. There is a correspondence between constructive null sets and randomness tests: a) For every randomness tests 3, the set # = b : 3 b = is a constructive { ( ) ∞} null set. b) For every constructive null set # there is a randomness test 3 with # = b : { 3 b = . ( ) ∞} Proof. Let 3 be a randomness test: then for each 9 the set = b : 3 b > 9 9 { ( ) } is a constructive open set, and by Markov’s inequality (wich is proved in the continuous case just as in the discrete case) we have % 2 9. The sets ( 9) ≤ − 9 witness that # is a constructive null set. Let # be a constructive null set with # ∞ , where is a uniform ⊆ 9=1 9 9 sequence of constructive open sets with % = 2 9. Without loss of generality ( 9) − assume that the sequence is decreasing. ThenÑ the function 3 b = sup 9 : 9 ( ) { b is lower semicomputable and satisﬁes % b : 3 b 9 2 9, so it ∈ 9 } { ( ) ≥ } ≤ − is a Martin-Löf test. Just as in Proposition 2.2.3, it is easy to check that 3 F ( )− 2 log 3 F 2 is an integrable test for some constant 2. ( )− Just as there is a universal constructive null set, there are universal randomness tests. Theorem 2.3.3 For a computable measure %, there is a universal integrable test d b . More exactly, the function b d b is lower semicomputable, satis- % ( ) 7→ % ( ) ﬁes (2.2.1) and for all integrable tests 3 b for %, we have ( ) 3 b <+ d b 3 % . ( ) % ( )+ ( )+ ( ) The proof is similar to the proof Theorem 1.6.3 on the existence of a universal constructive semimeasure. It uses the projection technique and a weighted sum.

2.3.6 Randomness and complexity We hope for a result connecting complexity with randomness similar to Theo- rem 2.2.1. Somewhat surprisingly, there is indeed such a result. This is a sur- prise since in an inﬁnite sequence, arbitrarily large oscillations of any quantity depending on the preﬁxes are actually to be expected.

56 2.3. Inﬁnite sequences

Theorem 2.3.4 Let - = ΣN be the set of inﬁnite sequences. For all computable measures ` over -, we have

d` b =+ sup log ` b1:< b1:< . (2.3.3) ( ) < (− ( )− ( ))

Here, the constant in =+ depends on the computable measure `. Corollary 2.3.30 Let _ be the uniform distribution over the set of inﬁnite binary sequences. Then d_ b =+ sup < b1:< . (2.3.4) ( ) < ( − ( )) In other words, an inﬁnite binary sequence is random (with respect to the uniform distribution) if and only if the complexity b of its initial segments ( 1:<) of length < never decreases below < by more than an additive constant.

Proof. To prove <+ , deﬁne the function

m A ` m b1:< ` m b1:< ` 5` b = 1A- b ( | ) = ( | ) sup ( | ) . ` A ` b ` b ( ) A ( ) < 1:< ≥ < 1:< Õ ( ) Õ ( ) ( ) The function b 5 b is lower semicomputable with `b 5 b 1, and hence 7→ ` ( ) ` ( ) ≤

d` b >+ log 5 b >+ sup log ` b1:< b1:< ` . ( ) ( ) < (− ( )− ( | ))

The proof of <+ reproduces the proof of Theorem 5.2 of [18]. Let us abbrevi- ate: 1 b = 1 b . G ( ) G- ( ) Since 3 b only takes integer values and is lower semicomputable, there are ⌈ ( )⌉ computable sequences G Σ and 9 N with 7 ∈ ∗ 7 ∈ 1 3 b = 97 97 2⌈ ( ) ⌉ sup 2 1G7 b 2 1G7 b 7 ( ) ≥ 2 ( ) Õ7 with the property that if 7 < 8 and 1 b = 1 b = 1 then 9 < 9 . The G7 ( ) G8 ( ) 7 8 inequality follows from the fact that for any ﬁnite sequence <1 < <2 < ..., < < 9 2 8 2 max 2 8 . The function W G = = 2 7 is lower semicomputable. 8 ≤ 8 ( ) G7 G With it, we have Í Í 297 1 b = 1 b W G . G7 ( ) G ( ) ( ) 7 G Σ Õ Õ∈ ∗

57 2. Randomness

Since `2 3 2, we have ` G W G 2, hence ` G W G <∗ m G , that is ⌈ ⌉ ≤ G ( ) ( ) ≤ ( ) ( ) ( ) W G <∗ m G ` G . It follows that ( ) ( )/ ( ) Í 3 b m G m b1:< 2 ( ) <∗ sup 1G b ( ) = sup ( ) . G Σ ( ) ` G < ` b1:< ∈ ∗ ( ) ( ) Taking logarithms, we obtain the <+ part of the theorem.

2.3.7 Universal semimeasure, algorithmic probability Universal semimeasures exist over inﬁnite sequences just as in the discrete space.

Definition 2.3.31 A function ` : Σ∗ R is a semimeasure if it satisfies → + ` F ` FA , ( ) ≥ ( ) A Σ Õ∈ ` Λ 1. ( ) ≤ If it is lower semicomputable we will call the semimeasure constructive. y These requirements imply, just as for measures, the following generalization to an arbitrary prefix-free set Σ : ⊆ ∗ ` F 1. ( ) ≤ F Õ∈ Standard technique gives the following: Theorem 2.3.5 (Universal semimeasure) There is a universal constructive semimeasure `, that is a constructive semimeasure with the property that for every other constructive semimeasure a there is a constant 2 > 0 with ` F 2 a F . a ( ) ≥ a ( ) Definition 2.3.32 Let us fix a universal constructive semimeasure over - and denote it by " F . y ( ) There is a graphical way to represent a universal semimeasure, via monotonic Turing machines, which can be viewed a generalization of self-delimiting machines. Definition 2.3.33 A Turing machine T is monotonic if it has no input tape or output tape, but can ask repeatedly for input symbols and can emit repeatedly output symbols. The input is assumed to be an infinite binary string, but it is not assumed that T will ask for all of its symbols, even in infinite time. If the finite or infinite input string is > = > > , the (finite or infinite) output string 1 2 · · · is written as ) > Σ ΣN. y ( ) ∈ ∗ ∪ 58 2.3. Infinite sequences

Just as we can generalize self-delimiting machines to obtain the notion of monotonic machines, we can generalize algorithmic probability, defined for a discrete space, to a version defined for infinite sequences. Imagine the monotonic Turing machine T computing the function ) > as in the definition, and assume ( ) that its input symbols come from coin-tossing. Definition 2.3.34 (Algorithmic probability over infinite sequences) For a string F Σ , let % F be the probability of ) c F, when c = c c is the ∈ ∗ ) ( ) ( ) ⊒ 1 2 · · · infinite coin-tossing sequence. y We can compute

: > % F = 2− ( ), ) ( ) ) > F,> minimal ( ) ⊒ Õ where “minimal” means that no prefix > of > gives ) > F. This expression ′ ( ′) ⊒ shows that % F is lower semicomputable. It is also easy to check that it is a ) ( ) semimeasure, so we have a constructive semimeasure. The following theorem is obtained using a standard construction: Theorem 2.3.6 Every constructive semimeasure ` F can be represented as ` F = ( ) ( ) % F with the help of an appropriate a monotonic machine T. ) ( ) This theorem justifies the following. Definition 2.3.35 From now on we will call the universal semimeasure " F ( ) also the algorithmic probability. A monotonic Turing machine T giving % F = ) ( ) " F will be called an optimal machine. y ( ) We will see that log " F should also be considered a kind of description − ( ) complexity: Definition 2.3.36 Denote " F = log " F . ( ) − ( ) y How does " F compare to F ? ( ) ( ) Proposition 2.3.37 We have the bounds

" F <+ F <+ " F : F . ( ) ( ) ( )+ ( ( )) Proof. To prove " F <+ F deﬁne the lower semicomputable function ( ) ( ) ` F = sup m FG , ( ) ( ) ( G ( Õ∈

59 2. Randomness

By its deﬁnition this is a constructive semimeasure and it is m F . This implies ≥ ( ) " F >∗ m F , and thus " F <+ F . ( ) ( ) ( ) ( ) For the other direction, deﬁne the following lower semicomputable function a over Σ∗:

a F = m : F " F . ( ) ( ( )) · ( ) To show that it is a semimeasure compute:

a F = m < " F m < 1. ( ) ( ) ( ) ≤ ( ) ≤ F Σ < F Σ< < Õ∈ ∗ Õ Õ∈ Õ It follows that m F >∗ a F and hence F <+ " F : F . ( ) ( ) ( ) ( )+ ( ( )) 2.3.8 Randomness via algorithmic probability Fix a computable measure % F over the space -. Just as in the discrete case, ( ) the universal semimeasure gives rise to a Martin-Löf test. Deﬁnition 2.3.38 Denote

" b1:< d%′ b = log sup ( ) . ( ) < % b ( 1:<) y

By Proposition 2.3.37, this function is is >∗ d b , defined in Theorem 2.3.3. % ( ) It can be shown that it is no longer expectation-bounded. But the theorem below shows that it is still a Martin-Löf (probability-bounded) test, so it defines the same infinite random sequences. Theorem 2.3.7 The function d b is a Martin-Löf test. %′ ( ) Proof. Lower semicomputability follows from the form of definition. By standard methods, produce a list of finite strings G1,G2,... with: a) " G % G > 2; ( 7)/ ( 7) b) The cylinder sets G - cover of , and are disjoint (thus the set G ,G ,... 7 ; { 1 2 } is prefix-free). We have

; ; % = % G < 2− " G 2− . ( ;) ( 7) ( 7) ≤ Õ7 Õ7

60 2.3. Inﬁnite sequences

Just as in the discrete case, the universality of " F implies ( ) " F <+ log % F % . ( ) − ( )+ ( ) On the other hand,

sup log % b1:< " b1:< < − ( )− ( ) is a randomness test. So for random b, the value " b remains within a ( 1:<) constant of log % F : it does not oscillate much. − ( )

3 Information

3.1 Information-theoretic relations

In this subsection, we use some material from [23].

3.1.1 The information-theoretic identity Information-theoretic entropy is related to complexity in both a formal and a more substantial way.

Classical entropy and classical coding Entropy is used in information theory in order to characterize the compressibility of a statistical source. Deﬁnition 3.1.1 The entropy of a discrete probability distribution % is deﬁned as H % = % F log % F . ( ) − F ( ) ( ) Õ y

A simple argument shows that this quantity is non-negative. It is instructive to recall a more general inequality, taken from [13], implying it:

Theorem 3.1.1 Let 07, 17 > 0, 0 = 7 07, 1 = 7 17. Then we have Í 07 Í 0 07 log 0 log , (3.1.1) 17 ≥ 1 Õ7 with equality only if 07 17 is constant. In particular, if 7 07 = 1 and 7 17 1 0/7 ≤ then we have 7 07 log 0. 17 ≥ Í Í Proof. Exercise,Í an application of Jensen’s inequality to the concave function log F.

63 3. Information

If % is a probability distribution and & F 0, & F 1 then this ( ) ≥ F ( ) ≤ inequality implies Í % % F log & F . ( ) ≤ F ( ) ( ) Õ We can interpret this inequality as follows. We have seen in Subsection 1.6.3 that if F H is a prefix code, mapping the objects F into binary strings H that are 7→ F F not prefixes of each other then 2 : HF 1. Thus, substituting & F = 2 : HF F − ( ) ≤ ( ) − ( ) above we obtain Shannon’s result: Í Theorem 3.1.2 (Shannon) If F H is a prefix code then for its expected code- 7→ F word length we have the lower bound:

% F : H % . ( ) ( F ) ≥ ( ) ÕF Entropy as expected complexity

Applying Shannon’s theorem 3.1.2 to the code obtained by taking HF as the shortest self-delimiting description of F, we obtain the inequality

% F F H % ( ) ( ) ≥ ( ) ÕF On the other hand, since m F is a universal semimeasure, we have m F >∗ ( ) ( ) % F : more precisely, F <+ log % F % , leading to the following theo- ( ) ( ) − ( )+ ( ) rem. Theorem 3.1.3 For a computable distribution % we have

H % % F F <+ H % % . (3.1.2) ( ) ≤ ( ) ( ) ( )+ ( ) ÕF Note that H % is the entropy of the distribution % while % is just the ( ) ( ) description complexity of the function F % F , it has nothing to do directly 7→ ( ) with the magnitudes of the values % F for each F as real numbers. These two ( ) relations give % =+ % F F % , ( ) F ( ) ( )− ( ) Õ the entropy is within an additive constant equal to the expected complexity. Our intended interpretation of F as information content of the individual object ( ) F is thus supported by a tight quantitative relationship to Shannon’s statistical concept.

64 3.1. Information-theoretic relations

Identities, inequalities The statistical entropy obeys meaningful identities which immediately follow from the definition of conditional entropy. Definition 3.1.2 Let - and . be two discrete random variables with a joint distribution. The conditional entropy of . with respect to - is defined as

H . - = % - = F % . = G - = F log % . = G - = F . ( | ) − F [ ] G [ | ] [ | ] Õ Õ The joint entropy of - and . is deﬁned as the entropy of the pair -, . . y ( ) The following additivity property is then veriﬁable by simple calculation:

H -, . = H - H . - . (3.1.3) ( ) ( )+ ( | ) Its meaning is that the amount of information in the pair -, . is equal to the ( ) amount of information in - plus the amount of our residual uncertainty about . once the value of - is known. Though intuitively expected, the following identity needs proof:

H . - H . . ( | ) ≤ ( ) We will prove it after we define the difference below. Definition 3.1.3 The information in - about . is defined by

I - : . = H . H . - . ( ) ( )− ( | ) y This is the amount by which our knowledge of - decreases our uncertainty about .. The identity below comes from equation (3.1.3).

I - : . = I . : - = H - H . H -, . ( ) ( ) ( )+ ( )− ( ) The meaning of the symmetry relation is that the quantity of information in . about - is equal to the quantity of information in - about .. Despite its simple derivation, this fact is not very intuitive; especially since only the quantities are equal, the actual contents are generally not (see [22]). We also call I - : . the ( ) mutual information of - and .. We can also write % - = F, . = G I - : . = % - = F, . = G log [ ] . (3.1.4) ( ) [ ] % - = F % . = G ÕF,G [ ] [ ] 65 3. Information

Proposition 3.1.4 The information I - : . is nonnegative and is 0 only if -, . ( ) are independent. Proof. Apply Theorem 3.1.1 to (3.1.4). This is 0 only if % - = F, . = G = [ ] % - = F % . = G for all F,G. [ ] [ ] The algorithmic addition theorem For the notion of algorithmic entropy F defined in Section 1.6, it is a seri- ( ) ous test of fitness to ask whether it has an additivity property analogous to the identity (3.1.3). Such an identity would express some deeper relationship than the trivial identity (3.1.3) since the conditional complexity is defined using the notion of conditional computation, and not by an algebraic formula involving the unconditional definition. A literal translation of the identity (3.1.3) turns out to be true for the function F as well as for Kolmogorov’s complexity F with good approximation. But ( ) ( ) the exact additivity property of algorithmic entropy is subtler. Theorem 3.1.4 (Addition) We have

F,G =+ F G F, F . (3.1.5) ( ) ( )+ ( | ( )) Proof. We first prove <+ . Let be the following s.d. interpreter. The machine computing > tries to parse > into > = C{ so that ) C is defined, then outputs ( ) ( ) the pair ) C ,) {, ) C ,: C . If C is a shortest description of F and { a ( ( ) ( h ( ) ( )i)) shortest description of G given the pair F, F then the output of is the pair ( ( )) F,G . ( ) Now we prove >+ . Let 2 be a constant (existing in force of Lemma 1.7.3) such that for all F we have 2 F,G 2 F 2. Let us define the semicomputable G − ( ) ≤ − ( )+ function 5 H,G by ( ) Í 9 F,G 2 5 F, 9 ,G = 2 − ( )− . (( ) ) Then for 9 = F we have 5 H,G 1, hence the generalization (1.6.13) ( ) G ( ) ≤ of the coding theorem implies G F, F <+ F,G F . Í ( | ( )) ( )− ( ) Recall the definition of F∗ as the first shortest description of F. Corollary 3.1.5 We have F∗,G =+ F,G . (3.1.6) ( ) ( ) The Addition Theorem implies

F,G <+ F G F <+ F G ( ) ( )+ ( | ) ( )+ ( ) 66 3.1. Information-theoretic relations while these relations do not hold for Kolmogorov’s complexity F , as pointed ( ) out in the discussion of Theorem 1.4.2. For an ordinary interpreter, extra information is needed to separate the descriptions of F and G. The self-delimiting interpreter accepts only a description that provides the neccesary information to determine its end. The term F cannot be omitted from the condition in G F, F in the ( ) ( | ( )) identity (3.1.5). Indeed, it follows from equation (1.7.2) that with G = F , ( ) we have F,G F =+ 0. But F F can be quite large, as shown in ( )− ( ) ( ( ) | ) the following theorem given here without proof. Theorem 3.1.5 (see [17]) There is a constant 2 such that for all < there is a binary string F of length < with

F F log < log log < 2. ( ( ) | ) ≥ − − We will prove this theorem in Section 3.3. For the study of information relations, let us introduce yet another notation. Deﬁnition 3.1.6 For any functions 5,6,ℎ with values in S, let

5 6 mod ℎ mean that there is a constant 2 such that 5 F 6 F , ℎ F 2 for all F. We ( ( ) | ( ) ( )) ≤ will omit modℎ if ℎ is not in the condition. The sign will mean inequality in ≍ both directions. y The meaning of F G is that the objects F and G hold essentially the same ≍ information. Theorem 1.4.1 implies that if 5 6 then we can replace 5 by 6 ≍ wherever it occurs as an argument of the functions or . As an illustration of this deﬁnition, let us prove the relation

F∗ F, F ≍ h ( )i which implies G F, F =+ G F , and hence ( | ( )) ( | ∗)

F,G F =+ G F∗ . (3.1.7) ( )− ( ) ( | ) The direction holds because we can compute F = ) F and F = : F ( ∗) ( ) ( ∗) from F . On the other hand, if we have F and F then we can enumerate the ∗ ( ) set F of all shortest descriptions of F. By the corollary of Theorem 1.7.1, the ( ) number of elements of this set is bounded by a constant. The estimate (1.6.4) implies F F, F =+ 0. ( ∗ | ( ))

67 3. Information

Though the string F and the pair F, F contain the same information, ∗ ( ( )) they are not equivalent under some stronger criteria on the decoder. To obtain F from the shortest description may take extremely long time. But even if this time B is acceptable, to ﬁnd any description of length F for F might require F ( ) about B2 ( ) steps of search. The Addition Theorem can be made formally analogous to its information- theoretic couterpart using Chaitin’s notational trick. Deﬁnition 3.1.7 Let G F = G F . y ∗( | ) ( | ∗) Of course, F = F . Now we can formulate the relation (3.1.7) as ∗ ( ) ( )

∗ F,G =+ ∗ F ∗ G F . ( ) ( )+ ( | ) However, we prefer to use the function G F since the function G F is not ( | ) ∗ ( | ) semicomputable. Indeed, if there was a function ℎ F,G upper semicomputable ( ) in F,G such that F,G F =+ ℎ F,G then by the argument of the proof h i ( )− ( ) ( ) of the Addition Theorem, we would have G F <+ ℎ F,G which is not true, ( | ) ( ) according to remark after the proof of the Addition Theorem. Deﬁnition 3.1.8 We deﬁne the algorithmic mutual information with the same formula as classical information:

F : G = G G F . ( ) ( )− ( | ) y Since the quantity F : G is not quite equal to F F G (in fact, ( ) ( ) − ( | ) Levin showed (see [17]) that they are not even asymptotically equal), we prefer to deﬁne information also by a symmetrical formula. Deﬁnition 3.1.9 The mutual information of two objects F and G is

∗ F : G = F G F,G . ( ) ( )+ ( )− ( ) y

The Addition Theorem implies the identity

∗ F : G =+ F∗ : G =+ G∗ : F . ( ) ( ) ( ) Let us show that classical information is indeed the expected value of algorithmic information, for a computable probability distribution >:

68 3.1. Information-theoretic relations

Lemma 3.1.10 Given a computable joint probability mass distribution > F,G ( ) over F,G we have ( ) I - : . > <+ > F,G ∗ F : G (3.1.8) ( )− ( ) F G ( ) ( ) Õ Õ <+ I - : . 2 > , ( )+ ( ) where > is the length of the shortest preﬁx-free program that computes > F,G ( ) ( ) from input F,G . ( ) Proof. We have

> F,G ∗ F : G =+ > F,G F G F,G . ( ) ( ) ( )[ ( )+ ( )− ( )] ÕF,G ÕF,G Deﬁne > F,G = > F and > F,G = > G to obtain G ( ) 1 ( ) F ( ) 2 ( )

>ÍF,G F : G =+ >1 FÍ F >2 G G > F,G F,G . F,G ( ) ( ) F ( ) ( )+ G ( ) ( )− F,G ( ) ( ) Õ Õ Õ Õ The distributions >7 (7 = 1, 2) are computable. We have seen in (3.1.2) that H ? <+ ? F F <+ H ? ? . ( ) F ( ) ( ) ( )+ ( ) Hence, H > <+ > F F <+ H > > . (7 = 1, 2), and H > <+ ( 7) F 7 ( ) ( ) ( 7) + ( 7) ( ) > F,GÍ F,G <+ H > > . On the other hand, the probabilistic mutual F,G ( ) ( ) ( )+ ( ) information is expressedÍ in the entropies by I - : . = H > H > H > . ( ) ( 1)+ ( 2)− ( ) ByÍ construction of the ? ’s above, we have > , > <+ > . Since the 7 ( 1) ( 2) ( ) complexities are positive, substitution establishes the lemma. Remark 3.1.11 The information F : G can be written as ∗( ) m F,G ∗ F : G =+ log ( ) . ( ) m F m G ( ) ( )

Formally, ∗ F : G looks like dm m F,G m m with the function d in- ( ) × (( ) | × ) · (·) troduced in (2.2.2). Thus, it looks like F : G measures the deficiency of ∗( ) randomness with respect to the distribution m m. The latter distribution ex- × presses our “hypothesis” that F,G are “independent” from each other. There is a serious technical problem with this interpretation: the function d F was only % ( ) defined for computable measures %. Though the definition can be extended, it is not clear at all that the expression d F will also play the role of universal test % ( ) for arbitrary non-computable distributions. Levin’s theory culminating in [33] develops this idea, and we return to it in later parts of these notes. y

69 3. Information

Let us examine the size of the defects of naive additivity and information symmetry. Corollary 3.1.12 For the defect of additivity, we have

F G F F,G =+ F : G F <+ F F . (3.1.9) ( )+ ( | )− ( ) ( ( ) | ) ( ( ) | ) For information asymmetry, we have

F : G G : F =+ G : F G F : G F . (3.1.10) ( )− ( ) ( ( ) | )− ( ( ) | ) Proof. Immediate from the addition theorem. Theorem 3.1.6 (Levin) The information F : G is not even asymptotically sym- ( ) metric. Proof. For some constant 2, assume F = < and F F log < log log < | | ( ( ) | ) ≥ − − 2. Consider F,F and F . Then we can check immediately the relations ∗ ( ) F : F <+ 3 log log < < log < log log < 2 <+ F∗ : F . ( ( )) − − ( ( )) and F : F =+ F : F∗ . ( ( ) ) ( ( ) ) It follows that symmetry breaks down in an exponentially great measure either on the pair F, F or the pair F , F . ( ( )) ( ∗ ( )) Data processing The following useful identity is also classical, and is called the data processing identity: I / : -, . = I / : - I / : . - . (3.1.11) ( ( )) ( )+ ( | ) Let us see what corresponds to this for algorithmic information. Here is the correct conditional version of the addition theorem: Theorem 3.1.7 We have

F,G H =+ F H G F, F H , H . (3.1.12) ( | ) ( | )+ ( | ( | ) ) Proof. The same as for the unconditional version. Remark 3.1.13 The reader may have guessed the inequality

F,G H =+ F H G F, F , H , ( | ) ( | )+ ( | ( ) ) but it is incorrect: taking H = F, G = F , the left-hand side equals F F , ( ) ( ∗ | ) and the right-hand side equals F F F F ,F =+ 0. y ( | )+ ( ( ) | ∗ ) 70 3.1. Information-theoretic relations

The following inequality is a useful tool. It shows that conditional complexity is a kind of one-sided distance, and it is the algorithmic analogue of the well- known (and not diﬃcult classical inequality)

H - . H / . H - / . ( | ) ≤ ( | )+ ( | ) Theorem 3.1.8 (Simple triangle inequality) We have

H F <+ G, H F <+ G F H G . (3.1.13) ( | ) ( | ) ( | )+ ( | ) Proof. According to (3.1.12), we have

G, H F =+ G F H G, G F ,F . ( | ) ( | )+ ( | ( | ) ) The left-hand side is >+ H F , the second term of the right-hand side is <+ ( | ) H G . ( | ) Equation (3.1.12) justiﬁes the following. Deﬁnition 3.1.14

∗ F : G H = F H G H F,G H . (3.1.14) ( | ) ( | )+ ( | )− ( | ) y

Then we have

∗ F : G H =+ G H G F, F H , H ( | ) ( | )− ( | ( | ) ) =+ F H F G, G H , H . ( | )− ( | ( | ) ) Theorem 3.1.9 (Algorithmic data processing identity) We have

∗ H : F,G =+ ∗ H : F ∗ H : G F∗ . (3.1.15) ( ( )) ( )+ ( | ) Proof. We have

∗ H : F,G =+ F,G H F,G,H ( ( )) ( )+ ( )− ( ) ∗ H : F =+ F H F, H ( ) ( )+ ( )− ( ) ∗ H : F,G ∗ H : F =+ F,G F, H F,G,H F ( ( )) − ( ) ( )+ ( )− ( )− ( ) =+ G F∗ H F∗ G, H F∗ ( | )+ ( | )− ( | ) =+ ∗ H : G F∗ , ( | ) where we used additivity and the deﬁnition (3.1.14) repeatedly.

71 3. Information

Corollary 3.1.15 The information ∗ is monotonic:

∗ F : G, H >+ ∗ F : G . (3.1.16) ( ( )) ( ) The following theorem is also sometimes useful. Theorem 3.1.10 (Starry triangle inequality) For all F,G,H, we have

F G∗ <+ F, H G∗ <+ H G∗ F H∗ . (3.1.17) ( | ) ( | ) ( | )+ ( | ) Proof. Using the algorithmic data-processing identity (3.1.15) and the deﬁni- tions, we have

H H G∗ F H∗ F G, G H∗ , H∗ ( )− ( | )+ ( | )− ( | ( | ) ) =+ ∗ G : H ∗ G : F H∗ =+ ∗ G : F, H ( )+ ( | ) ( ( )) =+ F, H F, H G∗ , ( )− ( | ) which gives, after cancelling H F H on the left-hand side with F, H ( )+ ( | ∗) ( ) on the right-hand side, to which it is equal according to additivity, and changing sign: F, H G∗ =+ H G∗ F G, G H∗ , H∗ . ( | ) ( | )+ ( | ( | ) ) From here, we proceed as in the proof of the simple triangle inequality (3.1.13)

3.1.2 Information non-increase In this subsection, we somewhat follow the exposition of [23]. The classical data processing identity has a simple but important application. Suppose that the random variables /,-,. form a Markov chain in this order. Then / : . - = 0 and hence / : -, . = / : - . In words: all ( | ) ( ( )) ( ) information in / about . is coming through -: a Markov transition from - to . cannot increase the information about /. Let us try to ﬁnd the algorithmic counterpart of this theorem. Here, we rigorously show that this is the case in the algorithmic statistics setting: the information in one object about another cannot be increased by any deterministic algorithmic method by more than a constant. With added randomization this holds with overwhelming probability. For more elaborate versions see [31, 33].

72 3.1. Information-theoretic relations

Suppose we want to obtain information about a certain object F. It does not seem to be a good policy to guess blindly. This is conﬁrmed by the following inequality. F:G m G 2 ( ) <∗ 1 (3.1.18) G ( ) Õ F:G which says that for each F, the expected value of 2 ( ) is small, even with respect to the universal constructive semimeasure m G . The proof is immediate ( ) if we recognize that by the Coding Theorem,

G F F:G 2− ( | ) 2 ( ) =∗ m G ( ) and use the estimate (1.6.10). We prove a strong version of the information non-increase law under deterministic processing (later we need the attached corollary):

Theorem 3.1.11 Given F and H, let ? be a program computing H from F∗. Then we have ∗ G : H <+ ∗ G : F ? . ( ) ( )+ ( ) Proof. By monotonicity (3.1.16) and the data processing identity (3.1.15):

∗ G : H <+ ∗ G : F, H =+ ∗ G : F ∗ G : H F∗ . ( ) ( ( )) ( )+ ( | ) By deﬁnition of information and the deﬁnition of ?:

∗ G : H F∗ <+ H F∗ <+ ? . ( | ) ( | ) ( )

Randomized computation can increase information only with negligible probability. Suppose that H is obtained from F by some randomized computation. The probability > H F of obtaining H from F is a semicomputable distribution over ( | ) the H’s. The information increase H : G F : G satisﬁes the theorem ∗ ( ) − ∗ ( ) below. Theorem 3.1.12 For all F,G,H we have

H:G F:G m H F∗ 2 ∗ ( )− ∗ ( ) <∗ m H F∗,G, G F∗ . ( | ) ( | ( | )) Therefore it is upperbounded by m H F <∗ m H F . ( | ) ( | ∗)

73 3. Information

Remark 3.1.16 The theorem implies

H:G F:G m H F∗ 2 ∗ ( )− ∗ ( ) <∗ 1. (3.1.19) ( | ) ÕH This says that the m F -expectation of the exponential of the increase is (· | ∗) bounded by a constant. It follows that for example, the probability of an in- 3 crease of mutual information by the amount 3 is <∗ 2− . y Proof. We have, by the data processing inequality:

∗ G : F, H ∗ G : F =+ ∗ G : H F∗ =+ H F∗ H G, G F∗ ,F∗ . ( ( )) − ( ) ( | ) ( | )− ( | ( | ) ) Hence, using also G : F, H >+ G : H by monotonicity: ∗( ( )) ∗( )

∗ G : H ∗ G : F H F∗ <+ H G, G F∗ ,F∗ . ( )− ( )− ( | ) − ( | ( | ) ) Putting both sides into the exponent gives the statement of the theorem.

Remark 3.1.17 The theorem on information non-increase and its proof look similar to the theorems on randomness conservation. There is a formal connection, via the observation made in Remark 3.1.11. Due to the diﬃculties mentioned there, we will explore the connection only later in our notes. y

3.2 The complexity of decidable and enumerable sets

If l = l 1 l 2 is an inﬁnite binary sequence, then we may be interested ( ) ( )··· in how the complexity of the initial segments

l 1 : < = l 1 l < ( ) ( )··· ( ) grows with <. We would guess that if l is “random” then l 1 : < is approx- ( ( )) imately <; this question will be studied satisfactorily in later sections. Here, we want to look at some other questions. Can we determine whether the sequence l is computable, by just looking at the complexity of its initial segments? It is easy to see that if l is computable then l 1 : < <+ log <. But of course, if l 1 : < <+ log < then it is ( ( )) ( ( )) not necessarily true yet that l is computable. Maybe, we should put < into the condition. If l is computable then l 1 : < < <+ 0. Is it true that ( ( ) | ) l 1 : < < <+ 0 for all < then indeed l is computable? Yes, but the proof ( ( ) | ) is quite diﬃcult (see either its original, attributed to Albert Meyer in [37], or

74 3.2. The complexity of decidable and enumerable sets in [61]). The suspicion arises that when measuring the complexity of starting segments of infinite sequences, neither Kolmogorov complexity nor its prefix version are the most natural choice. Other examples will support the suspicion, so let us introduce, following Loveland’s paper [37], the following variant. Definition 3.2.1 Let us say that a program > decides a string F = F 1 ,F 2 ,... on a Turing machine ) up to < if for all 7 1,...,< we ( ( ) ( ) ) ∈ { } have ) >, 7 = F 7 . ( ) ( ) Let us further define the decision complexity

F; < = min : > : > decides F on ) up to < . ) ( ) { ( ) } y

As for the Kolmogorov complexity, an invariance theorem holds, and there is an optimal machine )0 for this complexity. Again, we omit the index ), assuming that such an optimal machine has been fixed. The differences between F , F < and F; < are interesting to ex- ( ) ( | ) ( ) plore; intuitively, the important difference is that the program achieving the decision complexity of F does not have to offer precise information about the length of F. We clearly have

F < <+ F; < <+ F . ( | ) ( ) ( ) Examples that each of these inequalities can be strict, are left to exercises. Remark 3.2.2 If l 1 : < is a segment of an infinite sequence l then we can ( ) write l; < ( ) instead of l 1 : < ; < without confusion, since there is only one way to ( ( ) ) understand this expression. y Decision complexity offers an easy characterization of decidable infinite sequences. Theorem 3.2.1 Let l = l 1 , l 2 ,... be an infinite sequence. Then l is ( ( ) ( ) ) decidable if and only if l; < <+ 0. (3.2.1) ( ) Proof. If l is decidable then (3.2.1) is obviously true. Suppose now that (3.2.1) holds: there is a constant 2 such that for all < we have l 1 : < ; < < 2. ( ( ) ) 75 3. Information

Then there is a program > of length 2 with the property that for inﬁnitely ≤ many <, decides F on the optimal machine )0 up to <. Then for all 7, we have ) >, 7 = l 7 , showing that l is decidable. 0 ( ) ( ) Let us use now the new tool to prove a somewhat more surprising result. Let l be a 0-1 sequence that is the indicator function of a recursively enumerable set. In other words, the set 7 : l 7 = 1 is recursively enumerable. As we { ( ) } know such sequences are not always decidable, so l 1 : < ; < is not going to ( ( ) ) be bounded. How fast can it grow? The following theorem gives exact answer, showing that it grows only logarithmically. Theorem 3.2.2 (Barzdin, see [2]) a) Let l be the indicator function of a recursively enumerable set . Then we have

l; < <+ log <. ( ) b) The set can be chosen such that for all < we have l; < log <. ( ) ≥ Proof. Let us prove (a) first. Let <′ be the first power of 2 larger than <. Let 9 < = 7 < l 7 . For a constant 3, let > = > <,3 be a program that contains ( ) ≤ ′ ( ) ( ) an initial segment ? of size 3 followed by a the number 9 < in binary notation, ( ) and paddedÍ to log < 3 with 0’s. The program ? is self-delimiting, so it sees ⌈ ⌉+ where 9 < begins. ( ) The machine ) >, 7 works as follows, under the control of ?. From the 0 ( ) length of the whole >, it determines < . It begins to enumerate l 1 : < until ′ ( ′) it found 9 < 1’s. Then it knows the whole l 1 : < , so it outputs l 7 . ( ) ( ′) ( ) Let us prove now (b). Let us list all possible programs ? for the machine )0 as ? = 1, 2,.... Let l ? = 1 if and only if ) ?,? = 0. The sequence l is ( ) 0 ( ) obviously the indicator function of a recursively enumerable set. To show that l has the desired property, assume that for some < there is a > with ) >, 7 = l 7 0 ( ) ( ) for all 7 <. Then >>< since l > is defined to be different from ) >, > . It ≤ ( ) 0 ( ) follows that : > log <. ( ) ≥ Decision complexity has been especially convenient for the above theorem. Neither l 1 : < nor l 1 : < < would have been suitable to formulate ( ( )) ( ( ) | ) such a sharp result. To analyze the phenomenon further, we introduce some more concepts. Definition 3.2.3 Let us denote, for this section, by the set of those binary strings > on which the optimal prefix machine ) halts: B = > : ) > halts in < B steps , { ( ) } (3.2.2) = ∞.

76 3.2. The complexity of decidable and enumerable sets

Let j = j (3.2.3) be the infinite sequence that is the indicator function of the set , when the latter is viewed as a set of numbers. y It is easy to see that the set is complete among recursively enumerable sets with respect to many-one reduction. The above theorems show that though it contains an infinite amount of information, this information is not stored in the sequence j densely at all: there are at most log < bits of it in the segment j 1 : < . There is an infinite sequence, though, in which the same information ( ) is stored much more densely: as we will see later, maximally densely. Definition 3.2.4 (Chaitin’s Omega) Let

B : > Ω = 2− ( ), > B (3.2.4) Õ∈ Ω = Ω∞. y Let Ω 1 : < be the sequence of the ﬁrst < binary digits in the expansion of ( ) Ω, and let it also denote the binary number 0.Ω 1 Ω < . Then we have ( )··· ( ) < Ω 1 : < < Ω < Ω 1 : < 2− . ( ) ( )+ Theorem 3.2.3 The sequences Ω and j are Turing-equivalent.

Proof. Let us show first that given Ω as an oracle, a Turing machine can compute j. Suppose that we want to know for a given string > of length 9 whether j > = ( ) 1 that is whether ) > halts. Let B < be the first B for which ΩB > Ω 1 : < . Ifa ( ) B (< ) (: > ) < program > of length < is not in ( ) then it is not in at all, since 2− ( ) = 2 < Ω ΩB. It follows that j 1:2< can be computed from Ω 1 : < . − ( ) ( ) To show that Ω can be computed from , let us define the recursively enumerable set ′ = @ : @ rational, Ω > @ . { } The set ′ is reducible to since the latter is complete among recursively enumerable sets with respect to many-one reduction. On the other hand, Ω is obviously computable from ′. The following theorem shows that Ω stores the information more densely.

Theorem 3.2.4 We have Ω 1 : < >+ <. ( ( )) 77 3. Information

Proof. Let >1 be a self-delimiting program outputting Ω 1 : < . Recall the def- B B ( ) inition of the sets in (3.2.2) and the numbers Ω in (3.2.4). Let Ω1 denote the real number whose binary digits after 0 are given by Ω 1 : < , and let B be ( ) 1 the ﬁrst B with ΩB > Ω . Let F be the ﬁrst string F such that ) > ≠ F for any 1 1 ( ) > B1 . ∈ We have F <. On the other hand, we just computed F from > , so ( 1) ≥ 1 1 F > <+ 0. We found < F <+ > F > <+ > . ( 1 | 1) ≤ ( 1) ( 1)+ ( 1 | 1) ( 1) 3.3 The complexity of complexity

3.3.1 Complexity is sometimes complex This section is devoted to a quantitative estimation of the uncomputability of the complexity function F . Actually, we will work with the preﬁx complexity ( ) F , but the results would be very similar for F . The ﬁrst result shows that ( ) ( ) the value F is not only not computable from F, but its conditional complexity ( ) F F given F is sometimes quite large. How large can it be expected to be? ( ( ) | ) Certainly not much larger than log F 2 log log F , since we can describe ( )+ ( ) any value F using this many bits. But it can come close to this, as shown by ( ) Theorem 3.1.5. This theorem says that for all <, there exists F of length < with

F F >+ log < log log <. (3.3.1) ( ( ) | ) − Proof of Theorem 3.1.5. Let * >, F be the optimal self-delimiting machine for ( ) which G F = min* >,F =G : > . Let A log < be such that if : F = < then a ( | ) ( ) ( ) ≤ ( ) > of length F can be found for which * >, F = F . We will show ( ) ( ) A >+ log < log log <. (3.3.2) − Let us say that > 0, 1 A is suitable for F 0, 1 < if there exists a 9 = * >, F ∈ { } ∈ { } ( ) and a ? 0, 1 9 with * >, Λ = F. Thus, > is suitable for F if it produces the ∈ { } ( ) length of some program of F, not necessarily a shortest program. Let "7 denote the set of those F of length < for which there exist at least 7 diﬀerent suitable > of length A. We will examine the sequence

< 0, 1 = "0 "1 ... " 8 " 8 1 = , { } ⊇ ⊇ ⊇ ⊇ + ∅ where " ≠ . It is clear that 2A 8. To lowerbound 8, we will show that the 8 ∅ ≥ sets "7 decrease only slowly:

log "7 <+ log "7 1 4 log <. (3.3.3) | | | + |+ 78 3.3. The complexity of complexity

We can assume log "7 "7 1 log "7 1, (3.3.4) | \ + | ≥ | |− otherwise (3.3.3) is trivial. We will write a program that ﬁnds an element F of "7 "7 1 with the property F log "7 1. The program works as follows. \ + ( ) ≥ | |− • It ﬁnds 7,< with the help of descriptions of length log < 2 log log <, and A + with the help of a description of length 2 log log <.

• It ﬁnds "7 1 with the help of a description of length log "7 1 log < | + | | + | + + 2 log log <.

• From these data, it determines the set "7 1, and then begins to enumerate + the set "7 "7 1 as F1,F2,.... For each of these elements F@, it knows that \ + there are exactly 7 programs suitable for F@, find all those, and find the shortest program for F produced by these. Therefore it can compute F . ( @) • According to the assumption (3.3.4), there is an F with F log " 1. @ ( @) ≥ | 7 |− The program outputs the first such F@.

The construction of the program shows that its length is <+ log "7 1 4 log <, | + |+ hence for the F we found

log "7 1 F <+ log "7 1 4 log <, | |− ≤ ( ) | + |+ which proves (3.3.3). This implies 8 < 4 log < , and hence (3.3.2). ≥ /( ) 3.3.2 Complexity is rarely complex Let 5 F = F F . ( ) ( ( ) | ) We have seen that 5 F is sometimes large. Here, we will show that the sequen- ( ) ces F for which this happens are rare. Recall that we deﬁned j in (3.2.3) as the indicator sequence of the halting problem. Theorem 3.3.1 We have

j : F >+ 5 F 2.4 log 5 F . ( ) ( )− ( ) In view of the inequality (3.1.18), this shows that such sequences are rare, even in terms of the universal probability, so they are certainly rare if we measure them by any computable distribution. So, we may even call such sequences “exotic”.

79 3. Information

In the proof, we start by showing that the sequences in question are rare. Then the theorem will follow when we make use of the fact that 5 F is com- ( ) putable from j. We need a lemma about the approximation of one measure by another from below. Lemma 3.3.1 For any semimeasure a and measure ` a, let ( = F :2;` F ≤ ; { ( ) ≤ a F . Then ` ( 2 ;. If a F ` F < 2 < then a ( < 2 < 1. ( )} ( ;) ≤ − F ( ( )− ( )) − ( 1) − + The proof is straightforward.Í Let - < = F : 5 F = < . ( ) { ( ) } Lemma 3.3.2 We have 1.2 < m - < <∗ < 2− . ( ( )) Proof. Using the deﬁnition of B in (3.2.2), let

mB F = > : ) > = F in < B steps , ( ) { ( ) } B F = Õlog mB F . ( ) − ( )

Then according to the deﬁnition (1.6.14) we have m F = m F , and F =+ ( ) ∞( ) ( ) F . Let ∞( ) B 9 = min B : Ω 1 : 9 < ΩB , ( ) { ( ) } B 9 ` = m ( ) .

Clearly, B 9 F >+ F . Let us show that B 9 F is a good approximation for ( ) ( ) ( ) ( ) ( ) F for most F. Let ( ) B 9 . 9 = F : F ( ) F 1 . ( ) { ( ) ≤ ( )− } By deﬁnition, for F ∉ . 9 we have ( ) B 9 ( ) F F =+ 0. | ( )− ( )| B 9 On the other hand, applying Lemma 3.3.1 with ` = m ( ), a = m, we obtain

9 1 m . 9 < 2− + . (3.3.5) ( ( )) Note that B 9 ( ) F F, Ω 1 : 9 =+ 0, ( ( ) | ( )) 80 3.3. The complexity of complexity therefore for F ∉ . 9 we have ( ) F F, Ω 1 : 9 =+ 0, ( ( ) | ( )) F F <+ 9 1.2 log 9. ( ( ) | ) +

If < = 9 1.2 log 9 then 9 =+ < 1.2 log <, and hence, if F ∉ . < 1.2 log < + − ( − ) then F F <+ <. Thus, there is a constant 2 such that ( ( ) | ) - < . < 1.2 log < 2 . ( ) ⊆ ( − − ) Using (3.3.5) this gives the statement of the lemma. Proof of Theorem 3.3.1. Since 5 F is computable from Ω, the function ( ) 5 F 2.4 a F = m F 2 ( ) 5 F − ( ) ( ) ( ( )) is computable from Ω. Let us show that it is a semimeasure (within a multiplicative constant). Indeed, using the above lemma:

9 2.4 1.2 a F = a F = 2 9− m - 9 = 9− <∗ 1. ( ) ( ) ( ( )) F 9 F - 9 9 9 Õ Õ ∈Õ( ) Õ Õ

Since m Ω is the universal semimeasure relative to Ω we ﬁnd m F Ω >∗ (· | ) ( | ) a F , hence ( ) F Ω <+ log a F = F 5 F 2.4 log 5 F , ( | ) − ( ) ( )− ( )+ ( ) Ω : F >+ 5 F 2.4 log 5 F . ( ) ( )− ( ) Since Ω is equivalent to j, the proof is complete.

4 Generalizations

4.1 Continuous spaces, noncomputable measures

This section starts the consideration of randomness in continuous spaces and randomness with respect to noncomputable measures.

4.1.1 Introduction The algorithmic theory of randomness is well developed when the underlying space is the set of finite or infinite sequences and the underlying probability distribution is the uniform distribution or a computable distribution. These restrictions seem artificial. Some progress has been made to extend the theory to arbitrary Bernoulli distributions by Martin-Löf in [38], and to arbitrary distributions, by Levin in [30, 32, 33]. The paper [25] by Hertling and Weihrauch also works in general spaces, but it is restricted to computable measures. Similarly, Asarin’s thesis [1] defines randomness for sample paths of the Brownian motion: a fixed random process with computable distribution. The exposition here has been inspired mainly by Levin’s early paper [32] (and the much more elaborate [33] that uses different definitions): let us summarize part of the content of [32]. The notion of a constructive topological space X and the space of measures over X is introduced. Then the paper defines the notion of a uniform test. Each test is a lower semicomputable function `,F 5 F , satisfying 5 F ` 3F 1 for each measure `. There are ( ) 7→ ` ( ) ` ( ) ( ) ≤ also some additional conditions. The main claims are the following. ∫ a) There is a universal test t F , a test such that for each other test 5 there ` ( ) is a constant 2 > 0 with 5 F 2 t F . The deficiency of randomness is ` ( ) ≤ · ` ( ) defined as d F = log t F . ` ( ) ` ( ) b) The universal test has some strong properties of “randomness conservation”: these say, essentially, that a computable mapping or a computable random-

83 4. Generalizations

ized transition does not decrease randomness. c) There is a measure " with the property that for every outcome F we have t F 1. In the present work, we will call such measures neutral. " ( ) ≤ d) Semimeasures (semi-additive measures) are introduced and it is shown that there is a lower semicomputable semi-measure that is neutral (let us assume that the " introduced above is lower semicomputable). e) Mutual information F : G is defined with the help of (an appropriate ( ) version of) Kolmogorov complexity, between outcomes F and G. It is shown that F : G is essentially equal to d" " F,G . This interprets mutual ( ) × ( ) information as a kind of “deficiency of independence”. This impressive theory leaves a number of issues unresolved: 1. The space of outcomes is restricted to be a compact topological space, moreover, a particular compact space: the set of sequences over a finite alphabet (or, implicitly in [33], a compactified infinite alphabet). However, a good deal of modern probability theory happens over spaces that are not even locally compact: for example, in case of the Brownian motion, over the space of continuous functions. 2. The definition of a uniform randomness test includes some conditions (different ones in [32] and in [33]) that seem somewhat arbitrary. 3. No simple expression is known for the general universal test in terms of description complexity. Such expressions are nice to have if they are available. Here, we intend to carry out as much of Levin’s program as seems possible after removing the restrictions. A number of questions remain open, but we feel that they are worth to be at least formulated. A fairly large part of the exposition is devoted to the necessary conceptual machinery. This will also allow to carry further some other initiatives started in the works [38] and [30]: the study of tests that test nonrandomness with respect to a whole class of measures (like the Bernoulli measures). Constructive analysis has been developed by several authors, converging approximately on the same concepts, We will make use of a simplified version of the theory introduced in [57]. As we have not found a constructive measure theory in the literature fitting our purposes, we will develop this theory here, over (constructive) complete separable metric spaces. This generality is well supported by standard results in measure theoretical probability, and is sufficient for constructivizing a large part of current probability theory.

84 4.1. Continuous spaces, noncomputable measures

The appendix recalls some of the needed topology, measure theory, constructive analysis and constructive measure theory. We also make use of the notation introduced there.

4.1.2 Uniform tests We first define tests of randomness with respect to an arbitrary measure. Recall the definition of lower semicomputable real functions on a computable metric space X. Definition 4.1.1 Let us be given a computable complete metric space X = -,3,,U . For an arbitrary measure ` M X , a `-test of randomness is a ( ) ∈ ( ) `-lower semicomputable function 5 : - R with the property ` 5 1. We → + ≤ call an element F random with respect to ` if 5 F < for all `-tests 5 . But even ( ) ∞ among random elements, the size of the tests quantifies (non-)randomness. A uniform test of randomness is a lower semicomputable function 5 : X × M X R , written as F,` 5` F such that 5` is a `-test for each ( ) → + ( ) 7→ ( ) (·) `. y The condition ` 5 1 guarantees that the probability of those outcomes ≤ whose randomness is ; is at most 1 ;. The definition of tests is in the spirit ≥ / of Martin-Löf’s tests. The important difference is in the semicomputability condition: instead of restricting the measure ` to be computable, we require the test to be lower semicomputable also in its argument `. The following result implies that every `-test can be extended to a uniform test.

Theorem 4.1.1 Let q4, 4 = 1, 2,... be an enumeration of all lower semicomputable functions X Y M X R , where Y is also a computable metric space, × × ( ) → + and A : Y M X R a lower semicomputable function. There is a recursive × ( ) → function 4 4 with the property that 7→ ′ a) For each 4, the function q F,G,` is everywhere deﬁned with 4′ ( ) `Fq F,G,` A G,` . 4′ ( ) ≤ ( ) b) For each 4, G,`, if `F q F, G` < A G,` then q ,G,` = q ,G,` . 4 ( ) ( ) 4′ (· ) 4 (· ) This theorem generalizes a theorem of Hoyrup and Rojas (in allowing a lower semicomputable upper bound A G,` ). ( ) Proof. By Proposition B.1.38, we can represent q F,G,` as a supremum q = 4 ( ) 4 sup ℎ where ℎ F,G,` is a computable function of 4, 7 monotonically in- 7 4,7 4,7 ( ) creasing in 7. Similarly, we can represent A G,` as a supremum sup A G,` ( ) 8 8 ( )

85 4. Generalizations where A G,` is a computable function monotonically increasing in 8. The inte- 8 ( ) gral `F ℎ F,G,` is computable as a function of G,` , in particular it is upper 4,7 ( ) ( ) semicomputable. Define ℎ F,G,` as ℎ F,G,` for all 8, G,` with `F ℎ F,G,` < 4,7,8′ ( ) 4,7 ( ) 4,7 ( ) A G,` , and 0 otherwise. Since A G,` is computable this definition makes 8 ( ) 8( ) the function ℎ F, G,` lower semicomputable. The function ℎ F,G,` = 4,7,8′ ( ) 4,7′′ ( ) sup ℎ F,G,` is then also lower semicomputable, with ℎ F,G,` 8 4,7,8′ ( ) 4,7′′ ( ) ≤ ℎ F,G,` , and `F ℎ F,G,` A G,` . Also, ℎ F,G,` is monotonically 4,7 ( ) 4,7′ ( ) ≤ ( ) 4,7′′ ( ) increasing in 7. The function q F,G,` = sup ℎ F,G,` is then also lower 4′( ) 7 4,7′′ ( ) semicomputable, and by Fubini’s theorem we have `F q F, G,` A G,` . 4′( ) ≤ ( ) Define q F,G,` = q F, G,` . Consider any 4, G,` such that 4′ ( ) 4′( ) `Fq F, G,` < A G,` holds. Then for every 7 there is a 8 with `F ℎ F,G,` < 4 ( ) ( ) 4,7 ( ) A G,` , and hence ℎ F,G,` = ℎ F,G,` . It follows that ℎ F,G,` = 8 ( ) 4,7,8′ ( ) 4,7 ( ) 4,7′′ ( ) ℎ4,7 F,G,` for all 7 and hence q4′ F,G,` = q4 F, G,` . ( ) ( ) ( )

Corollary 4.1.2 (Uniform extension) There is an operation F,` F,` 4 ( ) 7→ 4′ ( ) with the property that F,` is a uniform test and if 2 ,` is a `-test then 4′ ( ) 4(· ) F,` = F,` . 4′ ( ) 4 ( ) Proof. In Theorem 4.1.1 set q F,G,` = 1 F,` with A G,` = 1. 4 ( ) 2 4 ( ) ( ) Corollary 4.1.3 (Universal generalized test) Let A : Y M X R a lower × ( ) → + semicomputable function. Let be the set of lower semicomputable functions q F,G,` ( ) ≥ 0 with `Fq F, G,` A G,` . There is a function k that is optimal in the ( ) ≤ ( ) ∈ sense that for all q there is a constant 2 with q 22 k. ∈ q ≤ q Proof. Apply the operation 4 4 of theorem 4.1.1 to the sequence q F, G,` 7→ ′ 4 ( ) (4 = 1, 2,...) of all lower semicomputable functions of F, G,`. The elements of the sequence q4′ F,G,` , 4 = 1, 2,... are in and and the sequence 2q4′ F, G,` , ( ) ( )4 4 = 1, 2,... contains all elements of . Hence the function k F, G,` = ∞ 2 q F,G,` ( ) 4=1 − 4′( ) is in and has the optimality property. Í Deﬁnition 4.1.4 A uniform test C is called universal if for every other test B there is a constant 2 > 0 such that for all F,` we have B F 2C F . y B ` ( ) ≤ ` ( ) Theorem 4.1.2 (Universal test,[26]) There is a universal uniform test. Proof. This is a special case of Corollary 4.1.3 with A G,` = 1. ( )

86 4.1. Continuous spaces, noncomputable measures

Definition 4.1.5 Let us fix a universal uniform test, called t F . An element ` ( ) F - is called random with respect to measure ` M X if t F < . ∈ ∈ ( ) ` ( ) ∞ The deficiency of randomness is defined as d F = log t F . y ` ( ) ` ( ) If the space is discrete then typically all elements are random with respect to `, but they will still be distinguished according to their different values of d F . ` ( ) 4.1.3 Sequences Let our computable metric space X = -,3,,U be the Cantor space of Exam- ( ) ple B.1.35.2: the set of sequences over a (finite or countable) alphabet ΣN. We may want to measure the non-randomness of finite sequences, viewing them as initial segments of infinite sequences. Take the universal test t F . For this, it ` ( ) is helpful to apply the representation of Proposition B.1.23, taking into account that adding the extra parameter ` does not change the validity of the theorem:

Proposition 4.1.6 There is a function 6 : M X Σ∗ R with t` b = ( ) × → + ( ) sup 6 b < , and with the following properties: < ` ( ≤ ) a) 6 is lower semicomputable. b) { | implies 6 { 6 | . ⊑ ` ( ) ≤ ` ( ) c) For all integer < 0 we have | Σ< ` | 6` | 1. ≥ ∈ ( ) ( ) ≤ The properties of the function 6 | clearly imply that sup 6 b < is a Í ` ( ) < ` ( ≤ ) uniform test. The existence of a universal function among the functions 6 can be proved by the usual methods: Proposition 4.1.7 Among the functions 6 | satisfying the properties listed in ` ( ) Proposition 4.1.6, there is one that dominates to within a multiplicative constant. These facts motivate the following definition. Definition 4.1.8 (Extended test) Over the Cantor space, we extend the definition of a universal test t F to finite sequences as follows. We fix a function ` ( ) t | with | Σ whose existence is assured by Proposition 4.1.7. For infinite ` ( ) ∈ ∗ sequences b we define t b = sup t b < . The test with values defined also ` ( ) < ` ( ≤ ) on finite sequences will be called an extended test. y We could try to define extended tests also over arbitrary constructive metric spaces, extending them to the canonical balls, with the monotonicity property

87 4. Generalizations that t { t | if ball | is manifestly included in ball {. But there is nothing ` ( ) ≤ ` ( ) simple and obvious corresponding to the integral requirement (c). Over the space ΣN for a ﬁnite alphabet Σ, an extended test could also be extracted directly from a test, using the following formula (as observed by Vyugin and Shen). Deﬁnition 4.1.9

C H = inf C l : l is an inﬁnite extension of H . ( ) { ( ) } y

This function is lower semicomputable, by Proposition B.1.31.

4.1.4 Conservation of randomness For 7 = 1, 0, let X = - ,3 , ,U be computable metric spaces, and let 7 ( 7 7 7 7) M = M X , f , a be the effective topological space of probability measures 7 ( ( 7) 7 7) over X7. Let Λ be a computable probability kernel from X1 to X0 as defined in Subsection B.2.3. In the following theorem, the same notation d F will refer ` ( ) to the deficiency of randomness with respect to two different spaces, X1 and X0, but this should not cause confusion. Let us first spell out the conservation theorem before interpreting it.

Theorem 4.1.3 For a computable probability kernel Λ from X1 to X0, we have

G _ tΛ G <∗ t F . (4.1.1) F ∗` ( ) ` ( ) Proof. Let t F be the universal test over X . The left-hand side of (4.1.1) can a ( ) 0 be written as = Λ Λ C` t ∗` .

According to (A.2.4), we have `C = Λ ` tΛ which is 1 since t is a test. If ` ( ∗ ) ∗` ≤ we show that `,F C F is lower semicomputable then the universality of ( ) 7→ ` ( ) t` will imply that C` <∗ t`. According to Proposition B.1.38, as a lower semicomputable function, t G a ( ) can be written as sup 6 a, G , where 6 a, G is a computable sequence of < < ( ) ( < ( )) computable functions. We pointed out in Subsection B.2.3 that the function ` Λ ` is computable. Therefore the function <,`,F 6 Λ `, 5 F 7→ ∗ ( ) 7→ < ( ∗ ( )) is also a computable. So, C F is the supremum of a computable sequence of ` ( ) computable functions and as such, lower semicomputable.

88 4.1. Continuous spaces, noncomputable measures

It is easier to interpret the theorem ﬁrst in the special case when Λ = Λℎ for a computable function ℎ : - - , as in Example B.2.12. Then the theorem 1 → 0 simpliﬁes to the following.

Corollary 4.1.10 For a computable function ℎ : - - , we have d ℎ F <+ 1 → 0 ℎ∗` ( ( )) d F . ` ( ) Informally, this says that if F is random with respect to ` in X then ℎ F is 1 ( ) essentially at least as random with respect to the output distribution ℎ∗` in X0. Decrease in randomness can only be caused by complexity in the deﬁnition of the function ℎ. Let us specialize the theorem even more: Corollary 4.1.11 For a probability distribution ` over the space - . let ` be × - its marginal on the space -. Then we have

d F <+ d F,G . `- ( ) ` (( )) This says, informally, that if a pair is random then each of its elements is random (with respect to the corresponding marginal distribution). In the general case of the theorem, concerning random transitions, we cannot bound the randomness of each outcome uniformly. The theorem asserts that the average nonrandomness, as measured by the universal test with respect to the G dΛ ` G + output distribution, does not increase. In logarithmic notation: _F 2 ∗ ( ) < dΛ G d F , or equivalently, 2 ∗` _ 3G <+ d F . ` ( ) ( ) F ( ) ` ( ) Λ Corollary 4.1.12 Let ∫be a computable probability kernel from X1 to X0. There is a constant 2 such that for every F X1, and integer ; > 0 we have ∈ ; _ G : dΛ G > d F ; 2 2− . F { ∗` ( ) ` ( )+ + } ≤ Thus, in a computable random transition, the probability of an increase of ; randomness deﬁciency by ; units (plus a constant 2) is less than 2− . The constant 2 comes from the description complexity of the transition Λ. A randomness conservation result related to Corollary 4.1.10 was proved in [25]. There, the measure over the space X0 is not the output measure of the transformation, but is assumed to obey certain inequalities related to the transformation.

89 4. Generalizations

4.2 Test for a class of measures

4.2.1 From a uniform test A Bernoulli measure is what we get by tossing a (possibly biased) coin repeatedly. Definition 4.2.1 Let - = BN be the set of infinite binary sequences, with the usual sequence topology. Let > be the measure on - that corresponds to tossing a coin independently with probability > of success: it is called the Bernoulli measure with parameter >. Let B denote the set of all Bernoulli measures. y Given a sequence F - we may ask the question whether F is random with ∈ respect to at least some Bernoulli measure. (It can clearly not be random with respect to two different Bernoulli measures since if F is random with respect to > then its relative frequencies converge to >.) This idea suggests two possible definitions for a test of the property of “Bernoulliness”:

1. We could define tB F = inf` B F . ( ) ∈ ( ) 2. We could define the notion of a Bernoulli test as a lower semicomputable function 5 F with the property 5 1 for all >. ( ) > ≤ We will see that in case of this class of measures the two definitions lead to essentially the same test. Let us first extend the definition to more general sets of measures, still having a convenient property. Definition 4.2.2 (Class tests) Consider a class C of measures that is effectively compact in the sense of Definition B.1.27 or (equivalently for metric spaces) in the sense of Theorem B.1.1. A lower semicomputable function 5 F is called a ( ) C-test if for all ` C we have ` 5 1. It is a universal C-test if it dominates all ∈ ≤ other C-tests to within a multiplicative constant. y Example 4.2.3 It is easy toshow that the class B is effectively compact. One way is to appeal to the general theorem in Proposition B.1.32 saying that applying a computable function to an effectively compact set (in this case the interval 0, 1 ), the image is also an effectively compact set. y [ ] For the case of infinite sequences, we can also define extended tests. Definition 4.2.4 (Extended class test) Let our space X be the Cantor space of infinite sequences ΣN. Consider a class C of measures that is effectively compact in the sense of Definition B.1.27 or (equivalently for metric spaces) in the sense of Theorem B.1.1. A lower semicomputable function 5 : Σ∗ R is called an → + extended C-test if it is monotonic with respect to the prefix relation and for all

90 4.2. Test for a class of measures

` C and integer < 0 we have ∈ ≥ ` F 5 F 1. ( ) ( ) ≤ F Σ< Õ∈ It is universal if it dominates all other extended C-tests to within a multiplicative constant. y The following observation is immediate. Proposition 4.2.5 A function 5 : ΣN R is a class test if and only if it can be → + represented as lim 6 b < where 6 F is an extended class test. < ( ≤ ) ( ) The following theorem deﬁnes a universal C-test.

Theorem 4.2.1 Let t` F be a universal uniform test. Then C F = inf` C t` F ( ) ( ) ∈ ( ) defines a universal C-test. Proof. Let us show first that C F is a C-test. It is lower semicomputable accord- ( ) ing to Proposition B.1.31. Also, for each ` we have `C `t 1, showing that ≤ ` ≤ C F is a C-test. ( ) Let us now show that it is universal. Let 5 F be an arbitrary C-test. By ( ) Corollary 4.1.2 there is a uniform test 6 F such that for all ` C we have ` ( ) ∈ 6 F = 5 F 2. It follows from the universality of the uniform test t F that ` ( ) ( )/ ` ( ) 5 F <∗ 6` F <∗ t` F for all ` C. But then 5 F <∗ inf` C t` F = C F . ( ) ( ) ( ) ∈ ( ) ∈ ( ) ( ) For the case of sequences, the same statement can be made for extended tests. (This is not completely automatic since a test is obtained from an extended test via a supremum, on the other hand a class test is obtained, according the theorem above, via an infimum.)

4.2.2 Typicality and class tests The set of Bernoulli measures has an important property shared by many classes considered in practice: namely that random sequences determine the measure to which it belongs. Deﬁnition 4.2.6 Consider a class C of measures over a computable metric space X = -,3,,U . We will say that a lower semicomputable function ( ) A : - C R × → + is a separating test for C if • A is a test for each ` C. ` (·) ∈ 91 4. Generalizations

• If ` ≠ a then A F A F = for all F -. ` ( )∨ a ( ) ∞ ∈ Given a separating test A F we call an element F typical for ` C if A F < ` ( ) ∈ ` ( ) . y ∞ A typical element determines uniquely the measure ` for which it is typical. Note that if a separating tests exists for a class then any two diﬀerent measures `1,`2 in the class are orthogonal to each other, that is there are disjoint measureable sets , with ` = 1. Indeed, let = F : A F < . 1 2 8( 8) 8 { ` 8 ( ) ∞} Let us show a nontrivial example: the class of B of Bernoulli measures. Recall that by Chebyshev’s inequality we have

< 1 2 1 2 2 F B : F 7 <> _< / > 1 > / _− . > ({ ∈ | ( )− | ≥ ( ( − )) }) ≤ Õ7 Since > 1 > 1 4, this implies ( − ) ≤ / < 1 2 2 F B : F 7 <>> _< / 2 < _− . >({ ∈ | ( )− / }) Õ7 Setting _ = <0.1 and ignoring the factor 1 2 gives / < 0.6 0.2 F B : F 7 <> > < < <− . >({ ∈ | ( )− | }) Õ7 Setting < = 29:

29 9 0.69 0.29 F B : F 7 2 > > 2 < 2− . (4.2.1) >({ ∈ | ( )− | }) Õ7 Now, for the example. Example 4.2.7 For a sequence b in BN, and for > 0, 1 let ∈ [ ] 29 = = 9 0.69 6> F 6> F sup 9 : b 7 2 > > 2 . ( ) ( ) { | = ( )− | } Õ7 1 Then we have

b 0.29 6 b 9 2− = 2 < > > ( ) ≤ · ∞ Õ9 for some constant 2, so A> b = 6> F 2 is a test for each >. The property A> b < 9 ( ) ( )/ ( ) implies that 2 9 2 b 7 converges to >. For a given b this is impossible for ∞ − 7=1 ( ) both > and ? for > ≠ ?, hence A> b A? b = . y Í ( )∨ ( ) ∞ 92 4.2. Test for a class of measures

The following structure theorem gives considerable insight. Theorem 4.2.2 Let C be an eﬀectively compact class of measures, let t F be the ` ( ) universal uniform test and let tC F be a universal class test for C. Assume that a ( ) separating test A F exists for C. Then we have the representation ` ( )

t F =∗ tC F A F ` ( ) ( )∨ ` ( ) for all ` C, F -. ∈ ∈ Proof. First, we have tC F A F <∗ t F . Indeed as we know from the Uni- ( )∨ ` ( ) ` ( ) form Extension Corollary 4.1.2, we can extend A F 2 to a uniform test, hence ` ( )/ A F <∗ t F . Also by deﬁnition tC F t F . ` ( ) ` ( ) ( ) ≤ ` ( ) On the other hand, let us show tC F A F t F . Suppose ﬁrst that F ( )∨ ` ( ) ≥ ` ( ) is not random with respect to any a C: then tC F = . Suppose now that F ∈ ( ) ∞ is random with respect to some a C, a ≠ `. Then A F = . Finally, suppose ∈ ` ( ) ∞ t` F < . Then ta F = for all a C, a ≠ `, hence tC F = infa C ta F = ( ) ∞ ( ) ∞ ∈ ( ) ∈ ( ) t F , so the inequality holds again. ` ( ) The above theorem separates the randomness test into two parts. One part tests randomness with respect to the class C, the other one tests typicality with respect to the measure `. In the Bernoulli example,

• Part tB b checks “Bernoulliness”, that is independence. It encompasses all ( ) the irregularity criteria. • Part A b checks (crudely) for the law of large numbers: whether relative >( ) frequency converges (fast) to >. If the independence of the sequence is taken for granted, we may assume that the class test is satisﬁed. What remains is typicality testing, which is similar to ordinary statistical parameter testing. Remarks 4.2.8 1. Separation is the only requirement of the test A F , otherwise, for example ` ( ) in the Bernoulli test case, no matter how crude the convergence criterion expressed by A F , the maximum tC F A F is always (essentially) the ` ( ) ( ) ∨ ` ( ) same universal test. 2. Though the convergence criterion can be crude, but one still seems to need some kind of constructive convergence of the relative frequencies if the separation test is to be deﬁned in terms of relative frequency convergence. y

93 4. Generalizations

Example 4.2.9 For Y > 0 let % Y be the Markov chain - ,- ,... with set of ( ) 1 2 states 0, 1 , with transition probabilities ) 0, 1 = ) 1, 0 = Y and ) 0, 0 = { } ( ) ( ) ( ) ) 1, 1 = 1 Y, and with % - = 0 = % - = 1 = 1 2. Let C X be the class ( ) − [ 1 ] [ 1 ] / ( ) of all % Y with Y X. For each X > 0, this is an eﬀectively compact class, and a ( ) ≥ separating test is easy to construct since an eﬀective law of large numbers holds for these Markov chains. We can generalize to the set of ;-state stationary Markov chains whose eigenvalue gap is Y. y ≥ This example is in contrast to V’yugin’s example [56] showing that, in the nonergodic case, in general no recursive speed of convergence can be guaranteed in the Ergodic Theorem (which is the appropriate generalization of the law of large numbers)1. We can show that if a computable Markov chain is ergodic then its law of large numbers does have a constructive speed of convergence. Hopefully, this observation can be extended to some interesting compact classes of ergodic pro- cesses.

4.2.3 Martin-Löf’s approach Martin-Löf also gave a deﬁnition of Bernoulli tests in [38]. For its deﬁnition let us introduce the following notation. Notation 4.2.10 The set of sequences with a given frequency of 1’s will be denoted as follows:

B <, 9 = F B< : F 7 = 9 . ( ) { ∈ ( ) } Õ7 y

Martin-Löf’s definition translates to the integral constraint version as follows: Definition 4.2.11 Let - = BN be the set of infinite binary sequences with the usual metrics. A combinatorial Bernoulli test is a function 5 : B∗ R with the → + following constraints: a) It is lower semicomputable. b) It is monotonic with respect to the prefix relation.

1This paper of Vyugin also shows that though there is no recursive speed of convergence, a certain constructive proof of the pointwise ergodic theorem still gives rise to a test of randomness.

94 4.2. Test for a class of measures

c) For all 0 9 < we have ≤ ≤ < 5 F . (4.2.2) ( ) ≤ 9 F B <,9 ∈Õ( ) y

The following observation is useful. Proposition 4.2.12 If a combinatorial Bernoulli test 5 F is given on strings F of ( ) length less than <, then extending it to longer strings using monotonicity we get a function that is still a combinatorial Bernoulli test.

Proof. It is suﬃcient to check the relation (4.2.2). We have (even for 9 = 0, when B < 1, 9 1 = ): ( − − ) ∅ 5 F 5 G 5 G ( ) ≤ ( )+ ( ) F B <,9 G B < 1,9 1 G B < 1,9 ∈Õ( ) ∈ (Õ− − ) ∈ Õ( − ) < 1 < 1 < − − = . ≤ 9 1 + 9 9 −

The following can be shown using standard methods: Proposition 4.2.13 (Universal combinatorial Bernoulli test) There is a universal combinatorial Bernoulli test 5 F , that is a combinatorial Bernoulli test with the ( ) property that for every combinatorial Bernoulli test ℎ F there is a constant 2 > 0 ( ) ℎ such that for all F we have ℎ F 2 6 F . ( ) ≤ ℎ ( ) Definition 4.2.14 Let us fix a universal combinatorial Bernoulli test 1 F and ( ) extend it to infinite sequences b by

< 1 b = sup 1 b≤ . ( ) < ( )

N Let tB b be a universal class test for Bernoulli measures, for b B . y ( ) ∈ Let us show that the general class test for Bernoulli measures and Martin- Löf’s Bernoulli test yield the same random sequences.

Theorem 4.2.3 With the above deﬁnitions, we have 1 b =∗ tB b . In words: a ( ) ( ) sequence is nonrandom with respect to all Bernoulli measures if and only if it is rejected by a universal combinatorial Bernoulli test.

95 4. Generalizations

Proof. We ﬁrst show 1 b tB b . Moreover, we show that 1 F is an ex- ( ) ≤ ( ) ( ) tended class test for the class of Bernoulli measures. We only need to check the sum condition, namely that for all 0 > 1, and all < > 0 the inequality ≤ ≤ F B< > F 1 F 1 holds. Indeed, we have ∈ ( ) ( ) ≤ Í < 9 < 9 F 5 F = > 1 > − 5 F >( ) ( ) ( − ) ( ) F B< 9=0 F B <,9 Õ∈ Õ ∈Õ( ) < 9 < 9 < > 1 > − = 1. ≤ = ( − ) 9 Õ9 0 On the other hand, let 5 F = tB F , F B be the extended test for tB b . For ( ) ( ) ∈ ∗ ( ) all integers # > 0 let < = # 2 . Then as # runs through the integers, < also ⌊ / ⌋ runs through all integers. For F B# let F = 5 F < . Since 5 F is lower p ∈ ( ) ( ≤ ) ( ) semicomputable and monotonic with respect to the preﬁx relation, this is also true of F . ( ) We need to estimate F B #, F . For this, note that for G B <, 9 we ∈ ( ) ( ) ∈ ( ) have Í # < F B #, : G F = − . (4.2.3) |{ ∈ ( ) ⊑ }| 9 − Now for 0 # we have ≤ ≤ F = 5 G F B #, : G F ( ) ( )|{ ∈ ( ) ⊑ }| F B #, G B< ∈Õ( ) Õ∈ < (4.2.4) # < = − 5 G . 9 ( ) 9=0 G B <,9 Õ − ∈Õ( ) # < # = = = Let us estimate −9 . If 9 0 then this is 1. If 9 < then it is < 1 − = # ···(#−<+1) . Otherwise, using > #: ···( − + ) / # < −9 # < # < 1 # < 9 1 9 ! − = ( − )( − − )···( − − ( − )+ )/( − ) # # # 1 # 1 ! ( − )···( − + )/ 9 1 # # < 9 1 = ···( − + ) · ( − )···( − − ( − )+ ) . # # < 1 9 < 9 ···( − + ) < # − 9 < 9 # ( − ) = > 1 > − . (4.2.5) ≤ # < < ( − ) # < ( − ) −

96 4.3. Neutral measure

Thus in all cases the estimate < # < # 9 < 9 # − > 1 > − < ≤ ( − ) # < − − holds. We have < # < < # = 1 4 2 # < 42, # < + # < ≤ ( − ) ≤ − − since we assumed 2<2 #. Substituting into (4.2.4) gives ≤ < 2 9 < 9 2 F 4 > 1 > − 5 G 4 , ( ) ≤ ( − ) ( ) ≤ F B #, 9=0 G B <,9 ∈Õ( ) Õ ∈Õ( ) since 5 F is an extended class test for Bernoulli measures. It follows that 2 ( ∗) ∗ < 4− F < 1 F , hence also F < 1 F . But we have tC b = sup< 5 b≤ = ( ) < ( ) ( ) ( ) ( ) ( ) sup b , hence tC b <∗ 1 b . < ( ≤ ) ( ) ( ) 4.3 Neutral measure

Let t F be our universal uniform randomness test. We call a measure " neutral ` ( ) if t F 1 for all F. If " is neutral then no experimental outcome F could " ( ) ≤ refute the theory (hypothesis, model) that " is the underlying measure to our experiments. It can be used as “apriori probability”, in a Bayesian approach to statistics. Levin’s theorem says the following: Theorem 4.3.1 If the space X is compact then there is a neutral measure over X. The proof relies on a nontrivial combinatorial fact, Sperner’s Lemma, which also underlies the proof of the Brouwer ﬁxpoint theorem. Here is a version of Sperner’s Lemma, spelled out in continuous form: < Proposition 4.3.1 Let >1,...,>9 be points of some ﬁnite-dimensional space R . Suppose that there are closed sets 1,...,9 with the property that for every subset 1 7 < < 7 9 of the indices, the simplex ( > ,...,> spanned by ≤ 1 · · · 8 ≤ ( 71 7 8 ) > ,...,> is covered by the union . Then the intersection of 71 7 8 71 ∪···∪ 7 8 7 7 all these sets is not empty. Ñ The following lemma will also be needed. Lemma 4.3.2 For every closed set X and measure `, if ` = 1 then there ⊂ ( ) is a point F with t F 1. ∈ ` ( ) ≤ 97 4. Generalizations

Proof. This follows easily from `B = `F1 F B F 1. ` ( ) ` ( ) ≤ Proof of Theorem 4.3.1. For every point F X, let be the set of measures for ∈ F which t F 1. If we show that for every finite set of points F ,...,F , we ` ( ) ≤ 1 9 have ≠ , (4.3.1) F1 ∩···∩ F9 ∅ then we will be done. Indeed, according to Proposition A.2.50, the compactness of X implies the compactness of the space M X of measures. Therefore if every ( ) finite subset of the family : F X of closed sets has a nonempty inter- { F ∈ } section, then the whole family has a nonempty intersection: this intersection consists of the neutral measures. To show (4.3.1), let ( F ,...,F be the set of probability measures con- ( 1 9) centrated on F1,...,F9. Lemma 4.3.2 implies that each such measure belongs to one of the sets . Hence ( F ,...,F , and the same F7 ( 1 9) ⊂ F1 ∪···∪ F9 holds for every subset of the indices 1, . . . , 9 . Sperner’s Lemma 4.3.1 implies { } ≠ . F1 ∩···∩ F9 ∅ When the space is not compact, there are generally no neutral probability measures, as shown by the following example. Proposition 4.3.3 Over the discrete space X = N of natural numbers, there is no neutral measure. Proof. It is sufficient to construct a randomness test B F with the property that ` ( ) for every measure `, we have sup B F = . Let F ` ( ) ∞ 9 B` F = sup 9 N : ` G > 1 2− . (4.3.2) ( ) { ∈ G0 B F 9 9>0 Õ Õ `Õ( ) ≥ Õ

Using a similar construction over the space NN of infinite sequences of natural numbers, we could show that for every measure ` there is a sequence F with t F = . ` ( ) ∞ Proposition 4.3.3 is a little misleading, since N can be compactified into N = N (as in Part 1 of Example A.1.22). Theorem 4.3.1 implies that there is a ∪{∞} 98 4.3. Neutral measure neutral probability measure " over the compactified space N. Its restriction to

N is, of course, not a probability measure, since it satisﬁes only F< " F 1. ∞ ( ) ≤ We called these functions semimeasures. Í Remark 4.3.4 1. It is easy to see that Theorem 4.6.1 characterizing randomness in terms of complexity holds also for the space N. 2. The topological space of semimeasures over N is not compact, and there is no neutral one among them. Its topology is not the same as what we get when we restrict the topology of probability measures over N to N. The difference is that over N, for example the set of measures ` : ` N > 1 2 is { ( ) / } closed, since N (as the whole space) is a closed set. But over N, this set is not necessarily closed, since N is not a closed subset of N. y

Neutral measures are not too simple, even over N, as the following theorem shows. Theorem 4.3.2 There is no neutral measure over N that is upper semicomputable over N or lower semicomputable over N. Proof. Let us assume that a is a measure that is upper semicomputable over N. Then the set F,@ : F N,@ Q, a F < @ { ( ) ∈ ∈ ( ) } is recursively enumerable: let F7,@7 be a particular enumeration. For each <, ( < ) < let 7 < be the ﬁrst 7 with @7 < 2− , and let G< = F7 < . Then a G< < 2− , and at ( ) ( ) ( ) the same time G <+ < . As mentioned, in Remark 4.3.4, Theorem 4.6.1 ( <) ( ) characterizing randomness in terms of complexity holds also for the space N. Thus, d G =+ log a G G a >+ < < . a ( <) − ( <)− ( < | ) − ( ) Suppose now that a is lower semicomputable over N. The proof for this case is longer. We know that a is the monotonic limit of a recursive sequence 7 a F 7→ 7 ( ) of recursive semimeasures with rational values a F . For every 9 = 0,..., 2< 2, 7 ( ) − let

< < < +<,9 = ` M N : 9 2− < ` 0,..., 2 1 < 9 2 2− , { ∈ ( ) <· ({ < − }) ( + ) · } = <, 9 : 9 2− < a 0,..., 2 1 . { ( ) · ({ − })} The set is recursively enumerable. Let us deﬁne the functions 8 : N and → F : 0,..., 2< 1 as follows: 8 <, 9 is the smallest 7 with a 0,..., 2< →{ − } ( ) 7 ({ − 99 4. Generalizations

1 > 9 2 <, and }) · − < < 1 F<,9 = min G < 2 : a 8 <,9 G < 2− + . { ( ) ( ) } Let us deﬁne the function 5 F, <, 9 as follows. We set 5 F, <, 9 = 2< 2 if the ` ( ) ` ( ) − following conditions hold: a) ` + ; ∈ <,9 b) ` F < 2 < 2; ( ) − + c) <, 9 and F = F . ( ) ∈ <,9 Otherwise, 5 F, <, 9 = 0. Clearly, the function `, F, <, 9 5 F, <, 9 is ` ( ) ( ) 7→ ` ( ) lower semicomputable. Condition (b) implies

< 2 < 2 ` G 5` G, <, 9 ` F<,9 5` F<,9, <, 9 < 2− + 2 − = 1. (4.3.3) G ( ) ( ) ≤ ( ) ( ) · Õ Let us show that a + implies ∈ <,9 < 2 5 F , <, 9 = 2 − . (4.3.4) a ( <,9 ) Consider F = F<,9. Conditions (a) and (c) are satisfied by definition. Let us show that condition (b) is also satisfied. Let 8 = 8 <, 9 . By definition, we have ( ) a F < 2 < 1. Since by definition a + and a a + , we have 8 ( ) − + 8 ∈ <,9 8 ≤ ∈ <,9 < 1 < 1 < 1 < 2 a F a F 2− + < 2− + 2− + = 2− + . ( ) ≤ 8 ( )+ + Since all three conditions (a), (b) and (c) are satisfied, we have shown (4.3.4). Now we define 1 6` F = 5` F, <, 9 . ( ) < < 1 ( ) < 2 9 Õ≥ ( + ) Õ Let us prove that 6 F is a uniform test. It is lower semicomputable by defini- ` ( ) tion, so we only need to prove ` F 5 F 1. For this, let = 9 : ` F ( ) ` ( ) ≤ <,` { ∈ + . Clearly by definition, 2. We have, using this last fact and the test <,9 } | <,` | ≤ property (4.3.3): Í 1 1 ` F 6` F = ` F 5` F, <, 9 2 1. ( ) ( ) < < 1 ( ) ( ) ≤ < < 1 · ≤ F < 2 9 F < 2 Õ Õ≥ ( + ) Õ∈ <,` Õ Õ≥ ( + ) Thus, 6 F is a uniform test. If a + then we have ` ( ) ∈ <,9 < 2 1 2 − ta F<,9 >∗ 6a F<,9 5` F<,9, <, 9 . ( ) ( ) ≥ < < 1 ( ) ≥ < < 1 ( + ) ( + ) Hence a is not neutral.

100 4.4. Monotonicity, quasi-convexity/concavity

Remark 4.3.5 In [32] and [33], Levin imposed extra conditions on tests which allow to find a lower semicomputable neutral semimeasure. y The universal lower semicomputable semimeasure m F has a certain prop- ( ) erty similar to neutrality. According to Theorem 2.3.4 specialized to one-element sequences, for every computable measure ` we have d F =+ log ` F F ` ( ) − ( )− ( ) (where the constant in =+ depends on `). So, for computable measures, the expression d F = log ` F F (4.3.5) ` ( ) − ( )− ( ) can serve as a reasonable deficiency of randomness. (We will also use the test t = 2d.) If we substitute m for ` in d F , we get 0. This substitution is not ` ( ) justified, of course. The fact that m is not a probability measure can be helped, using compactification as above, and extending the notion of randomness tests. But the test d` can replace d` only for computable `, while m is not computable. Anyway, this is the sense in which all outcomes might be considered random with respect to m, and the heuristic sense in which m may be considered “neutral”.

4.4 Monotonicity, quasi-convexity/concavity

Some people find that `-tests as defined in Definition 4.1.1 are too general, in case ` is a non-computable measure. In particular, randomness with respect to computable measures has a certain—intuitively meaningful—monotonicity property: roughly, if a is greater than ` then if F is random with respect to `, it should also be random with respect to a. Proposition 4.4.1 For computable measures `, a we have for all rational 2 > 0:

9 2− ` a d F <+ d F 9 9 . (4.4.1) ≤ ⇒ a ( ) ` ( )+ + ( ) Here the constant in <+ depends on `, a, but not on 9.

Proof. We have 1 at 2 9`t , hence 2 9t is a `-test. Using the method of ≥ a ≥ − a − a Theorem 4.1.1 in ﬁnding universal tests, one can show that the sum

9 9 2− − ( )ta 9 9:2−Õ`ta<1 is a `-test, and hence <∗ t`. Therefore this is true of each member of the sum, which is just what the theorem claims.

101 4. Generalizations

There are other properties true for tests on computable measures that we may want to require for all measures. For the following properties, let us define quasi-convexity, which is a weakening of the notion of convexity. Definition 4.4.2 A function 5 : + R defined on a vector space + is called → quasi-convex if for every real number F the set { : 5 { F is convex. It is { ( ) ≤ } quasi-concave if 5 is quasi-convex. y − It is easy to see that quasi-convexity is equivalent to the inequality

5 _C 1 _ { 5 C 5 { ( + ( − ) ) ≤ ( )∨ ( ) for all C, { and 0 <_< 1, while quasi-concavity is equivalent to

5 _C 1 _ { 5 C 5 { ( + ( − ) ) ≥ ( )∧ ( ) The uniform test with respect to computable measures is approximately both quasi-convex and quasi-concave. Let a = _` 1 _ ` . 1 + ( − ) 2 Quasi-convexity means, roughly, that if F is random with respect to both `1 and `2 then it is also random with respect to a. This property strengthens monotonicity in the cases where it applies.

Proposition 4.4.3 Let `1,`2 be computable measures and 0 <_< 1 computable, with a = _` 1 _ ` . Then we have 1 + ( − ) 2 d F <+ d F d F _ . a ( ) `1 ( )∨ `2 ( )+ ( ) Proof. The relation 1 at = _` t 1 _ ` t implies 1 ` t for some ≥ a 1 a + ( − ) 2 a ≥ 7 a 7 1, 2 . Then d <+ d _ (since _ was used to deﬁne a and thus t ). ∈ { } a `7 + ( ) a Quasi-concavity means, roughly, that if F is non-random with respect to both `1 and `2 then it is also nonrandom with respect to a:

Proposition 4.4.4 Let `1,`2 be computable measures and 0 <_< 1 arbitrary (not even necessarily computable), with a = _` 1 _ ` . Then we have 1 + ( − ) 2 d >+ d d . a `1 ∧ `2 Proof. The function t t is lower semicomputable, and is a ` -test for each `1 ∧ `2 7 7. Therefore it is also a a-test, and as such is <∗ ta. Here, the constant in the <∗ depends only on (the programs for) `1,`2, and not on _. These properties do not survive for arbitrary measures and arbitrary constants.

102 4.5. Algorithmic entropy

Example 4.4.5 Let measure ` be uniform over the interval 0, 1 2 , let ` be 1 [ / ] 2 uniform over 1 2, 1 . For 0 F 1 let [ / ] ≤ ≤ a = 1 F ` F` . F ( − ) 1 + 2 Then a1 2 is uniform over 0, 1 . Let qF be the uniform distribution over F,F / [ ] [ + 1 2 , and k the uniform distribution over 0,F F 1 2, 1 . / ] F [ ] ∪ [ + / ] Let > < 1 2 be random with respect to the uniform distribution a1 2. / / 1. The relations 1 = a1 2 >− a>, da1 2 > < , da> > / ≤ / ( ) ∞ ( ) ∞ show that any statement analogous to the monotonicity property of Proposi- tion 4.4.1 fails when the measures involved are not required to be computable. 2. For rational @ with 0 <@<> the relations = = a : 1 > a@ >a1 @, da > , da@ > < , da1 @ > < ( − ) + − ( ) ∞ ( ) ∞ − ( ) ∞ provide a similar counterexample to Proposition 4.4.3. 3. The relations = = = a1 2 a> a1 > 2, da1 2 > < , da> > da1 > > / ( + − )/ / ( ) ∞ ( ) − ( ) ∞ provide a similar counterexample to Proposition 4.4.4. The following counterexample relies on a less subtle eﬀect: = = = a1 2 q> k> 2, da1 2 > < , dq> > dk> > , / ( + )/ / ( ) ∞ ( ) ( ) ∞ since as a boundary point of the support, > is computable from both q> and k> in a uniform way. For a complete proof, uniform tests must be provided for each of the cases: this is left as exercise for the reader. y The non-monotonicity example could be used to argue that the we allowed too many `-tests, that the test t F should not be allowed to depend on prop- ` ( ) erties of ` that exploit the computational properties of ` so much stronger than its quantitative properties.

4.5 Algorithmic entropy

Some properties of description complexity make it a good expression of the idea of individual information content.

103 4. Generalizations

4.5.1 Entropy The entropy of a discrete probability distribution ` is deﬁned as

H ` = ` F log ` F . ( ) − ( ) ( ) ÕF To generalize entropy to continuous distributions the relative entropy is defined as follows. Let `, a be two measures, where ` is taken (typically, but not always), to be a probability measure, and a another measure, that can also be a probability measure but is most frequently not. We define the relative entropy H ` as a ( ) follows. If ` is not absolutely continuous with respect to a then H ` = . a ( ) −∞ Otherwise, writing 3` ` 3F = ( ) =: 5 F 3a a 3F ( ) ( ) for the (Radon-Nikodym) derivative (density) of ` with respect to a, we define

3` F ` 3F F Ha ` = log 3` = ` log ( ) = a 5 F log 5 F . ( ) − 3a − a 3F − ( ) ( ) ¹ ( ) Thus, H ` = H ` is a special case. ( ) # ( ) Example 4.5.1 Let 5 F be a probability density function for the distribution ` ( ) over the real line, and let _ be the Lebesgue measure there. Then

H ` = 5 F log 5 F 3F. _ ( ) − ( ) ( ) ¹ y In information theory and statistics, when both ` and a are probability measures, then H ` is also denoted ` a , and called (after Kullback) the − a ( ) ( k ) information divergence of the two measures. It is frequently used in the role of a distance between ` and a. It is not symmetric, but can be shown to obey the triangle inequality, and to be nonnegative. Let us prove the latter property: in our terms, it says that relative entropy is nonpositive when both ` and a are probability measures. Proposition 4.5.2 Over a space -, we have

` - Ha ` ` - log ( ) . (4.5.1) ( )≤− ( ) a - ( ) In particular, if ` - a - then H ` 0. ( ) ≥ ( ) a ( ) ≤ 104 4.5. Algorithmic entropy

Proof. The inequality 0 ln 0 0 ln 1 1 0 expresses the concavity of the − ≤ − + − logarithm function. Substituting 0 = 5 F and 1 = ` - a - and integrating ( ) ( )/ ( ) by a: ` - ` - ln 2 H ` = aF 5 F ln 5 F ` - ln ( ) ( ) a - ` - ( ) a ( ) − ( ) ( )≤− ( ) a - + a - ( )− ( ) ( ) ( ) ` - = ` - ln ( ) , − ( ) a - ( ) giving (4.5.1).

4.5.2 Algorithmic entropy Le us recall some facts on description complexity. Let us fix some (finite or infinite) alphabet Σ and consider the discrete space Σ∗. The universal lower semicomputable semimeasure m F over Σ was defined ( ) ∗ in Definition 1.6.9. It is possible to turn m F into a measure, by compactifying ( ) the discrete space Σ∗ into Σ = Σ∗ ∗ ∪ {∞} (as in part 1 of Example A.1.22; this process makes sense also for a constructive discrete space), and setting m = 1 F Σ m F . The extended measure m (∞) − ∈ ∗ ( ) is not quite lower semicomputable since the number ` Σ 0 is not necessarily ( ∗\{ }) lower semicomputable. Í Σ Remark 4.5.3 A measure ` is computable over ∗ if and only if the function F ` F is computable for F Σ . This property does not imply that the 7→ ( ) ∈ ∗ number 1 ` = ` Σ∗ = ` F − (∞) ( ) ( ) F Σ Õ∈ ∗ is computable. y Let us allow, for a moment, measures ` that are not probability measures: they may not even be finite. Metric and computability can be extended to this case, the universal test t F can also be generalized. The Coding Theorem 1.6.5 ` ( ) and other considerations suggest the introduction of the following notation, for an arbitrary measure `: Definition 4.5.4 We define the algorithmic entropy of a point F with respect to measure ` as F = d F = log t F . (4.5.2) ` ( ) − ` ( ) − ` ( ) y

105 4. Generalizations

Then, with # deﬁned as the counting measure over the discrete set Σ∗ (that is, # ( = ( ), we have ( ) | | F =+ F . ( ) # ( ) This allows viewing F as a generalization of description complexity. ` ( ) The following theorem generalizes an earlier known theorem stating that over a discrete space, for a computable measure, entropy is within an additive F constant the same as “average complexity”: H ` =+ ` F . ( ) ( ) Theorem 4.5.1 Let ` be a probability measure. Then we have

H ` `F F ` . (4.5.3) a ( ) ≤ a ( | ) If - is a discrete space then the following estimate also holds:

F H ` >+ ` F ` . (4.5.4) a ( ) a ( | ) Proof. Let X be the measure with density ta F ` with respect to a: ta F ` = X 3F ( | ) ( | ) a(3F) . Then X - 1. It is easy to see from the maximality property of ta F ` ( ) ( ) ≤ ( | ) a 3F = that ta F ` > 0, therefore according to Proposition A.2.23, we have X(3F) ( 1| ) ( ) X 3F − a(3F) . Using Proposition A.2.23 and 4.5.2: ( ) F ` 3F Ha ` = ` log ( ) , ( ) − a 3F ( ) X 3F a 3F `F F ` = `F log ( ) = `F log ( ) , − a ( | ) a 3F − X 3F ( ) ( ) F F ` 3F ` - Ha ` ` a F ` = ` log ( ) ` - log ( ) 0. ( )− ( | ) − X 3F ≤− ( ) X - ≤ ( ) ( ) This proves (4.5.3). ` 3F = ` F Over a discrete space -, the function F,`,a a (3F) a (F) is computable, ( ) 7→ ( ) `( )3F ∗ therefore by the maximality property of a F ` we have a (3F) < ta F ` , ` 3F ( | ) ( ) ( | ) H = F + F hence a ` ` log a (3F) > ` a F ` . ( ) − ( ) ( | ) 4.5.3 Addition theorem The Addition Theorem (3.1.5) can be generalized to the algorithmic entropy F introduced in (4.5.2) (a somewhat similar generalization appeared in [54]). ` ( ) The generalization, deﬁning `,a = ` a, is × F,G =+ F a G F, F a , ` . (4.5.5) `,a ( ) ` ( | )+ a ( | ` ( | ) ) 106 4.5. Algorithmic entropy

Before proving the general addition theorem, we establish a few useful facts. Proposition 4.5.5 We have

G F,G F a <+ log a 2− `,a ( ) . ` ( | ) − Proof. The function 5 F,`,a that is the right-hand side, is upper semicom- ( ) putable by deﬁnition, and obeys `F2 5 F,`,a 1. Therefore the inequality fol- − ( ) ≤ lows from the minimum property of F . ` ( ) Let H N, then the inequality ∈ F <+ H F H (4.5.6) ` ( ) ( )+ ` ( | ) will be a simple consequence of the general addition theorem. The following lemma, needed in the proof of the theorem, generalizes this inequality somewhat: Lemma 4.5.6 For a computable function G, H 5 G, H over N, we have ( ) 7→ ( ) F G <+ H F 5 G, H . ` ( | ) ( )+ ` ( | ( )) Proof. The function

` F 5 G,H H F,G,` 6` F,G = 2− ( | ( ))− ( ) ( ) 7→ ( ) H Õ

F H ∗ is lower semicomputable, and ` 6` F,G H 2− ( ) 1. Hence 6` F,G < F G ( ) ≤ ≤ ( ) 2− ` ( | ). The left-hand side is a sum, hence the inequality holds for each element of the sum: just what we had to prove. Í

The following monotonicity property will be needed: Lemma 4.5.7 For 7 < 8 we have

7 F 7 <+ 8 F 8 . + ` ( | ) + ` ( | ) Proof. From Lemma 4.5.6, with 5 7,< = 7 < we have ( ) + F 7 F 8 <+ 8 7 <+ 8 7. ` ( | )− ` ( | ) ( − ) −

Let us generalize the minimum property of F . ` ( )

107 4. Generalizations

Proposition 4.5.8 Let G, a G be an upper semicomputable function with ( ) 7→ a( ) values in Z = Z , . Then by Corollary 4.1.3 among the lower semicom- ∪ {−∞ ∞} putable functions F, G,a 6 F,G with aF 6 F,G 2 a G there is one ( ) 7→ a ( ) a ( ) ≤ − ( ) that is maximal to within a multiplicative constant. Choosing 5 F,G as such a a ( ) function we have for all F with G > : a ( ) −∞ G 5 F,G =∗ 2− a ( )t F G, G , a ( ) a ( | a ( )) or in logarithmic notation log 5 F,G =+ G F G, G . − a ( ) a ( )+ a ( | a ( )) Proof. To prove the inequality >∗ , deﬁne 7 6a F, G,; = max 2− ta F G,7 . ( ) 7 ; ( | ) ≥ Function 6 F, G,; is lower semicomputable and decreasing in ;. Therefore a ( ) 6 F,G = 6 F,G, G a ( ) a ( a ( )) is also lower semicomputable since it is obtained by substituting an upper semicomputable function for ; in 6 F, G,; . The multiplicative form of Lemma 4.5.7 a ( ) implies ; 6 F, G,; =∗ 2− t F G,; , a ( ) a ( | ) G 6 F,G =∗ 2− a ( )t F G, G . a ( ) a ( | a ( )) We have, since ta is a test: F ; ; a 2− t F G,; 2− , a ( | ) ≤ F G a 6 F,G <∗ 2− a ( ), a ( ) implying 6 F,G <∗ 5 F,G by the optimality of 5 F,G . a ( ) a ( ) a ( ) To prove the upper bound, consider all lower semicomputable functions q F, G,;,a 4 ( ) (4 = 1, 2,... ). By Theorem 4.1.1, there is a recursive mapping 4 4 with the 7→ ′ property that aF q F, G,;,a 2 ; 1, and for each G,;,a if aF q F, G,;,a < 4′ ( ) ≤ − + 4 ( ) 2 ; 1 then q = q . Let us apply this transformation to the function q F, G,;,a = − + 4 4′ 4 ( ) 5 F,G . The result is a lower semicomputable function 5 F, G,; = q F, G,;,a a ( ) a′( ) 4′ ( ) with the property that aF 5 F, G,; 2 ; 1, further aF 5 F,G 2 ; implies a′( ) ≤ − + a ( ) ≤ − 5 F, G,; = 5 F,G . Now F, G,;,a 2; 1 5 F, G,; is a uniform test of a′( ) a ( ) ( ) 7→ − a′( ) F conditional on G,; and hence it is <∗ t F G,; . Substituting G for ; the a ( | ) a ( ) relation aF 5 F,G 2 ; is satisﬁed and hence we have a ( ) ≤ − G 1 5 F,G = 5 ′ F,G, G <∗ 2− a ( )+ t F G, G . a ( ) a ( a ( )) a ( | a ( ))

108 4.5. Algorithmic entropy

As mentioned above, the theory generalizes to measures that are not probability measures. Taking 5 F,G = 1 in Proposition 4.5.8 gives the inequality ` ( ) F log ` - <+ log ` - , ` ( | ⌊ ( )⌋) ( ) with a physical meaning when ` is the phase space measure. Using (4.5.6), this implies F <+ log ` - log ` - . (4.5.7) ` ( ) ( )+ (⌊ ( )⌋) Theorem 4.5.2 (General addition) The following inequality holds:

F,G =+ F a G F, F a , ` . `,a ( ) ` ( | )+ a ( | ` ( | ) )

Proof. To prove the inequality <+ deﬁne `,a F, G,; = min7 ; 7 a G F,7,` . ( ) ≥ + ( | ) This function is upper semicomputable and increasing in ;. Therefore function

F,G = F,G, F a `,a ( ) `,a ( ` ( | )) is also upper semicomputable since it is obtained by substituting an upper semicomputable function for ; in F, G,; . Lemma 4.5.7 implies `,a ( ) F, G,; =+ ; G F,;,` , `,a ( ) + a ( | ) F,G =+ F a G F, F a ,` . `,a ( ) ` ( | )+ a ( | ` ( | ) ) Now, we have

G ; G F,;,` ; a 2− − a ( | ) 2− , ≤ G F,G F ` a 2− `,a ( ) <∗ 2− ` ( | ) .

Integrating over F by ` gives `F aG2 <∗ 1, implying F,G <+ F,G by − `,a ( ) `,a ( ) the minimality property of F,G . This proves the <+ half of the theorem. `,a ( ) To prove the inequality >+ let 5 F, G,` = 2 `,a F,G . Proposition 4.5.5 a ( ) − ( ) implies that there is a constant 2 with aG 5 F, G,` 2 ` F a 2. Let a ( ) ≤ − ( | )+ F,` = F a . a( ) ` ( | ) Proposition 4.5.8 gives (substituting G for F and F,` for G): ( ) F,G = log 5 F,G,` >+ F,` G F, F,` ,` , `,a ( ) − a ( ) a ( )+ a ( | a ( ) ) which is what needed to be proved.

109 4. Generalizations

The function l behaves quite diﬀerently for diﬀerent kinds of measures ` ( ) `. Recall the following property of complexity:

5 F G <+ F 6 G <+ F . (4.5.8) ( ( ) | ) ( | ( )) ( ) for any computable functions 5 , 6. This implies

G <+ F,G . ( ) ( ) In contrast, if ` is a probability measure then

G >+ l, G . a ( ) `,a ( ) This comes from the fact that 2 a G is a test for ` a. − ( ) × 4.5.4 Information Mutual information has been defined in Definition 3.1.9 as F : G = F ∗ ( ) ( )+ G F,G . By the Addition theorem, we have F : G =+ G G F, F =+ ( )− ( ) ∗ ( ) ( )− ( | ( )) F F G, G . The two latter expressions show that in some sense, ( ) − ( | ( )) F : G is the information held in F about G as well as the information held ∗( ) in G about F. (The terms F , G in the conditions are logarithmic-sized ( ) ( ) corrections to this idea.) Using (4.3.5), it is interesting to view mutual information F : G as a deficiency of randomness of the pair F,G in terms of the ∗( ) ( ) expression d , with respect to m m: ` × ∗ F : G = F G F,G = dm m F,G . ( ) ( )+ ( )− ( ) × ( ) Taking m as a kind of “neutral” probability, even if it is not quite such, allows us to view F : G as a “deficiency of independence”. Is it also true that ∗( ) ∗ F : G =+ dm m F ? This would allow us to deduce, as Levin did, “information ( ) × ( ) conservation” laws from randomness conservation laws.2 Expression dm m F must be understood again in the sense of compactifica- × ( ) tion, as in Section 4.3. There seem to be two reasonable ways to compactify the space N N: we either compactify it directly, by adding a symbol , or we form × ∞ the product N N. With either of them, preserving Theorem 4.6.1, we would × have to check whether F,G m m =+ F,G . But, knowing the function ( | × ) ( ) m F m G we know the function F m F =∗ m F m 0 , hence also ( ) × ( ) 7→ ( ) ( ) × ( ) the function F,G m F,G = m F,G . Using this knowledge, it is possi- ( ) 7→ ( ) (h i) ble to develop an argument similar to the proof of Theorem 4.3.2, showing that F,G m m =+ F,G does not hold. ( | × ) ( ) 2 We cannot use the test t` for this, since it can be shown easily that it does not to obey randomness conservation.

110 4.6. Randomness and complexity

Question 1 Is there a neutral measure " with the property that F : G = ∗ ( ) d" " F,G ? Is this true maybe for all neutral measures "? If not, how far apart × ( ) are the expressions d" " F,G and ∗ F : G from each other? × ( ) ( ) 4.6 Randomness and complexity

We have seen in the discrete case that complexity and randomness are closely related. The connection is more delicate technically in the continuous case, but its exploration led to some nice results.

4.6.1 Discrete space It is known that for computable `, the test d F can be expressed in terms of ` ( ) the description complexity of F (we will prove these expressions below). Assume that X is the (discrete) space of all binary strings. Then we have

d F = log ` F F $ ` . (4.6.1) ` ( ) − ( )− ( )+ ( ( )) The meaning of this equation is the following. The expression log ` F is an − ( ) upper bound (within $ ` ) of the complexity F , and nonrandomness of ( ( )) ( ) F is measured by the diﬀerence between the complexity and this upper bound. Assume that X is the space of inﬁnite binary sequences. Then equation (4.6.1) must be replaced with

< < d` F = sup log ` F ≤ F ≤ $ ` . (4.6.2) ( ) < − ( )− ( )+ ( ( )) For noncomputable measures, we cannot replace $ ` in these relations with ( ( )) anything ﬁnite, as shown in the following example. Therefore however attractive and simple, exp log ` F F is not a universal uniform test of randomness. (− ( )− ( )) Proposition 4.6.1 There is a measure ` over the discrete space X of binary strings such that for each <, there is an F with d F = < < and log ` F F <+ ` ( ) − ( ) − ( )− ( ) 0. Proof. Let us treat the domain of our measure ` as a set of pairs F,G . Let < ( ) F< = 0 , for < = 1, 2,.... For each <, let G< be some binary string of length < with the property F ,G > <. Let ` F ,G = 2 <, and 0 elsewhere. Then ( < <) ( < <) − log ` F ,G F ,G < < = 0. On the other hand, let B F,G be the − ( < <)− ( < <) ≤ − ` ( ) test nonzero only on pairs of strings F,G of the form F ,G : ( ) ( < ) m < B` F<,G = ( ) . ( ) H B< ` F<, H ∈ ( ) 111Í 4. Generalizations

The form of the deﬁnition ensures semicomputability and we also have

` F,G B F,G m < < 1, ( ) ` ( ) ≤ ( ) ÕF,G Õ< therefore B is indeed a test. Hence t F,G >∗ B F,G . Taking logarithms, ` ` ( ) ` ( ) d F ,G >+ < < . ` ( < <) − ( ) The same example shows that the test defined as exp log ` F F (− ( ) − ( )) over discrete sets, does not satisfy the randomness conservation property. Proposition 4.6.2 The test defined as 5 F = exp log ` F F over dis- ` ( ) (− ( )− ( )) crete spaces X does not obey the conservation of randomness. Proof. Let us use the example of Proposition 4.6.1. Consider the function c : F,G F. The image of the measure ` under the projection is ( ) 7→ c` F = ` F,G . Thus, c` F = ` F ,G = 2 <. Then we have ( )( ) G ( ) ( )( <) ( < <) − seen that log 5` F<,G< 0. On the other hand, Í ( ) ≤ log 5 c F ,G = log c` F F =+ < < . c` ( ( < <)) − ( )( <)− ( <) − ( ) Thus, the projection c takes a random pair F ,G into an object F that is very ( < <) < nonrandom (when randomness is measured using the test 5`). In the example, we have the abnormal situation that a pair is random but one of its elements is nonrandom. Therefore even if we would not insist on universality, the test exp log ` F F is unsatisfactory. (− ( )− ( )) Looking into the reasons of the nonconservation in the example, we will notice that it could only have happened because the test 5` is too special. The fact that log c` F F is large should show that the pair F ,G can be − ( )( <)− ( <) ( < <) enclosed into the “simple” set F Y of small probability; unfortunately, this { <}× observation does not reflect on log ` F,G F,G (it does for computable − ( )− ( ) `). It is a natural idea to modify equation (4.6.1) in sucha way that the complexity F is replaced with F ` . However, this expression must be understood ( ) ( | ) properly. We need to use the definition of F as log m F directly, and not ( ) − ( ) as prefix complexity. Let us mention the following easy fact:

Proposition 4.6.3 If ` is a computable measure then F ` =+ F . The con- ( | ) ( ) stant in =+ depends on the description complexity of `.

112 4.6. Randomness and complexity

Theorem 4.6.1 If X is the discrete space Σ∗ then we have

d F =+ log ` F F ` . (4.6.3) ` ( ) − ( )− ( | ) Note that in terms of the algorithmic entropy notation introduced in (4.5.2), this theorem can be expressed as

F =+ F ` log ` F . ` ( ) ( | )+ ( ) Proof. In exponential notation, equation (4.6.3) can be written as t F =∗ m F ` ` F . ` ( ) ( | )/ ( ) Let us prove >∗ first. We will show that the right-hand side of this inequalityisa test, and hence <∗ t F . However, the right-hand side is clearly lower semicom- ` ( ) putable in F,` and when we “integrate” it (multiply it by ` F and sum it), ( ) ( ) its sum is 1; thus, it is a test. ≤ Let us prove <∗ now. The expression t F ` F is clearly lower semicom- ` ( ) ( ) putable in F,` , and its sum is 1. Hence, it is <+ m F ` . ( ) ≤ ( | ) Remark 4.6.4 It important not to consider relative computation with respect to ` as oracle computation in the ordinary sense. Theorem 4.3.1 below will show the existence of a measure with respect to which every element is random. If randomness is defined using ` as an oracle then we can always find elements nonrandom with respect to `. For similar reasons, the proof of the Coding Theorem does not transfer to the function F ` since an interpreter function should have the property of ( | ) intensionality, depending only on ` and not on the sequence representing it. (It does transfer without problem to an oracle version of ` F .) The Coding ( ) Theorem still may hold, at least in some cases: this is currently not known. Until we know this, we cannot interpret F ` as description complexity in terms ( | ) of interpreters and codes. (Thanks to Alexander Shen for this observation: this remark corrects an error in the paper [21].) y

4.6.2 Non-discrete spaces For non-discrete spaces, unfortunately, we can only provide a less intuitive expression. Proposition 4.6.5 let X = -,3,,U be a complete computable metric space, and ( ) let E be the enumerated set of bounded Lipschitz functions introduced in (A.1.3),

113 4. Generalizations but for the space M X X. The uniform test of randomness t F , can be expressed ( )× ` ( ) as m 5 ` t F =∗ 5 `,F ( | ) . (4.6.4) ` ( ) ( ) ` G 5 `, G 5 E Õ∈ ( ) Proof. For >∗ , we will show that the right-hand side of the inequality is a test, and hence <∗ t F . For simplicity, we skip the notation about the enumeration ` ( ) of E and treat each element 5 as its own name. Each term of the sum is clearly lower semicomputable in 5,F,` , hence the sum is lower semicomputable in ( ) F,` . It remains to show that the `-integral of the sum is 1. But, the `- ( ) ≤ integral of the generic term is m 5 ` , and the sum of these terms is 1 by ≤ ( | ) ≤ the deﬁnition of the function m . Thus, the sum is a test. (·|·) For <∗ , note that `,F t F , as a lower semicomputable function, is the ( ) 7→ ` ( ) supremum of functions in E. Denoting their diﬀerences by 5 `,F , we have 7 ( ) t F = 5 `,F . The test property implies `F 5 `,F 1. Since the ` ( ) 7 7 ( ) 7 7 ( ) ≤ function `,7 `F 5 `,F is lower semicomputable, this implies `F 5 `,F <∗ (Í ) 7→ 7 ( ) Í 7 ( ) m 7 ` , and hence ( | ) m 7 ` 57 `,F <∗ 57 `,F ( | ) . ( ) ( ) `F 5 `,F 7 ( ) It is easy to see that for each 5 E we have ∈ m 7 ` ` 5 ` , = ( | ) ≤ ( | ) 7:Õ57 5 which leads to (4.6.4).

Remark 4.6.6 If we only want the >∗ part of the result, then E can be replaced with any enumerated computable sequence of bounded computable functions. y

4.6.3 Inﬁnite sequences In case of the space of inﬁnite sequences and a computable measure, Theo- rem 2.3.4 gives a characterization of randomness in terms of complexity. This theorem does not seem to transfer to a more general situation, but under some conditions, at least parts of it can be extended. For arbitrary measures and spaces, we can say a little less:

114 4.6. Randomness and complexity

Proposition 4.6.7 For all measures ` M - , for the deﬁciency of randomness ∈ ' ( ) d F , we have ` ( ) < < d` F >+ sup log ` F ≤ F ≤ ` . (4.6.5) ( ) < − ( )− ( | ) Proof. Consider the function

< < m A ` m F ≤ ` m F ≤ ` 5 F = 1Γ F ( | ) = ( | ) sup ( | ) . ` A Γ < < ( ) ( ) ` A ` F ≤ ≥ < ` F ≤ ÕA ( ) Õ< ( ) ( ) The function `,F 5 F is clearly lower semicomputable and satisﬁes ( ) 7→ ` ( ) `F 5 F 1, and hence ` ( ) ≤ < < d` F >+ log 5 F >+ sup log ` F ≤ F ≤ ` . ( ) ( ) < − ( )− ( | )

Definition 4.6.8 Let M - be the set of measures ` with ` - = '. y ' ( ) ( ) We will be able to prove the >+ part of the statement of Theorem 2.3.4 in a more general space, and without assuming computability. Assume that a separating sequence 11, 12,... is given as defined in Subsection 4.7, along with the set - 0. For each F - 0, the binary sequence F ,F ,... has been defined. Let ∈ 1 2

` Γ = ' ` Γ : : A = : A′ , A′ ≠ A . ( A) − { ( A′) ( ) ( ) } Õ Then A, ` ` Γ is lower semicomputable, and A, ` ` Γ is upper ( ) 7→ ( A) ( ) 7→ ( A) semicomputable. And, every time that the functions 1 F form a partition with 7 ( ) `-continuity, we have ` Γ = ` Γ for all A. ( A) ( A) Theorem 4.6.2 Suppose that the space - is eﬀectively compact. Then for all computable measures ` M0 - , for the deﬁciency of randomness d F , the charac- ∈ ' ( ) ` ( ) terization (2.3.3) holds.

Proof. The proof of part >+ of the inequality follows directly from Proposition 4.6.7, just as in the proof of Theorem 2.3.4. The proof of <+ is also similar to the proof of that theorem. The only part that needs to be reproved is the statement that for every lower semicomputable function 5 over -, there are computable sequences G N and ? Q with 7 ∈ ∗ 7 ∈ 5 F = sup ? 1 F . This follows now, since according to Proposition 4.7.7, the ( ) 7 7 G7 ( ) cells ΓG form a basis of the space -.

115 4. Generalizations

4.6.4 Bernoulli tests In this part, we will give characterize Bernoulli sequences in terms of their complexity growth. Recall Definition 4.2.11: A function 5 : B R is a combinatorial Bernoulli ∗ → test if a) It is lower semicomputable. b) It is monotonic with respect to the prefix relation. < c) For all 0 9 < we have F B <,9 5 F 9 . ≤ ≤ ∈ ( ) ( ) ≤ According to Theorem 4.2.3, a sequence b is nonrandom with respect to all Í Bernoulli measures if and only if sup 1 b < = , where 1 F is a combinatorial < ( ≤ ) ∞ ( ) Bernoulli test. We need some definitions. < Definition 4.6.9 For a finite or infinite sequence F let (< F = 7=1 F 7 . ( ) < ( ) For 0 > 1 and integers 0 9 <, denote <, 9 = >9 1 > < 9. ≤ ≤ ≤ ≤ >( ) 9 ( − ) − An upper semicomputable function : N2 N, defined forÍ< 1, 0 9 → ≥ ≤ ≤ < will be called a gap function if < <,9 <, 9 2− ( ) 1 (4.6.6) >( ) ≤ < 1 9=0 Õ≥ Õ holds for all 0 > 1. A gap function <, 9 is optimal if for every other gap ≤ ≤ ( ) function <, 9 there is a 2 with <, 9 <, 9 2 . y ′( ) ′ ( ) ≤ ′( )+ ′ Proposition 4.6.10 There is an optimal gap function <, 9 <+ < . ( ) ( ) Proof. The existence is proved using the technique of Theorem 4.1.1. For the inequality it is sufficient to note that < is a gap function. Indeed, we have ( ) < < < <, 9 2− ( ) = 2− ( ) 1. >( ) ≤ < 1 9=0 < 1 Õ≥ Õ Õ≥ Definition 4.6.11 Let us fix an optimal gap function and denote it by Δ <, 9 . ( )y

Now we can state the test characterization theorem for Bernoulli tests. Theorem 4.6.3 Denoting by 1 b the universal class test for the Bernoulli sequen- ( ) ces, we have 1 b =∗ 1 b , where ( ) ( ) < < log 1 b = sup log b≤ <, 9, Δ <, 9 Δ <, 9 , ( ) 9 − ( | ( )) − ( ) <

116 4.6. Randomness and complexity with 9 = ( b . < ( ) Proof. Let X <, 9 = 2 Δ <,9 . ( ) − ( ) Claim 4.6.12 Consider lower semicomputable functions W : B∗ R such that → + for all <, 9 we have

Δ <,9 W G 2− ( ) . ( ) ≤ G B <,9 ∈Õ( ) Among these functions there is one that is optimal (maximal to within a multiplicative constant). Calling it X G we have ( ) X G =∗ X <, 9 m G <, 9, Δ <, 9 . ( ) ( ) · ( | ( )) Thus, the right-hand side is equal, within a multiplicative constant, to a lower semicomputable function of G.

Proof. This follows immediately from Proposition 4.5.8. with a = # (the counting measure) over the sets B <, 9 . ( ) Let us show 1 b <∗ 1 b . We have with 9 = ( b in the ﬁrst line: ( ) ( ) < ( )

< < 1 b = sup m b≤ <, 9, Δ <, 9 X <, 9 ( ) < 1 9 ( | ( )) ( ) ≥ < < = sup X <, 9 1G b m G <, 9, Δ <, 9 < 1 9 ( ) ( ) ( | ( )) 9=0 G B <,9 ≥ Õ ∈Õ( ) < < 1 b X <, 9 m G <, 9, Δ <, 9 ≤ 9 G ( ) ( ) ( | ( )) < 1 9=0 G B <,9 Õ≥ Õ ∈Õ( ) < < =∗ 1G b X G , 9 ( ) ( ) < 1 9=0 G B <,9 Õ≥ Õ ∈Õ( ) using the notation of Claim 4.6.12 above. Let B b denote the right-hand side ( ) here, which is thus a lower semicomputable function. We have for all >:

< b B b =∗ <, 9 X <, 9 m G <, 9, Δ <, 9 1, > ( ) >( ) ( ) ( | ( )) ≤ < 1 9=0 G B <,9 Õ≥ Õ ∈Õ( ) so B b >∗ 1 b is a Bernoulli test, showing 1 b <∗ 1 b . ( ) ( ) ( ) ( )

117 4. Generalizations

To show 1 b <∗ 1 b we will follow the method of the proof of Theo- ( ) ( ) rem 2.3.4. Replace 1 b with a rougher version: ( ) 1 < < 1′ b = max 2 :2 < 1 b , ( ) 2 { ( )} then we have 21′ < 1. There are computable sequences G7 B∗ and 97 N with = 97 ∈ = ∈ = 1′ b sup7 2 1G7 b with the property that if 7 < 8 and 1G7 b 1G8 b 1 ( ) ( ) ( ) 97 ( ) then 97 < 9 8. As in the imitated proof, we have 21′ b 7 2 1G7 b . The 9 ( ) ≥ ( ) function W G = = 2 7 is lower semicomputable. We have ( ) G7 G Í < Í 9 1 21′ b 2 7 1 b = 1 b W G = 1 b W G . ≥ ( ) ≥ G7 ( ) G ( ) ( ) G ( ) ( ) 7 G N < 9=0 G B <,9 Õ Õ∈ ∗ Õ Õ ∈Õ( ) (4.6.7) < By Theorem 4.2.3 we can assume G B <,9 W G 9 . Let ∈ ( ) ( ) ≤ 1 Í < − X′ G = W G 1, ( ) ( ) 9 ≤ X′ <, 9 = X′ G . ( ) ( ) G B <,9 ∈Õ( ) Since 1 1 21 for all >, we have ≥ > ≥ >( ′) < < b 9 < 9 1 W G 1 b = > 1 > − W G ≥ ( ) > G ( ) ( − ) ( ) < 9=0 G B <,9 < 9=0 G B <,9 Õ Õ ∈Õ( ) Õ Õ ∈Õ( ) < < = <, 9 X′ G = <, 9 X′ <, 9 . >( ) ( ) >( ) ( ) < 9=0 G B <,9 < 9=0 Õ Õ ∈Õ( ) Õ Õ Δ Thus X <, 9 is a gap function, hence X <, 9 <∗ 2 <,9 , and by Claim 4.6.12 ′( ) ′( ) − ( ) we have 1 < − W G = X′ G <∗ X G =∗ X <, 9 m G <, 9, Δ <, 9 . ( ) 9 ( ) ( ) ( ) · ( | ( )) Substituting back into (4.6.7) ﬁnishes the proof of 1 b <∗ 1 b . ( ) ( ) 4.7 Cells

This section allows to transfer some of the results on sequence spaces to more general spaces, by encoding the elements into sequences. The reader who is only interested in sequences can skip this section.

118 4.7. Cells

4.7.1 Partitions The coordinates of the sequences into which we want to encode our elements will be obtained via certain partitions. Recall from Definition A.2.33 that a measurable set is said to be a `- continuity set if ` m = 0 where m is the boundary of . ( ) Definition 4.7.1 (Continuity partition) Let 5 : - R be a bounded com- → putable function, and let U < < U be rational numbers, and let ` be a 1 · · · 9 computable measure with the property that ` 5 1 U = 0 for all 8. − ( 8) In this case, we will say that U 8 are `-continuity points of 5 . Let U0 = , 1 −∞ U9 1 = , and for 8 = 0, . . . , 9, let Let * 8 = 5 − 8, 8 1 . The sequence of dis- + ∞ (( + )) joint computably enumerable open sets * ,...,* will be called the partition ( 0 9) generated by 5,U1,...,U9. If we have several partitions * ,...,* , generated by different functions ( 70 7,9) 5 (7 = 1,...,;) and cutoff sequences U : 8 = 1, . . . , 9 made up of `- 7 ( 78 7) continuity points of 57 then we can form a new partition generated by all possible intersections + = * * . 81,...,8< 1,81 ∩···∩ ;,8;

A partition of this kind will be called a continuity partition. The sets +81,...,8< will be called the cells of this partition. y The following is worth noting.

Proposition 4.7.2 In a partition as given above, the values `+81,...,8< are computable from the names of the functions 57 and the cutoﬀ points U78. Proof. Straightforward.

Let us proceed to defining cells. Definition 4.7.3 (Separating sequence) Assume that a computable sequence of functions 1 F , 1 F ,... over - is given, with the property that for every pair 1 ( ) 2 ( ) F ,F - with F ≠ F , there is a 8 with 1 F 1 F < 0. Such a sequence 1 2 ∈ 1 2 8 ( 1) · 8 ( 2) will be called a separating sequence. Let us give the correspondence between the set BN of infinite binary sequences and elements of the set

- 0 = F - : 1 F ≠ 0, 8 = 1, 2,... . { ∈ 8 ( ) } For a binary string A A = A B , let 1 · · · < ∈ ∗

ΓA

119 4. Generalizations

be the set of elements of - with the property that for 8 = 1,...,<, if A 8 = 0 then 1 l < 0, otherwise 1 l > 0. 8 ( ) 8 ( ) The separating sequence will be called `-continuity sequence if ` - 0 = ( ) 0. y This correspondence has the following properties. a) ΓΛ = -. b) For each A B, the sets Γ and Γ are disjoint and their union is contained ∈ A0 A1 in ΓA. 0 Γ c) For F - , we have F = F Γ A. ∈ { } ∈ A Γ Deﬁnition 4.7.4 (Cells) If stringÑA has length < then A will be called a canonical <-cell, or simply canonical cell, or <-cell. From now on, whenever Γ denotes a subset of -, it means a canonical cell. We will also use the notation

: Γ = : A . ( A) ( ) y The three properties above say that if we restrict ourselves to the set - 0 then the canonical cells behave somewhat like binary subintervals: they divide - 0 in half, then each half again in half, etc. Moreover, around each point, these canonical cells become “arbitrarily small”, in some sense (though, they may not Γ Γ be a basis of neighborhoods). It is easy to see that if A1 , A2 are two canonical cells then they either are disjoint or one of them contains the other. If Γ Γ A1 ⊂ A2 then A is a preﬁx of A . If, for a moment, we write Γ0 = Γ - 0 then we have 2 1 A A ∩ the disjoint union Γ0 = Γ0 Γ0 . A A0 ∪ A1 Let us use the following notation. Deﬁnition 4.7.5 For an <-element binary string A, for F Γ , we will write ∈ A < A = F ≤ = F F , ` A = ` Γ . 1 · · · < ( ) ( A) The 2< cells (some of them possibly empty) of the form Γ for : A = < form a A ( ) partition P< of - 0. y 0 Thus, for elements of - , we can talk about the <-th bit F< of the description of F: it is uniquely determined. Examples 4.7.6

120 4.7. Cells

1. If X is the set of inﬁnite binary sequenceswith its usual topology, the functions 1 F = F 1 2 generate the usual cells, and X0 = X. < ( ) < − / 2. If X is the interval 0, 1 , let 1 F = sin 2 0 . Let G = > such that G >. Then G , and com- 7 ( ) } ( ) ∈ ⊂ G ( ) pactness implies that there is a ﬁnite sequence G ,...,G with 9 G . 1 9 Ð ⊂ 8=1 ( 8) Clearly, there is a cell Ð 9

F ΓA F,@ G8 . ∈ ⊂ ( )\ = ( ) Ø8 1

3It was noted by Hoyrup and Rojas that the qualiﬁcation “eﬀectively” is necessary here.

121 4. Generalizations

4.7.2 Computable probability spaces If a separating sequence is given in advance, we may restrict attention to the class of measures that make our sequence a `-continuity sequence: Deﬁnition 4.7.8 Let M0 - be the set of those probability measures ` in M - ( ) ( ) for which ` - - 0 = 0. y ( \ ) On the other hand, for each computable measure `, a computable separating sequence can be constructed that is a `-continuity sequence. Recall that F,@ ( ) is the ball of center F and radius @. Let = A , A ,... = U 1 ,U 2 ,... be { 1 2 } { ( ) ( ) } the canonical enumeration of the canonical dense set . Theorem 4.7.1 (Hoyrup-Rojas) There is a sequence of uniformly computable reals @< < N such that A7,@< 7,< is a basis of balls that are `-continuity sets. This ( ) ∈ ( ( )) basis is constructively equivalent to the original one, consisting of all balls A ,@ , ( 7 ) @ Q. ∈ Corollary 4.7.9 There is a computable separating sequence with the `-continuity property.

Proof. Let us list all balls A ,@ into a single sequence A ,@ . The func- ( 7 <) ( 79 <9 ) tions

1 F = 3 A ,F @ 9 ( ) ( 79 )− <9 give rise to the desired sequence. For the proof of the theorem, we use some preparation. Recall from Deﬁni- tion A.2.5 that an atom is a point with positive measure. Lemma 4.7.10 Let - be R or R or 0, 1 . Let ` be a computable probability + [ ] measure on -. Then there is a sequence of uniformly computable reals F which ( <)< is dense in - and contains no atoms of `.

Proof. Let be a closed rational interval. We construct F with ` F = 0. ∈ ({ }) To do this, we construct inductively a nested sequence of closed intervals 9 of 9 1 measure < 2− + , with 0 = . Suppose 9 = 0, 1 has been constructed, with 9 1 [ ] ` 9 < 2− + . Let ; = 1 0 3: one of the intervals 0, 0 ; and 1 ;, 1 ( ) (9 − )/ [ + ] [ − ] must have measure < 2− , and we can ﬁnd it eﬀectively—let it be 9 1. + From a constructive enumeration of all the dyadic intervals, we can ( <)< construct F uniformly. < ∈ <

122 4.7. Cells

Corollary 4.7.11 Let X,` be a computable metric space with a computable mea- ( ) sure and let 5 be a sequence of uniformly computable real valued functions on ( 7)7 -. Then there is a sequence of uniformly computable reals F that is dense in R ( <)< and such that each F< is a `-continuity point of each 57. = 1 Proof. Consider the uniformly computable measures `7 ` 57− and deﬁne 7 ◦ a = 7 2− `7. It is easy to see that a is a computable measure and then, by Lemma 4.7.10, there is a sequence of uniformly computable reals F which ( <)< is denseÍ in R and contains no atoms of a. Since a = 0 iﬀ ` = 0 for all 7, ( ) 7 ( ) we get ` 5 1 F = 0 for all 7,<. ({ 7− ( <)}) Proof of Theorem 4.7.1. Apply Corollary 4.7.11 to 5 F = 3 A ,F . 7 ( ) ( 7 ) Since every ball A ,@ can be expressed as a computably enumerable union ( 7 ) of the balls of the type A ,@ just constructed, the two bases are constructively ( 7 <) equivalent.

123

5 Exercises and problems

@ Exercise 1 Deﬁne, for any two natural numbers @, A, a standard encoding cnvA of base @ strings F into base A strings with the property log @ cnv@ F F 1. (5.0.1) | A ( )|≤| | log A +

# # Solution. We will use X = # , X@ = /@ for the sets of inﬁnite strings of natural numbers and @-ary digits respectively. For a sequence > X , let > denote ∈ @ [ ]@ the real number in the interval 0, 1 which 0.> denotes in the base @ number [ ] system. For > in S , let > = >? : ? X . @ [ ]@ { [ ]@ ∈ @ } For the @-ary string F, let { be the size of the largest A-ary intervals G [ ]A contained in the @-ary interval F . If H is the leftmost among these intervals, [ ]@ [ ]A then let cnv@ F = H. This is a one-to-one encoding. We have @ F < 2A{, since A ( ) −| | any 2A consecutive A-ary intervals of length { contain an A-ary interval of length A{. Therefore log @ log 2 H = log { < F 1 | | − A | | log A + + log A hence, since 2 A and F is integer, we have the inequality (5.0.1). ≤ | | Exercise 2 A function from S S to S is called an @-ary interpreter. Prove @ × the following generalization of the Invariance Theorem. For any A, there is a p.r. A-ary interpreter * such that for any p.r. interpreter there is a constant 2 < such that for all F,G we have ∞ F G F G 2. (5.0.2) * ( | ) ≤ ( | )+ Solution. Let + : Z Z S S be a partial recursive function which is ∗A × ∗A × → universal: such that for any p.r. A-ary interpreter , there is a string 0 such that for all >, F, we have >, F = + 0, >,F . ( ) ( ) The machine computing * >, F tries to decompose > into C={ and outputs ( ) + C, {,F . Let us verify that * is optimal. Let be a p.r. @-ary interpreter, an ( ) 125 5. Exercises and problems

A-ary p.r. interpreter such that cnv@ > ,F = >, F for all >, F, 0 a binary ( A ( ) ) ( ) string such that >, F = * 0, >,F for all >, F. Let F,G be two strings. If ( ) ( ) F G = , then the inequality (5.0.2) holds trivially. Otherwise, let > be a ( | ) ∞ binary string of length F G log @ with >, G = F. Then ( | )/ ( ) * 0=cnv@ > ,G = + 0, cnv@ > ,G = cnv@ > ,G = >, G = F. ( A ( ) ) ( A ( ) ) ( A ( ) ) ( ) Since cnv@ > > log @ log A 1 = F G log A 1, | A ( )|≤| | / + ( | )/ + we have

F G 2 0 F G log A 1 log A = F G 2 0 1 log A. * ( | ) ≤ ( | |+ ( | )/ + ) ( | ) + ( | |) + )

Exercise 3 ((; -theorem) Prove that there is a binary string 1 such that * >,?,F = < ( ) * 1= >=?, F holds for all binary strings >, ? and arbitrary strings x. ( ) Exercise 4 (Schnorr) Notice that, apart from the conversion between representations, what our optimal interpreter does is the following. It treats the program > as a pair > 1 , > 2 whose ﬁrst member is the Gödel number of some in- ( ( ) ( )) terpreter for the universal p.r. function + > , > ,F , and the second argument ( 1 2 ) as a program for this interpreter. Prove that indeed, for any recursive pairing function | >, ? , if there is a function 5 such that | >, ? 5 > ? then | ( ) | ( )| ≤ ( ) + | | can be used to deﬁne an optimal interpreter.

Exercise 5 Refute the inequality F,G <+ F G . ( ) ( )+ ( ) Exercise 6 Prove the following sharpenings of Theorem 1.4.2.

F,G <+ F G F, F , ( ) ( ( )) + ( | ( )) F,G <+ F G F, F . ( ) ( )+ ( ( | ( ))) Prove Theorem 1.4.2 from here.

Exercise 7 (Schnorr) Prove F F =+ F . Can you generalize this result? ( + ( )) ( ) Exercise 8 (Schnorr) Prove that if ;<< then ; ; <+ < < . + ( ) + ( ) Exercise 9 Prove

< < < 1 < log =+ 9 log < 9 log log . 9 9 + ( − ) < 9 + 2 9 < 9 − ( − )

126 Exercise 10 (Kamae) Prove that for any natural number 9 there is a string F such that for for all but ﬁnitely many strings G, we have

F F G 9. ( )− ( | ) ≥ In other words, there are some strings F such that almost any string G contains a large amount of information about them. Solution. Strings F with this property are for example the ones which contain information obtainable with the help of any sufficiently large number. Let be a r.e. set of integers. Let 404142 ... be the infinite string which is the characteristic function of , and let F 9 = 4 ...4 9 . We can suppose that F 9 9. Let ( ) 0 2 ( ( )) ≥ < ,< ,... be a recursive enumeration of without repetition, and let U 9 = 1 2 ( ) max 7 : < 29 . Then for any number B U 9 we have F 9 B <+ log 9. { 7 ≤ } ≥ ( ) ( ( ) | ) Indeed, a binary string of length 9 describes the number 9. Knowing B we can enumerate < ,...,< and thus learn F 9 . Therefore with any string G of length 1 B ( ) U 9 we have F F G >+ 9 log 9. ≥ ( ) ( )− ( | ) − Exercise 11 a) Prove that a real function 5 is computable iff there is a recursive function 6 F,< with rational values, and 5 F 6 F,< < 1 <. ( ) | ( )− ( )| / b) Prove that a function 5 is semicomputable iff there exists a recursive function with rational values, (or, equivalently, a computable real function) 6 F,< ( ) nondecreasing in <, with 5 F = lim< 6 F,< . ( ) →∞ ( ) Exercise 12 Prove that in Theorem 1.5.2 one can write “semicomputable” for “partial recursive”. Exercise 13 (Levin) Show that there is an upper semicomputable function >,F, G ( ) which for different finite binary strings > enumerates all upper semicomputable functions F,G satisfying the inequality (1.5.1). Prove ( ) F G =+ inf >,F, G > . ( | ) > ( )+ (| |) Exercise 14 Prove m > =∗ m < . ( ) ( ) > B< Õ∈ Exercise 15 (Solovay) Show that we cannot find effectively infinitely many places where some recursive upper bound of < is sharp. Moreover, suppose ( ) that < is a recursive upper bound of < . Then there is no recursive function ( ) ( ) D < ordering to each < a finite set of natural numbers (represented for exam- ( ) ple as a string) larger than < such that for each < there is an F in D < with ( ) 127 5. Exercises and problems

F F . Notice that the function log < (or one almost equal to it) has this ( ) ≤ ( ) property for < . ( ) Solution. Suppose that such a function exists. Then we can select a sequence < < < < ... of integers with the property that the sets D < are all disjoint. 1 2 ( 7) Let F 0 < = 2− ( ) . ( ) F D < ∈Õ( ) Then the sequence 0 <9 is computable and 9 0 <9 < 1. It is now easy to ( ) = ( ) construct a computable subsequence ;7 <97 of <9 and a computable sequence 1 such that 1 0 ; and 1 < 1. LetÍ us deﬁne the semimeasure ` 7 7/ ( 7)→∞ 7 7 setting Í F ` F = 2− ( )1 0 ; ( ) 7/ ( 7) for any F in D ; and 0 otherwise. Then for any F in D ; we have F <+ ( 7) ( 7) ( ) log ` F = F log 2 where 2 = 1 0 ; , so we arrived a contradic- − ( ) ( )− 7 7 7/ ( 7)→∞ tion with the assumption.

Exercise 16 Prove that there is a recursive upper bound < of < and a ( ) ( ) constant 2 with the property that there are infinitely many natural numbers < such that for all 9 > 0, the quantity of numbers F < with F < F 9 is 9 ≤ ( ) ( )− less than 2<2− . Solution. Use the upper bound found in Theorem 1.7.4 and define < = ( ) log < log < . The property follows from the facts that log < log < is + (⌊ ⌋) + (⌊ ⌋) a sharp upper bound for F for “most” F less than < and that 9 is infinitely ( ) ( ) often close to 9 . ( ) Exercise 17 Give an example of a computable sequence 0< > 0 of with the property that 0 < but for any other computable sequence 1 > 0, if < < ∞ < 1< 0< 7<5BG then < 1< = . / → Í ∞ Hint: Let @< be a recursive,Í increasing sequence of rational numbers with lim< @< = F m F and let 0< = @< 1 @<. ( ) + − ÍExercise 18 (Universal coding, Elias) Let 5 < = log2 < 2 log2 log2 <. Show ( ) + that when % runs over all nonincreasing probability distributions over # then

1 lim % − % < 5 < = 1. % ( ) ( ) ( ) ( )→∞ Õ<

128 Exercise 19 (T. Cover) Let log∗ < = log < log log < ... (all positive terms). 2 2 + 2 2 + Prove that log < 2− 2∗ < , < ∞ Õ hence < <+ log∗ <. For which logarithm bases does < <+ log∗ < hold? ( ) 2 ( ) 2 Exercise 20 Prove that Kamae’s result in Exercise 10 does not hold for F G . ( | ) Exercise 21 Prove that for each Y there is an ; such that if H % > ; then ( ) % F F H % 1 < Y. | F ( ) ( )/ ( )− | ExerciseÍ 22 If a ﬁnite sequence F of length < has large complexity then its bits are certainly not predictable. Let ℎ 9,< = 9 < :=6 9 < 1 9 < :=6 1 ( ) −( / ) ( / )−( −( / )) ( − 9 < . A quantitative relation of this sort is the following. If, for some 9>< 2, ( / )) / a program of length ; can predict F7 from F1,...,F7 1 for at least < 9 values − − of 7 then F < ; <ℎ 9,< = < . ( ) + ( )+ ( ) Exercise 23 This is a more open-ended problem, having to do with the converse of the previous exercise. Show (if true) that for each 2 there is a 3 such that predictability of F at signiﬁcantly more than < 2 2 places is equivalent to the / + possibility to enclose F into a set of complexity = < and size < 3<. ( ) −

129

A Background from mathematics

A.1 Topology

In this section, we summarize the notions and results of topology that are needed in the main text.

A.1.1 Topological spaces A topological space is a set of points with some kind of—not quantitatively expressed—closeness relation among them. Definition A.1.1 A topology on a set - is defined by a class g of its subsets called open sets. It is required that the empty set and - are open, and that arbitrary union and finite intersection of open sets is open. The pair -, g is called a ( ) topological space. A set is called closed if its complement is open. y Having a set of open and closed sets allows us to speak about closure operations. Definition A.1.2 A set is called the neighborhood of a set if contains an open set that contains . We denote by , o the closure (the intersection of all closed sets containing ) and the interior of (the union of all open sets in ) respectively. Let m = o \ denote the boundary of set . y An alternative way of defining a topological space is via a basis. Definition A.1.3 A basis of a topological space is a subset V of g such that every open set is the union of some elements of V. A neighborhood of a point is a basis element containing it. A basis of neighborhoods of a point F is a set # of neighborhoods of F with the property that each neighborhood of F contains an

131 A. Background from mathematics element of #. A subbasis is a subset f of g such that every open set is the union of finite intersections from f. y Examples A.1.4 1. Let - be a set, and let V be the set of all points of -. The topology with basis V is the discrete topology of the set -. In this topology, every subset of - is open (and closed). 2. Let - be the real line R, and let V be the set of all open intervals 0, 1 . The R ( ) topology gR obtained from this basis is the usual topology of the real line. When we refer to R as a topological space without qualification, this is the topology we will always have in mind. 3. Let - = R = R , , and let V consist of all open intervals 0, 1 and ∪ {−∞ ∞} R ( ) in addition of all intervals of the forms ,0 and 0, . It is clear how [−∞ ) ( ∞] the space R is defined. + > 4. Let - be the real line R. Let VR be the set of all open intervals , 1 . The > (−∞ ) topology with basis VR is also a topology of the real line, different from the usual one. Similarly, let V< be the set of all open intervals 1, . R ( ∞) 5. Let Σ be a finite or countable alphabet. On the space ΣN of infinite sequences with elements in Σ, let g = ΣN : Σ be called the topology of the { ⊆ ∗ } Cantor space (over Σ). Note that if a set has the form = ΣN where is finite then is both open and closed. y

Starting from open sets, we can deﬁne some other kinds of set that are still relatively simple:

Definition A.1.5 A set is called a X set if it is a countable intersection of open sets, and it is an f set if it is a countable union of closed sets. y Different topologies over the same space have a natural partial order relation among them: Definition A.1.6 A topology g on - is called larger, or finer than g if g g. ′ ′ ⊇ For two topologies g , g over the same set -, we define the topology g g = 1 2 1 ∨ 2 g g , and g g as the smallest topology containing g g . In the example 1 ∩ 2 1 ∧ 2 1 ∪ 2 topologies of the real numbers above, we have g = g< g>. y R R ∧ R Most topologies used in practice have some separion property.

Deﬁnition A.1.7 A topology is said to have the )0 property if every point is determined by the class of open sets containing it. This is the weakest one of a number of other possible separation y

132 A.1. Topology

Both our above example topologies of the real line have this property. All topologies considered in this survey will have the )0 property. A stronger such property is the following:

Definition A.1.8 A space is called a Hausdorff space, or )2 space, if for every pair of different points F,G there is a pair of disjoint open sets , with F , ∈ G . y ∈ > The real line with topology gR in Example A.1.4.4 above is not a Hausdorff space. A space is Hausdorff if and only if every open set is the union of closures of basis elements.

A.1.2 Continuity We introduced topology in order to be able to speak about continuity: Definition A.1.9 Given two topological spaces - , g (7 = 1, 2), a function ( 7 7) 5 : - - is called continuous if for every open set - its inverse image 1 → 2 ⊆ 2 5 1 is also open. If the topologies g , g are not clear from the context then − ( ) 1 2 we will call the function g , g -continuous. Clearly, the property remains the ( 1 2) same if we require it only for all elements of a subbasis of -2. If there are two continuous functions between - and . that are inverses of each other then the two spaces are called homeomorphic. We say that function 5 is continuous at point F if for every neighborhood + of 5 F there is a neighborhood * of F with 5 * +. y ( ) ( ) ⊆ Clearly, function 5 is continuous if and only if it is continuous in each point. There is a natural sense in which every subset of a topological space is also a topological space: Definition A.1.10 A subspace of a topological space -, g is defined by a subset ( ) . -, and the topology g = . : g , called the induced topology ⊆ . { ∩ ∈ } on .. This is the smallest topology on . making the identity mapping F F 7→ continuous. A partial function 5 : - / with dom 5 = . is continuous iff → ( ) 5 : . / is continuous. y → Given some topological spaces, we can also form larger ones using for example the product operation: Definition A.1.11 For two topological spaces - , g (7 = 1, 2), we define the ( 7 7) product topology on their product - .: this is the topology defined by the × subbasis consisting of all sets - and all sets - with g . The product 1× 2 1× 2 7 ∈ 7 topology is the smallest topology making the projection functions F,G F, ( ) 7→

133 A. Background from mathematics

F,G G continuous. Given topological spaces -,.,/ we call a two-argument ( ) 7→ function 5 : - . / continuous if it is continuous as a function from - . × 7→ × to /. The product topology is defined similarly for the product 7 -7 of an arbitrary number of spaces, indexed by some index set . We say that∈ a function is g ,...,g ,` -continuous if it is g g ,` -continuous.Î y ( 1 < ) ( 1 ×···× < ) Examples A.1.12 1. The space R R with the product topology has the usual topology of the × Euclidean plane. 2. Let - be a set with the discrete topology, and - N the set of infinite sequences with elements from -, with the product topology. A basis of this topology is provided by all sets of the form C- N where C - . The elements of this basis ∈ ∗ are closed as well as open. When - = 0, 1 then this topology is the usual { } topology of infinite binary sequences. y

In some special cases, one of the topologies in the definition of continuity is fixed and known in advance: Definition A.1.13 A real function 5 : - R is called continuous if it is 1 → g , g -continuous. y ( 1 R) A.1.3 Semicontinuity For real functions, a restricted notion of continuity is often useful. Definition A.1.14 A function 5 : - R is lower semicontinuous if the set → F,@ - R : 5 F > @ is open. The definition of upper semicontinuity is { ( ) ∈ × ( ) } similar. Lower semicontinuity on subspaces is defined similarly to continuity on subspaces. y Clearly, a real function 5 is continuous if and only if it is both lower and upper semicontinuous. The requirement of lower semicontinuity of 5 is equivalent to saying that for each @ R, the set F : 5 F > @ is open. This can be seen to ∈ { ( ) } be equivalent to the following characterization. Proposition A.1.15 A real function 5 over X = -, g is lower semicontinuous if ( ) and only if it is g, g< -continuous. ( R) Example A.1.16 The indicator function 1 F of an arbitrary open set is lower ( ) semicontinuous. y The following is easy to see.

134 A.1. Topology

Proposition A.1.17 The supremum of any set of lower semicontinuous functions is lower semicontinuous. The following representation is then also easy to see. Proposition A.1.18 Let - be a topological space with basis V. The function 5 : - R is lower semicontinuous if and only if there is a function 6 : V R with → + → + 5 F = supF V 6 V . ( ) ∈ ( ) Corollary A.1.19 Let - be a topological space with basis V and 5 : - R a → + lower semicontinuous function deﬁned on a subset of -. Then 5 can be extended in a lower semicontinuous way to the whole space -.

Proof. Indeed by the above proposition there is a function 6 : V R with → + 5 F = supF V 6 V for all F . Let us define 5 by this same formula for all ( ) ∈ ( ) ∈ F -. ∈ In the important special case of Cantor spaces, the basis is given by the set of finite sequences. In this case we can also require the function 6 | to be ( ) monotonic in the words |: Proposition A.1.20 Let - = ΣN be a Cantor space as defined in Example A.1.4.5. Then 5 : - R is lower semicontinuous if and only if there is a function 6 : Σ → + ∗ R monotonic with respect to the relation C {, with 5 b = supC b 6 C . → + ⊑ ( ) ⊑ ( ) A.1.4 Compactness There is an important property of topological spaces that, when satisfied, has many useful implications. Definition A.1.21 Let -, g be a topological space, and a subset of -. An ( ) open cover of is a family of open sets whose union contains . A subset of - is said to be compact if every open cover of has a finite subcover. y Compact sets have many important properties: for example, a continuous function over a compact set is bounded. Example A.1.22 1. Every finite discrete space is compact. An infinite discrete space X = -, g ( ) is not compact, but it can be turned into a compact space X by adding a new element called : let - = - , and g = g - : - closed . ∞ ∪ {∞} ∪{ \ ⊂ } More generally, this simple operation can be performed with every space that is locally compact, that each of its points has a compact neighborhood.

135 A. Background from mathematics

2. In a ﬁnite-dimensional Euclidean space, every bounded closed set is compact.

3. It is knownthat if X7 7 is a family of compact spaces then their direct prod- ( ) ∈ uct is also compact. y

There are some properties that are equivalent to compactness in simple cases, but not always: Deﬁnition A.1.23 A subset of - is said to be sequentially compact if every sequence in has a convergent subsequence with limit in . The space is locally compact if every point has a compact neighborhood. y

A.1.5 Metric spaces Metric spaces are topological spaces with more structure: in them, the closeness concept is quantifiable. In our examples for metric spaces, and later in our treatment of the space of probability measures, we refer to [5]. Definition A.1.24 A metric space is given by a set - and a distance function 3 : - - R satisfying the triangle inequality 3 F, H 3 F,G 3 G, H × → + ( ) ≤ ( ) + ( ) and also property that 3 F,G = 0 implies F = G. For @ R , the sets ( ) ∈ + F,@ = G : 3 F,G < @ , F,@ = G : 3 F,G @ ( ) { ( ) } ( ) { ( ) ≤ } are called the open and closed balls with radius @ and center F. A metric space is bounded when 3 F,G has an upper bound on -. y ( ) A metric space is also a topological space, with the basis that is the set of all open balls. Over this space, the distance function 3 F,G is obviously continu- ( ) ous. Each metric space is a Hausdorff space; moreover, it has the following stronger property.

Definition A.1.25 A topological space is said to have the )3 property if for every pair of different points F,G there is a continuous function 5 : - R with → 5 F ≠ 5 G . y ( ) ( ) To see that metric spaces are ) , take 5 H = 3 F, H . 3 ( ) ( ) Definition A.1.26 A topological space is called metrizable if its topology can be derived from some metric space. y

It is known that a topological space is metrizable if and only if it has the )3 property.

136 A.1. Topology

Notation A.1.27 For an arbitrary set and point F let

3 F, = inf 3 F,G , ( ) G ( ) ∈ Y = F : 3 F, < Y . (A.1.1) { ( ) } y

Examples A.1.28 1. A discrete topological space - can be turned into a metric space as follows: 3 F,G = 0 if F = G and 1 otherwise. ( ) 2. The real line with the distance 3 F,G = F G is a metric space. The ( ) | − | topology of this space is the usual topology gR of the real line. 3. The space R R with the Euclidean distance gives the same topology as the × product topology of R R. × 4. An arbitrary set - with the distance 3 F,G = 1 for all pairs F,G of different ( ) elements, is a metric space that induces the discrete topology on -. 5. Let - be a bounded metric space, and let . = - N be the set of infinite sequences F = F ,F ,... with distance function 3N F,G = 2 73 F ,G . ( 1 2 ) ( ) 7 − ( 7 7) The topology of this space is the same as the product topology defined in Example A.1.12.2. Í 6. Specializing the above example, if Σ is the discrete space defined in Example 1 above then we obtain a metrization of the Cantor space of Example A.1.45. For every finite sequence F Σ and every infinite sequence b F the ball ∈ ∗ ⊒ b, 2 : F is equal to a basis element that is the open-closed cylinder set ( − ( )) FΣN. 7. Let - be a metric space, and let . = - N be the set of infinite bounded sequences F = F ,F ,... with distance function 3 F,G = sup 3 F ,G . ( 1 2 ) ( ) 7 ( 7 7) 8. Let - be a topological space, and let - be the set of bounded continuous ( ) functions over - with distance function 3 5 , 6 = sup 3 5 F , 6 F . A ′( ) F ( ( ) ( )) special case is 0, 1 where the interval 0, 1 of real numbers has the usual [ ] [ ] topology.

9. Let :2 be the set of inﬁnite sequences F = F1,F2,... of real numbers with 2 ( ) = the property that 7 F7 < . The metric is given by the distance 3 F,G 2 1 2 ∞ ( ) 7 F7 G7 / . ( | − | ) Í y Í

137 A. Background from mathematics

In metric spaces, certain previously deﬁned topological objects have richer properties. Examples A.1.29 Each of the following facts holds in metric spaces and is relatively easy to prove.

1. Every closed set is a X set (and every open set is an f set). 2. A set is compact if and only if it is sequentially compact. 3. A set is compact if and only if it is closed and for every Y, there is a ﬁnite set of Y-balls (balls of radius Y) covering it. y

In metric spaces, the notion of continuity can be strengthened. Definition A.1.30 A function 5 : - . between metric spaces -, . is uni- → formly continuous if for each Y > 0 there is a X > 0 such that 3 0, 1 < X - ( ) implies 3 5 0 , 5 1 < Y. y . ( ( ) ( )) It is known that over a compact metric space, every continuous function is uniformly continuous. Definition A.1.31 (Lipschitz) Given a function 5 : - . between met- → ric spaces and V > 0, let Lip -, . denote the set of functions (called the V ( ) Lipschitz V functions, or simply Lipschitz functions) satisfying ( ) 3 5 F , 5 G V3 F,G . (A.1.2) . ( ( ) ( )) ≤ - ( ) Let Lip - = Lip -, R = Lip be the set of real Lipschitz functions over ( ) ( ) V V -. y Ð As an example, every differentiable real function 5 F with 5 F 1 ev- ( ) | ′( )| ≤ erywhere is a Lipschitz 1 function. ( ) All these functions are uniformly continuous. We introduce a certain fixed, enumerated sequence of Lipschitz functions that will be used frequently as “building blocks” of other functions. Definition A.1.32 (Hat functions) Let

6 F = 1 3 F, C @ + Y +. C,@,Y ( ) | − | ( )− | / | This is a continuous function that is 1 in the ball C, @ , it is 0 outside the ball ( ) C, @ Y , and takes intermediate values in between. It is clearly a Lipschitz 1 Y ( + ) ( / ) function.

138 A.1. Topology

If a dense set is ﬁxed, let F = F be the set of functions of the form 0 0 ( ) 6C,@,1 < where C , @ is rational, < = 1, 2,.... Let F1 = F1 be the maxi- / ∈ ( ) mum of a ﬁnite number of elements of F . Each element 5 of F is bounded 0 ( ) 1 between 0 and 1. Let

E = E = 6 , 6 ,... F (A.1.3) ( ) { 1 2 } ⊃ 1 be the smallest set of functions containing F0 and the constant 1, and closed under , and rational linear combinations. For each element of E, from its ∨ ∧ definition we can compute a bound V such that 5 Lip . y ∈ V For the effective representation of points in a topological space the following properties are important. Definition A.1.33 A topological space has the first countability property if each point has a countable basis of neighborhoods. y Every metric space has the first countability property since we can restrict ourselves to balls with rational radius. Definition A.1.34 Given a topological space -, g and a sequence F = ( ) F ,F ,... of elements of -, we say that F converges to a point G if for ev- ( 1 2 ) ery neighborhood of G there is a 9 such that for all ; > 9 we have F . ; ∈ We will write G = lim< F<. y →∞ It is easy to show that if spaces - , g 7 = 1, 2 have the first countabil- ( 7 7) ( ) ity property then a function 5 : - . is continuous if and only if for every → convergent sequence F we have 5 lim F = lim 5 F . ( <) ( < <) < ( <) Definition A.1.35 A topological space has the second countability property if the whole space has a countable basis. y For example, the space R has the second countability property for all three < > topologies gR, gR, gR. Indeed, we also get a basis if instead of taking all intervals, we only take intervals with rational endpoints. On the other hand, the metric space of Example A.1.28.7 does not have the second countability property. Definition A.1.36 In a topological space -, g , a set of points is called dense ( ) at a point F if it intersects every neighborhood of F. It is called everywhere dense, or dense, if it is dense at every point. A metric space is called separable if it has a countable everywhere dense subset. y It is easy to see that a metric space is separable if and only if as a topological space it has the second countability property.

139 A. Background from mathematics

Example A.1.37 In Example A.1.28.8, for - = 0, 1 , we can choose as our [ ] everywhere dense set the set of all polynomials with rational coefficients, or alternatively, the set of all piecewise linear functions whose graph has finitely many nodes at rational points. More generally, let - be a compact separable metric space with a dense set . Then it can be shown that in the metric space - , the set of functions ( ) E introduced in Definition A.1.32 is dense, and turns it into a complete (not ( ) necessarily compact!) separable metric space. y

Definition A.1.38 In a metric space, let us call a sequence F1,F2,... a Cauchy sequence if for all 7 < 8 we have 3 F ,F < 2 7. y ( 7 8) − It is easy to see that if an everywhere dense set is given then every element of the space can be represented as the limit of a Cauchy sequence of elements of . But not every Cauchy sequence needs to have a limit. Definition A.1.39 A metric space is called complete if every Cauchy sequence in it has a limit. y For example, if - is the real line with the point 0 removed then - is not complete, since there are Cauchy sequences converging to 0, but 0 is not in -. It is well-known that every metric space can be embedded (as an everywhere dense subspace) into a complete space. Example A.1.40 Consider the set 0, 1 of functions over 0, 1 that are right [ ] [ ] continuous and have left limits everywhere. The book [5] introduces two different metrics for them: the Skorohod metric 3 and the 30 metric. In both metrics, two functions are close if a slight monotonic continuous deformation of the coordinates makes them uniformly close. But in the 30 metric, the slope of the deformation must be close to 1. It is shown that the two metrics give rise to the same topology; however, the space with metric 3 is not complete, and the space with metric 30 is. y We will develop the theory of randomness over separable complete metric spaces. This is a wide class of spaces encompassing most spaces of practical interest. The theory would be simpler if we restricted it to compact or locally compact spaces; however, some important spaces like 0, 1 (the set of contin- [ ] uouos functions over the interval 0, 1 , with the maximum difference as their [ ] distance) are not locally compact.

140 A.2. Measures

A.2 Measures

For a survey of measure theory, see for example [40].

A.2.1 Set algebras Event in a probability space are members of a class of sets that is required to be a so-called f-algebra (sigma-algebra). Definition A.2.1 A (Boolean set-) algebra is a set of subsets of some set - closed under intersection and complement (and then, of course, under union). It is a f-algebra (sigma-algebra) if it is also closed under countable intersection (and then, of course, under countable union). A semialgebra is a set L of subsets of some set - closed under intersection, with the property that the complement of every element of L is the disjoint union of a finite number of elements of L. y If L is a semialgebra then the set of finite unions of elements of L is an algebra. Examples A.2.2

1. The set L1 of left-closed intervals of the line (including intervals of the form ,0 ) is a semialgebra. (−∞ ) 2. The set L2 of all intervals of the line (which can be open, closed, left-closed or right-closed), is a semialgebra. 3. In the set 0, 1 N of inﬁnite 0-1-sequences, the set L of all subsets of the { } 3 form C 0, 1 N with C 0, 1 , is a semialgebra. { } ∈ { }∗ 4. The f-algebra B generated by L1, is the same as the one generated by L2, and is also the same as the one generated by the set of all open sets: it is called the family of Borel sets of the line. The Borel sets of the extended real line R are deﬁned similarly. 5. More generally, the class of Borel sets in an arbitrary topological space is the smallest f-algebra containing all open sets. 6. Given f-algebras A, B in sets -, ., the product f-algebra A B in the space × - . is the one generated by all elements . and - for A and × × × ∈ B. ∈ y

A.2.2 Measures Probability is an example of the more general notion of a measure.

141 A. Background from mathematics

Definition A.2.3 A measurable space is a pair -, S where S is a f-algebra of ( ) sets of -.A measure on a measurable space -, S is a function ` : R that ( ) → + is f-additive: this means that for every countable family 1, 2,... of disjoint elements of S we have ` = ` . A measure ` is f-finite if the whole ( 7 7) 7 ( 7) space is the union of a countable set of subsets whose measure is finite. It is finite if ` - < . ItisaÐprobabilityÍ measure if ` - = 1. y ( ) ∞ ( ) Example A.2.4 (Delta function) For any point F, the measure XF is defined as follows:

1 if F , XF = ∈ ( ) (0 otherwise. y Definition A.2.5 If ` is a measure, a point F is called an atom if ` F > 0. y ( ) Generally, we will consider measures over either a countable set (a discrete measure for which the union of atoms has total measure) or an uncountable one, with no atoms. But mixed cases are possible. It is important to understand how a measure can be defined in practice. Alge- bras are generally simpler to grasp constructively than f-algebras; semialgebras are yet simpler. Suppose that ` is defined over a semialgebra L and is additive. Then it can always be uniquely extended to an additive function over the algebra generated by L. The following is an important theorem of measure theory. Proposition A.2.6 (Caratheodory’s extension theorem) Suppose that a nonnegative set function defined over a semialgebra L is f-additive. Then it can be extended uniquely to the f-algebra generated by L. Examples A.2.7 1. Let F be point and let ` = 1 if F and 0 otherwise. In this case, we ( ) ∈ say that ` is concentrated on the point F.

2. Consider the the line R, and the algebra L1 defined in Example A.2.2.1. Let 5 : R R be a monotonic real function. We define a set function over L → 1 as follows. Let 0 , 1 , (7 = 1,...,<) be a set of disjoint left-closed intervals. [ 7 7) Then ` 0 , 1 = 5 1 5 0 . It is easy to see that ` is additive. It ( 7 [ 7 7)) 7 ( 7)− ( 7) is f-additive if and only if 5 is left-continuous. Ð Í 3. Let = 0, 1 , and consider the set N of infinite 0-1-sequences, and the { } semialgebra L3 of Example A.2.2.3. Let ` : ∗ R be a function. Let us → + write ` CN = ` C for all C . Then it can be shown that the following ( ) ( ) ∈ ∗

142 A.2. Measures

conditions are equivalent: ` is f-additive over L3; it is additive over L3; the equation ` C = ` C0 ` C1 holds for all C . ( ) ( )+ ( ) ∈ ∗ 4. The nonnegative linear combination of any finite number of measures is also a measure. In this way, it is easy to construct arbitrary measures concentrated on a finite number of points. 5. Given two measure spaces -, A,` and ., B, a it is possible to define the ( ) ( ) product measure ` a over the measureable space - ., A B . The × ( × × ) definition is required to satisfy ` a = ` a , and is determined × ( × ) ( )× ( ) uniquely by this condition. If a is a probability measure then, of course, ` = ` a . . ( ) × ( × ) y

Let us finally define measureable functions. Definition A.2.8 (Measureable functions) Let -, A , ., B be two measure- ( ) ( ) able spaces. A function 5 : - . is called measureable if and only if 5 1 A → − ( ) ∈ for all B. y ∈ The following is easy to check. Proposition A.2.9 Let -, A be a measureable space and R, B be the measure- ( ) ( ) able space of the real numbers, with the Borel sets. Then 5 : - R is measureable → if and only if all sets of the form 5 1 @, = F : 5 F > @ are measureable, − (( ∞)) { ( ) } where @ is a rational number. Remark A.2.10 Example A.2.7.3 shows a particularly attractive way to define measures. Keep splitting the values ` C in an arbitrary way into ` C0 and ( ) ( ) ` C1 , and the resulting values on the semialgebra define a measure. Exam- ( ) ple A.2.7.2 is less attractive, since in the process of defining ` on all intervals and only keeping track of finite additivity, we may end up with a monotonic function that is not left continuous, and thus with a measure that is not f-additive. In the subsection on probability measures over a metric space, we will find that even on the real line, there is a way to define measures in a step-by-step manner, and only checking for consistency along the way. y A probability space, according to the axioms introduced by Kolmogorov, is just a measureable space with a normed measure. Definition A.2.11 A probability space is a triple -, S,% where -, S is a mea- ( ) ( ) surable space and % is a probability measure over it. Let - , S (7 = 1, 2) be measurable spaces, and let 5 : - . be a mapping. ( 7 7) → Then 5 is measurable if and only if for each element of S2, its inverse image

143 A. Background from mathematics

5 1 is in S . If ` is a measure over - , S then ` defined by ` = − ( ) 1 1 ( 1 1) 2 2 ( ) ` 5 1 is a measure over - called the measure induced by 5 . y 1 ( − ( )) 2 A.2.3 Integral The notion of integral also generalizes to arbitrary measures, and is sometimes also used to define measures. First we define integral on very simple functions. Definition A.2.12 A measurable function 5 : - R is called a step function if → its range is finite. The set of step functions is closed with respect to linear combinations and also with respect to the operations , . Any such set of functions is called a ∧ ∨ Riesz space. y

Definition A.2.13 Given a step function 5 which takes values F7 on sets 7, and a finite measure `, we define

` 5 = ` 5 = 5 3` = 5 F ` 3F = F ` . ( ) ( ) ( ) 7 ( 7) ¹ ¹ Õ7 y

This is a linear positive functional on the set of step functions. Moreover, it can be shown that it is continuous on monotonic sequences: if 5 0 then ` 5 7 ց 7 ց 0. The converse can also be shown: Let ` be a linear positive functional on step functions that is continuous on monotonic sequences. Then the set function ` = ` 1 is a finite measure. ( ) ( ) Proposition A.2.14 Let E be any Riesz space of functions with the property that 1 E. Let ` be a positive linear functional on E continuous on monotonic sequences, ∈ with `1 = 1. The functional ` can be extended to the set E of monotonic limits + of nonnegative elements of E, by continuity. In case when E is the set of all step functions, the set E is the set of all nonnegative measurable functions. + Now we extend the notion of integral to a wider class of functions. Definition A.2.15 Let us fix a finite measure ` over a measurable space -, S . ( ) A measurable function 5 is called integrable with respect to ` if ` 5 < and | |+ ∞ ` 5 < . In this case, we define ` 5 = ` 5 ` 5 . y | |− ∞ | |+ − | |− The set of integrable functions is a Riesz space, and the positive linear functional ` on it is continuous with respect to monotonic sequences. The continuity over monotonic sequences also implies the following theorem.

144 A.2. Measures

Proposition A.2.16 (Bounded convergence theorem) Suppose that functions 5< are integrable and 5 < 6 for some integrable function 6. Then 5 = lim 5 is | < | < < integrable and ` 5 = lim< ` 5<. Definition A.2.17 Two measurable functions 5 , 6 are called equivalent with respect to measure ` if ` 5 6 = 0. y ( − ) For two-dimensional integration, the following theorem holds. Proposition A.2.18 (Fubini theorem) Suppose that function 5 , is integrable (· ·) over the space - ., A B,` a . Then for `-almost all F, the function 5 F, ( × × × ) ( ·) is integrable over ., B, a , and the function F aG 5 F,G is integrable over ( ) 7→ ( ) -, A,` with ` a 5 = `F ` G 5 . ( ) ( × ) To express a continuity property of measures, we can say the following (recall the definition of - in Example A.1.28.8). ( ) Proposition A.2.19 Let - be a metric space and ` a measure. Then ` is a bounded (and thus continuous) linear functional over the space - . ( ) A.2.4 Density When does one measure have a density function with respect to another? Definition A.2.20 Let `, a be two measures over the same measurable space. We say that a is absolutely continuous with respect to `, or that ` dominates a, if for each set , ` = 0 implies a = 0. y ( ) ( ) Every nonnegative integrable function 5 defines a new measure a via the formula a = ` 5 1 . This measure a is absolutely continuous with respect ( ) ( · ) to `. The Radon-Nikodym theorem says that the converse is also true. Proposition A.2.21 (Radon-Nikodym theorem) If a is dominated by ` then there is a nonnegative integrable function 5 such that a = ` 5 1 for all ( ) ( · ) measurable sets . The function 5 is defined uniquely to within equivalence with respect to `. Definition A.2.22 The function 5 of the Radom-Nikodym Theorem above is called the density of a with respect to `. We will denote it by

` 3F 3` 5 F = ( ) = . ( ) a 3F 3a ( ) y

The following theorem is also standard.

145 A. Background from mathematics

Proposition A.2.23 (Chain rule and inverse function) 1. Let `, a, [ be measures such that [ is absolutely continuous with respect to ` and ` is absolutely continuous with respect to a. Then the “chain rule” holds:

3[ 3[ 3` = . (A.2.1) 3a 3` 3a

a 3F 2. If `( 3F) > 0 for all F then ` is also absolutely continuous with respect to a and ( ) 1 ` 3F = a 3F − a (3F) `( 3F) . ( ) ( ) There is a natural distance to be used between measures, though later we will see that it is not the preferred one in metric spaces. Definition A.2.24 (Total variation distance) Let `, a be two measures, then both are dominated by some measure [ (for example by [ = ` a). Let their + densities with respect to [ be 5 and 6. Then we define the total variation distance of the two measures as `, a = [ 5 6 . ( ) (| − |) It is independent of the dominating measure [. y Example A.2.25 Suppose that the space - can be partitioned into disjoint sets , such that a = ` = 0. Then `, a = ` a = ` - a - . y ( ) ( ) ( ) ( )+ ( ) ( )+ ( ) A.2.5 Random transitions What is just a transition matrix in case of a Markov chain also needs to be defined more carefully in the non-discrete cases. We follow the definition given in [40]. Definition A.2.26 Let -, A , ., B be measureable spaces (defined in Sub- ( ) ( ) section A.2.2). Suppose that a family of probability measures Λ = _ : F - { F ∈ } on B is given. We call it a probability kernel, (or Markov kernel, or conditional distribution) if the map F _ is measurable for each B. y 7→ F ∈ When -, . are finite sets then _ is a Markov transition matrix. The following theorem shows that _ assigns a joint distribution over the space - ., A B ( × × ) to each input distribution `. Proposition A.2.27 For each nonnegative A B-measureable function 5 over - ., × × 1. The function G 5 F,G is B-measurable for each fixed F. 7→ ( ) 2. The function F _ G 5 F,G is A-measurable. 7→ F ( ) 3. The integral 5 `F _ G 5 F,G defines a measure on A B. 7→ F ( ) × 146 A.2. Measures

The above proposition allows to define a mapping over measures. Definition A.2.28 According to the above proposition, given a probability kernel Λ, to each measure ` over A corresponds a measure over A B. We will × denote its marginal over B as Λ∗`. (A.2.2) For every measurable function 6 G over ., we can define the measurable funct- G ( ) ion 5 F = _ 6 = _ 6 G : we write ( ) F F ( ) 5 = Λ6. (A.2.3) y

The operator Λ is linear, and monotone with Λ1 = 1. By these deﬁnitions, we have ` Λ6 = Λ∗` 6. (A.2.4) ( ) ( ) An example is the simple case of a deterministic mapping: Example A.2.29 Let ℎ : - . be a measureable function, and let _ be the → F measure Xℎ F concentrated on the point ℎ F . This operator, denoted Λℎ is, in ( ) ( ) fact, a deterministic transition, and we have Λ 6 = 6 ℎ. In this case, we will ℎ ◦ simplify the notation as follows: = Λ ℎ∗` ℎ∗. y

A.2.6 Probability measures over a metric space We follow the exposition of [5]. Whenever we deal with probability measures on a metric space, we will assume that our metric space is complete and separable (Polish space). Let X = -,3 be a complete separable metric space. Then X gives rise to a ( ) measurable space, where the measurable sets are its Borel sets. It can be shown that, if is a Borel set and ` is a ﬁnite measure then there are sets ⊆ ⊆ where is an set, is a set, and ` = ` . f X ( ) ( ) It can be shown that a measure is determined by its values on the elements of any a basis of open sets closed under intersections. The following proposition follows then essentially from Proposition A.2.6.

Proposition A.2.30 Let B be a basis of open sets closed under intersections. Let B∗ be the set algebra generated by this basis and let ` be any f-additive set function on B with ` - = 1. Then ` can be extended uniquely to a probability measure. ∗ ( ) 147 A. Background from mathematics

Weak topology One can introduce the notion of convergence of measures in a number of ways. We have already introduced the total variation distance in Deﬁnition A.2.24 above. But in some cases, the requirement of being close in this distance is too strong. Let M X ( ) be the set of probability measures on the metric space X. Let F< be a sequence of points converging to point F but with F< ≠ F. We would like to say that the delta measure XF< (concentrated on point F<, see Example A.2.4) converges to X . But the total variation distance X , X is 2 for all <. F ( F< F ) Deﬁnition A.2.31 (Weak convergence) We say that a sequence of probability measures ` over a metric space -,3 weakly converges to measure ` if for all < ( ) bounded continuous real functions 5 over - we have ` 5 ` 5 . < → For a bounded continuous function 5 and real numbers 2 let

= ` : `5 <2 5,2 { } y A topology of weak convergence M, g can be defined using a number of dif- ( |) ferent subbases. The one used in the original definition is the subbasis consisting of all sets of the form 5,2 above. We also get a subbasis (see for example [40]) if we restrict ourselves to the set Lip - of Lipschitz functions defined in (A.1.2). Another possible subbasis ( ) giving rise to the same topology consists of all sets of the form

= ` : ` > 2 (A.2.5) ,2 { ( ) } for open sets and real numbers 2. Let us find some countable subbases. Since the space X is separable, there is a sequence *1, *2,... of open sets that forms a basis of X. Then we can restrict the subbasis of the space of measures to those sets ,2 where is the union of a finite number of basis elements *7 of X and 2 is rational. This way, the space M, g itself has the second countability property. ( |) It is more convenient to define a countable subbasis using bounded continuous functions 5 , since the function ` ` 5 is continuous on such functions, 7→ while ` `* is typically not continuous when * is an open set. 7→ Example A.2.32 If X = R and * is the open interval 0, 1 , the sequence of ( ) probability measures X1 < (concentrated on 1 <) converges to X0, but X1 < * = / / / ( ) 1, and X * = 0. y 0 ( ) 148 A.2. Measures

For some fixed dense set , let F = F be the set of functions introduced 1 1 ( ) in Definition A.1.32. Definition A.2.33 We say that a set is a continuity set of measure ` if ` m = ( ) 0: the boundary of has measure 0. y Proposition A.2.34 The following conditions are equivalent:

1. `< weakly converges to `. 2. ` 5 ` 5 for all 5 F . < → ∈ 1 3. For every Borel set , that is a continuity set of `, we have ` ` . < ( ) → ( ) 4. For every closed set , lim inf ` ` . < < ( ) ≥ ( ) 5. For every open set , lim sup ` ` . < < ( ) ≤ ( ) Deﬁnition A.2.35 To deﬁne the topological space M - of the set of measures ( ) over the metric space -, we choose as subbasis

fM (A.2.6) the sets ` : `5 <@ and ` : `5 >@ for all 5 F and @ Q. y { } { } ∈ 1 ∈ The simple functions we introduced can also be used to define measure and integral in themselves. Recall the definition of the set E in (A.1.3).T This set is a Riesz space as defined in Subsection A.2.3. A reasoning combining Proposi- tions A.2.6 and A.2.14 gives the following. Proposition A.2.36 Suppose that a positive linear functional ` with `1 = 1 is defined on E that is continuous with respect to monotone convergence. Then ` can be extended uniquely to a probability measure over X with ` 5 = 5 F ` 3F for ( ) ( ) all 5 E. ∈ ∫ Having a topology over the set of measures we can also extend Proposi- tion A.2.19: Proposition A.2.37 Let - be a complete separable metric space and M - the ( ) space of bounded measures over - with the weak topology. The function `, 5 ( ) 7→ ` 5 is a continuous function M - - R. ( )× ( ) → As mentioned above, for an open set the value ` is not a continuous ( ) function of the measure `. We can only say the following: Proposition A.2.38 Let - be a complete separable metric space, and M - the ( ) space of bounded measures over - with the weak topology, and - an open set. ⊆ The function ` ` is lower semicontinuous. 7→ ( )

149 A. Background from mathematics

Distances for the weak topology The definition of measures in the style of Proposition A.2.36 is not sufficiently constructive. Consider a gradual definition of the measure `, extending it to more and more elements of E, while keeping the positivity and linearity property. It can happen that the function ` we end up with in the limit, is not continuous with respect to monotone convergence. Let us therefore metrize the space of measures: then an arbitrary measure can be defined as the limit of a Cauchy sequence of simple meaures. One metric that generates the topology of weak convergence is the following. Definition A.2.39 (Prokhorov distance) The Prokhorov distance d `, a of two ( ) measures is the infimum of all those Y for which, for all Borel sets we have (using the notation (A.1.1))

` a Y Y. ( ) ≤ ( )+ y

It can be shown that this is a metric and it generates the weak topology. In computer science, it has been reinvented by the name of “earth mover distance”. The following important result helps visualize it: Proposition A.2.40 (Coupling Theorem, see [50]) Let `, a be two probability measures over a complete separable metric space X with d `, a Y. Then there is ( ) ≤ a probability measure P on the space X X with marginals ` and a such that for × a pair of random variables b, [ having joint distribution P we have

P 3 b, [ > Y Y. { ( ) } ≤ Since weak topology has the second countability property, the metric space deﬁned by the distance d , is separable. This can also be seen directly: let us (· ·) deﬁne a dense set in weak topology.

Definition A.2.41 For each point F, let us define by XF the measure which con- centrates its total weight 1 in point F. Let be a countable everywhere dense set of points in -. Let M be the set of finite convex rational combinations of measures of the form X where F , that is those probability measures that are F7 7 ∈ concentrated on finitely many points of and assign rational values to them. y

It can be shown that M is everywhere dense in the metric space M - ,d ; ( ( ) ) so, this space is separable. It can also be shown that M - ,d is complete. ( ( ) ) Thus, a measure can be given as the limit of a Cauchy sequence of elements `1,`2,... of M.

150 A.2. Measures

The definition of the Prokhorov distance uses quantification over all Borel sets. However, in an important simple case, it can be handled efficiently. Proposition A.2.42 Assume that measure a is concentrated on a finite set of points ( -. Then the condition d a, ` < Y is equivalent to the finite set of conditions ⊂ ( ) ` Y > a Y (A.2.7) ( ) ( )− for all (. ⊂ Another useful distance for measures over a bounded space is the Wasserstein distance. Definition A.2.43 Over a bounded metric space, we define the Wasserstein distance by

, `, a = sup ` 5 a 5 . ( ) 5 Lip - | − | ∈ 1 ( ) y

The Wasserstein distance also has a coupling representation: Proposition A.2.44 (See [51]) Let `, a be two probability measures over a complete separable metric space X with d `, a Y. Then there is a probability mea- ( ) ≤ sure P on the space X X with marginals ` and a such that ×

3 F,G P 3F3G = , `, a . ( ) ( ) ( ) ¹ Proposition A.2.45 The Prokhorov and Wasserstein metrics are equivalent: the identity function creates a uniformly continuous homeomorphism between the two metric spaces.

Proof. Let " = supF,G - 3 F,G . The proof actually shows for Y < 1: ∈ ( ) d `, a Y , `, a " 1 Y, ( ) ≤ ⇒ ( ) ≤ ( + ) , `, a Y2 d `, a Y. ( ) ≤ ⇒ ( ) ≤ Suppose d `, a Y < 1. By the Coupling Theorem (Proposition A.2.40), there ( ) ≤ are random variables b, [ over - with a joint distribution, and having respectively the distribution ` and a, such that P 3 b, [ > Y Y. Then we have { ( ) } ≤ ` 5 a 5 = E 5 b E 5 [ E 5 b 5 [ | − | | ( )− ( )| ≤ | ( )− ( )| Y P d b, [ Y " P d b, [ > Y " 1 Y. ≤ { ( ) ≤ }+ { ( ) } ≤ ( + )

151 A. Background from mathematics

Now suppose , `, a Y2 < 1. For a Borel set deﬁne 6 F = 1 ( ) ≤ Y ( ) | − d F, Y . Then we have ` ` 6 and a6 a Y . Further Y6 Lip , ( )/ |+ ( ) ≤ ( Y ) Y ≤ ( ) Y ∈ 1 and hence , `, a Y2 implies Y` 6 Ya Y6 Y2. This concludes the ( ) ≤ ( Y ) ≤ ( Y ) + proof by

` ` 6 a Y6 Y a Y Y. ( ) ≤ ( Y ) ≤ ( Y )+ ≤ ( )+

Relative compactness Convergence over the set of measures, even over a noncompact space, is sometimes obtained via compactness properties. Deﬁnition A.2.46 A set Π of measures in M - ,d is called sequentially com- ( ( ) ) pact if every sequence of elements of Π contains a convergent subsequence. A set of Π of measures is called tight if for every Y there is a compact set such that ` > 1 Y for all ` in Π. y ( ) − The following important theorem is using our assumptions of separability and completeness of the underlying metric space -,d . ( ) Proposition A.2.47 (Prokhorov) A set of measures is sequentially compact if and only if it is tight and if and only if its closure is compact in the metric space M - ,d . ( ( ) ) The following, simplest example is interesting in its own right. Example A.2.48 The one-element set ` is compact and therefore by Prokhorov’s { } theorem tight. Here, tightness says that for each Y a mass of size 1 Y of ` is − concentrated on some compact set. y The following theorem strengthens the observation of this example. Proposition A.2.49 A ﬁnite measure ` over a separable complete metric space has the property

` = sup ` : compact ( ) { ( ) ⊆ } for all Borel sets . In case of compact metric spaces, the following known theorem helps. Proposition A.2.50 The metric space M X ,d of measures is compact if and ( ( ) ) only if the underlying metric space -,3 is compact. ( )

152 A.2. Measures

So, if -,3 is not compact then the set of measures is not compact. But ( ) still, by Proposition A.2.49, each ﬁnite measure is “almost” concentrated on a compact set.

Semimeasures Let us generalize then notion of semimeasure for the case of general Polish spaces. We use Proposition A.2.36 as a starting point. Definition A.2.51 A function 5 ` 5 defined over the set of all bounded con- 7→ tinuous functions is called a semimeasure if it has the following properties: a) Nonnegative: 5 0 implies ` 5 0. ≥ ≥ b) Positive homogenous: we have ` 0 5 = 0` 5 for all nonnegative real 0 > 0. ( ) c) Superadditive: ` 5 6 ` 5 `6. ( + ) ≥ + d) Normed: `1 = 1. y Remark A.2.52 The weaker requirement `1 1 suffices, but if we have `1 1 ≤ ≤ we can always set simply `1 = 1 without violating the other requirements. y

Functionals of measures For a deeper study of randomness tests, we will need to characterize certain functionals of ﬁnite measures over a Polish space. Proposition A.2.53 Let X = -,3 be a complete separable metric space with ( ) M˜ X the set of ﬁnite measures over it. A weakly continuous linear function : ( ) M˜ X R can always be written as ` = ` 5 for some bounded continuous ( ) → ( ) function 5 over X.

Proof. Deﬁne 5 F = X . The weak continuity of implies that 5 F is con- ( ) ( F ) ( ) tinuous. Let us show that it is bounded. Suppose it is not. Then there is a sequence of distinct points F -, < = 1, 2,... with 5 F > 2 <. Let ` be the < ∈ ( <) − measure 2 1 = , < − F< ( ) < − ( <) < ∞ which is not allowed. The functionÍ ` ` 5 is continuous and coincidesÍ with ` onÍ the dense 7→ ( ) set of points M, so they are equal.

153

B Constructivity

B.1 Computable topology

B.1.1 Representations There are several equivalent ways that notions of computability can be extended to spaces like real numbers, metric spaces, measures, and so on. We use the the concepts of numbering (notation) and representation, as defined in [57]. Notation B.1.1 We will denote by N the set of natural numbers and by B the set 0, 1 . { } Given a set (an alphabet) Σ we denote by ΣN = ΣN the set of infinite sequences with elements in Σ. If for some finite or infinite sequences F, G,H,|, we have H = |FG then we write | H (| is a prefix of H) and F ⊳ H. After [57], let us define the wrapping ⊑ function ] : Σ Σ by ∗ → ∗ ] 0 0 0 = 1100 00 0 0 011. (B.1.1) ( 1 2 · · · <) 1 2 · · · < Note that : ] F = 2: F 5 6. (B.1.2) ( ( )) ( ( )+ )∨ For strings F,F Σ , >, > ΣN, 9 1, appropriate tupling functions F ,...,F , 7 ∈ ∗ 7 ∈ ≥ h 1 9i F, > , >, F , and so on can be defined with the help of , and ] . y h i h i h· ·i (·) Definition B.1.2 Given a countable set , a numbering (or notation) of is a surjective partial mapping X : N . Given some finite alphabet Σ 0, 1 → ⊇ { } and an arbitrary set (, a representation of ( is a surjective partial mapping j : ΣN (. A naming system is a notation or a representation. y → Here are some standard naming systems: N 1. id, the identity over Σ∗ or Σ .

2. aN, aZ, aQ for the set of natural numbers, integers and rational numbers.

155 B. Constructivity

3. Cf : ΣN 2N, the characteristic function representation of sets of natural → numbers, is defined by Cf > = 7 : > 7 = 1 . ( ) { ( ) } 4. En : ΣN 2N, the enumeration representation of sets of natural numbers, is → < 1 defined by En > = < N : 110 + 11 ⊳ > . ( ) N{ ∈ Δ } 5. For Δ Σ, EnΔ : Σ 2 ∗ , the enumeration representation of subsets of Δ , ⊆ → ∗ is defined by EnΔ > = | Σ : ] | ⊳ > . ( ) { ∈ ∗ ( ) } Using Turing machines with infinite input tapes, work tapes and output tapes, one can define names for all computable functions between spaces that N are Cartesian products of terms of the kind Σ∗ and Σ . (One wonders whether Σ ΣN is not needed, since a Turing machine with an infinite input tape may ∗ ∪ still produce only a finite output. But in this case one can also encode the result into an infinite sequence.) Then, the notion of computability can be transferred to other spaces as follows. Definition B.1.3 Let X : . - , 7 = 1, 0 be naming systems of the spaces - . 7 7 → 7 7 Let 5 : - - , 6 : . . . We say that function 6 realizes function 5 if ⊆ 1 → 0 ⊆ 1 → 0 5 X G = X 6 G (B.1.3) ( 1 ( )) 0 ( ( )) holds for all G for which the left-hand side is defined. Realization of multi-argument functions is defined similarly. We say that a function 5 : - - - is X , X , X -computable if there is a computable 1 × 2 → 0 ( 1 2 0) function 6 : . . . realizing it. In this case, a name for 5 is naturally ⊆ 1 × 2 → 0 derived from a name of 6.1 y Definition B.1.4 For representations b, [, we write b [ if there is a comput- ≤ able partial function 5 : ΣN ΣN with b F = [ 5 F . In words, we say that b → ( ) ( ( )) is reducible to [, or that 5 reduces (translates) b to [. There is a similar definition of reduction for notations. We write b [ if b [ and [ b. y ≡ ≤ ≤ B.1.2 Constructive topological space Let us start with the definition of topology with the help of a subbasis of (possibly empty) open sets. Definition B.1.5 A constructive topological space X = -,f,a is a topological ( ) space over a set - with a subbasis f effectively enumerated (not necessarily without repetitions) as a list f = a 1 , a 2 ,... , and having the ) property (thus, { ( ) ( ) } 0 every point is determined uniquely by the subset of elements of f containing it).

1Any function 6 realizing 5 via (B.1.3) automatically has a certain extensionality property: if X G = X G then 6 G = 6 G . 1 ( ) 1 ( ′) ( ) ( ′) 156 B.1. Computable topology

By definition, a constructive topological space satisfies the second countability axiom. We obtain a basis f∩ of the space X by taking all possible finite intersections of elements of f. It is easy to produce an effective enumeration for f∩ from a. We will denote this enumeration by a∩. This basis will be called the canonical basis. For every nonempty subset . of the space -, the subspace of - will naturally get the same structure Y = .,f,a defined by + . : + f . ( ) { ∩ ∈ } The product operation is defined over constructive topological spaces in the natural way. y Remark B.1.6 The definition of a subspace shows that a constructive topological space is not a “constructive object” by itself, since the set - itself is not necessarily given effectively. For example, any nonempty subset of the real line is a constructive topological space, as a subspace of the real line. y Examples B.1.7 The following examples will be used later. 1. A discrete topological space, where the underlying set is finite or countably infinite, with a fixed enumeration. 2. The real line, choosing the basis to be the open intervals with rational endpoints with their natural enumeration. Product spaces can be formed to give the Euclidean plane a constructive topology. > 3. The real line R, with the subbasis fR defined as the set of all open intervals , 1 with rational endpoints 1. The subbasis f<, defined similarly, leads (−∞ ) R to another topology. These two topologies differ from each other and from the usual one on the real line, and they are not Hausdorff spaces. 4. This is the constructive version of Example A.1.4.5. Let - be a set with a constructive discrete topology, and - N the set of infinite sequences with elements from -, with the product topology: a natural enumerated basis is also easy to define. y

Definition B.1.8 Due to the )0 property, every point in our space is determined uniquely by the set of open sets containing it. Thus, there is a representation WX of X defined as follows. We say that W > = F if EnΣ > = | : F a | . If X ( ) ( ) { ∈ ( )} W > = F then we say that the infinite sequence > is a complete name of F: it X ( ) encodes all names of all subbasis elements containing F. From now on, we will call WX the complete standard representation of the space X. y

157 B. Constructivity

Remark B.1.9 Here it becomes important that representations are allowed to be partial functions. y Deﬁnition B.1.10 (Constructive open sets) In a constructive topological space X = -,f,a , a set - is called constructive open, or lower semicomputable ( ) ⊆ open in set if there is a computably enumerable set with = | a∩ | . ∈ ( )∩ It is constructife open if it is constructive open in -. y Ð In the special kind of spaces in which randomness has been developed until now, constructive open sets have a nice characterization:

Proposition B.1.11 Assume that the space X = -,f,a has the form .1 .< N ( ) ×···× where each .7 is either Σ∗ or Σ . Then a set is constructive open iﬀ it is open and the set | ,...,| : a | is recursively enumerable. { ( 1 <) 7 ( 7) ⊂ } Proof. The proof is notÑ diﬃcult, but it relies on the discrete nature of the space N Σ∗ and on the fact that the space Σ is compact and its basis consists of sets that are open and closed at the same time. It is easy to see that if two sets are constructive open then so is their union. The above remark implies that a space having the form .1 .< where each .7 N ×···× is either Σ∗ or Σ , also the intersection of two recursively open sets is recursively open. We will see that this statement holds, more generally, in all computable metric spaces.

B.1.3 Computable functions Definition B.1.12 Let X = - , f , a be constructive topological spaces, and 7 ( 7 7 7) let 5 : - - be a function. As we know, 5 is continuous iff the inverse image 1 → 0 5 1 of each open set is open in its domain. Computability is an effective − ( ) version of continuity: it requires that the inverse image of basis elements is uniformly constructively open. More precisely, 5 : - - is computable if the 1 → 0 set 1 5 − + + ( )×{ } + f Ø∈ 0∩ is a constructive open subset of - f . Here the basis f of X is treated as a 1 × 0∩ 0∩ 0 discrete constructive topological space, with its natural enumeration. A partial function is computable if its restriction to the subspace that is its domain, is computable. y

158 B.1. Computable topology

The above definition depends on the enumerations a1, a0. The following theorem shows that this computability coincides with the one obtained by transfer via the representations WX7 . Proposition B.1.13 (Hertling) For 7 = 0, 1, let X = - , f , a be constructive 7 ( 7 7 7) topological spaces. Then a function 5 : - - is computable iff it is W , W - 1 → 0 ( X1 X0 ) computable for the representations WX7 defined above. The notion of computable functions helps define morphisms between constructive topological spaces.

Definition B.1.14 Let us call two spaces -1 and -0 effectively homeomorphic if there are computable maps between them that are inverses of each other. In the special case when -0 = -1, we say that the enumerations of subbases a0, a1 are equivalent if the identity mapping is a effective homeomorphism. This means that there are recursively enumerable sets , such that

a { = a∩ | for all {, a | = a∩ { for all |. 1 ( ) 0 ( ) 0 ( ) 1 ( ) {,| |,{ ( Ø) ∈ ( Ø) ∈ y

B.1.4 Computable elements and sequences Let us define computable elements. Definition B.1.15 Let U = 0 , f , a be the one-element space turned into a ({ } 0 0) trivial constructive topological space, and let X = -,f,a be another construc- ( ) tive topological space. We say that an element F - is computable if the function ∈ 0 F is computable. It is easy to see that this is equivalent to the requirement 7→ that the set C : F a C is recursively enumerable. Let X = - , f , a , { ∈ ( )} 8 ( 8 8 8) for 7 = 0, 1 be constructive topological spaces. A sequence 57, 7 = 1, 2,... of functions with 5 : - - is a computable sequence of computable functions if 7 1 → 0 7, F 5 F is a computable function. y ( ) 7→ 7 ( ) It is easy to see that this statement is equivalent to the statement that there is a recursive function computing from each 7 a name for the computable function 57. To do this formally, one sometimes using the s-m-n theorem of recursion theory, or calls the method “currification”. The notion of computability can be relativized.

Deﬁnition B.1.16 Let - 8, 8 = 1, 2 be two constructive topological spaces (or more general, representations) and let F - be given. We say that F is F - 8 ∈ 8 2 1

159 B. Constructivity computable if there is a computable partial function 5 : - - with 5 F = 1 → 2 ( 1) F2. y Remark B.1.17 The requirement that 5 computes G from every representation of F makes F-computability a very diﬀerent requirement of mere Turing reducibility of G to F. We will point out an important implication of this diﬀerence later, in the notion of `-randomness. It is therefore preferable not to use the term “oracle computation” when referring to computations on representations . y

B.1.5 Semicomputability Lower semicomputability is a constructive version of lower semicontinuity, as given in Definition A.1.14, but the sets that are required to be open there are required to be constructive open here. The analogue of Proposition A.1.17 and Corollary A.1.19 holds also: a lower semicomputable function is the supremum of simple constant functions defined on basis elements, and a lower semicomputable function defined on a subset can always be extended to one over the whole space. Definition B.1.18 Let X = -,f,a be a constructive topological space. A par- ( ) tial function 5 : - R with domain is called lower semicomputable if the set → + F,@ : 5 F > @ is a constructive open subset of R . { ( ) ( ) } × + We define the notion of F-lower semicomputable similarly to the notion of F-computability. y = < < Let Y R , fR, aR be the effective topological space introduced in Ex- ( + ) > ample B.1.7.2, in which aR is an enumeration of all open intervals of the form @, with rational @. The following characterization is analogous to Proposi- ( ∞] tion A.1.15. Proposition B.1.19 A function 5 : - R is lower semicomputable if and only if → it is a, a> -computable. ( R) As a name of a computable function, we can use the name of the enumeration algorithm derived from the definition of computability, or the name derivable using this representation theorem. The following example is analogous to Example A.1.16. Example B.1.20 The indicator function 1 F of an arbitrary constructive open ( ) set is lower semicomputable. y The following fact is analogous to Proposition A.1.17:

160 B.1. Computable topology

Proposition B.1.21 Let 51, 52,... be a computable sequence of lower semicomputable functions (via their names) over a constructive topological space X. Then sup7 57 is also lower semicomputable. The following fact is analogous to Proposition A.1.18: Proposition B.1.22 Let X = -,f,a be a constructive topological space with ( ) enumerated basis V = f∩ and 5 : - R a lower semicomputable function. Then → + there is a lower semicomputable function 6 : V R (where V is taken with the → + discrete topology) with 5 F = supF V 6 V . ( ) ∈ ( ) In the important special case of Cantor spaces, the basis is given by the set of ﬁnite sequences. In this case we can also require the function 6 | to be ( ) monotonic in the words |: Proposition B.1.23 Let - = ΣN be a Cantor space as deﬁned in Example B.1.7.4. Then 5 : - R is lower semicomputable if and only if there is a lower semicom- → + putable function 6 : Σ∗ R (where Σ∗ is taken as a discrete space) monotonic → + with respect to the relation C {, with 5 b = supC b 6 C . ⊑ ( ) ⊑ ( ) Remark B.1.24 In the above representation, we could require the function 6 to be computable. Indeed, we can just replace 6 | with 6 | where 6 | = ( ) ′( ) ′( ) max{ | 6 {,: | , and 6 {,< is as much of 6 { as can be computed in < steps. ⊑ ( ( )) ( ) ( ) y

Upper semicomputability is defined analogously: Definition B.1.25 We will call a closed set upper semicomputable (co-recursively enumerable) if its complement is a lower semicomputable (r.e.) open set. y The proof of the following statement is not difficult. Proposition B.1.26 Let X = - , f , a for 7 = 1, 2, 0 be constructive topological 7 ( 7 7 7) spaces, and let 5 : - - - , and assume that F - is a computable 1 × 2 → 0 1 ∈ 1 element. 1. If 5 is computable then F 5 F ,F is also computable. 2 7→ ( 1 2) 2. If X = R, and 5 is lower semicomputable then F 5 F ,F is also lower 0 2 7→ ( 1 2) semicomputable.

B.1.6 Eﬀective compactness Compactness is an important property of topological spaces. In the constructive case, however, a constructive version of compactness is frequently needed.

161 B. Constructivity

Deﬁnition B.1.27 A constructive topological space - with canonical basis V has recognizeable covers if the set

( V : ( is finite and * = - { ⊆ } * ( Ø∈ is recursively enumerable. A compact space with recognizeable covers will be called effectively compact. y Example B.1.28 Let U 0, 1 be a real number such that the set of rationals ∈ [ ] less than U is recursively enumerable but the set of rationals larger than U are F not. (It is known that there are such numbers, for example F N 2− ( ).) Let - be the subspace of the real line on the interval 0,U , with the induced∈ topology. [ ] A basis of this topology is the set V of all nonempty intervals ofÍ the form 0,U , ∩[ ] where is an open interval with rational endpoints. This space is compact, but not effectively so. y The following is immediate. Proposition B.1.29 In an effectively compact space, in every recursively enumerable set of basis elements covering the space one can effectively find a finite covering. It is known that every closed subset of a compact space is compact. This statement has a constructive counterpart: Proposition B.1.30 Every upper semicomputable closed subset of an effectively compact space - is also effectively compact.

Proof. *1,...,*9 be a finite set covering . Let us enumerate a sequence +7 of canonical basis elements whose union is the complement of . If * 1 ∪···∪ *9 covers then along with the sequence +7 it will cover -, and this will be recognized in a finite number of steps. On an effectively compact space, computable functions have computable ex- trema: Proposition B.1.31 Let - be an effectively compact space, let 5 F be lower semi- ( ) computable function on -. Then the infimum (which is always a minimum) of 5 F ( ) is lower semicomputable uniformly from the definition of - and 5 . Proof. For any rational number @ we must be able to recognize that the minimum is greater than @. For each @ the set F : 5 F > @ is a lower semicomputable { ( ) } open set. It is the union of an enumerated sequence of canonical basis elements. If it covers - this will be recognized in a finite number of steps.

162 B.1. Computable topology

It is known that a continuous function map compact sets into compact ones. This statement also has a constructive counterpart. Proposition B.1.32 Let - be an eﬀectively compact space, and 5 a computable function from - into another constructive topological space .. Then 5 - is eﬀec- ( ) tively compact.

Proof. Let V- and V. be the enumerated bases of the space - and . respectively. Since 5 is computable, there is a recursively enumerable set E V- V. such 1 ⊆ × that 5 − + = *,+ * holds for all + V. . Let E+ = * : *,+ E , then ( ) ( ) ∈ ∈ { ( ) ∈ } 5 1 + = E . − ( ) + Consider someÐ finite cover 5 - + + with some + V . We ( ) ⊆ 1 ∪···∪ 9 7 ∈ . will showÐ that it will be recognized. Let U = E , then - U, and the 7 +7 ⊆ whole set U V is enumerable. By compactness, there is a finite number of ⊆ - elements * ,...,* U with - * . By effectiveÐ compactness,Ð every one 1 < ∈ ⊆ 7 7 of these finite covers will be recognized. And any one such recognition will serve to recognize that + ,...,+ is a coverÐ of 5 - . 1 9 ( ) B.1.7 Computable metric space Following [7], we define a computable metric space as follows. Definition B.1.33 A constructive metric space is a tuple X = -,3,,U where ( ) -,3 is a metric space, with a countable dense subset and an enumeration U ( ) of . It is assumed that the real function 3 U { ,U | is computable. y ( ( ) ( )) As F runs through elements of and @ through positive rational numbers, we obtain the enumeration of a countable basis F,@ : F ,@ Q (of balls { ( ) ∈ ∈ } or radius @ and center F) of X, giving rise to a constructive topological space X˜.

Deﬁnition B.1.34 Let us call a sequence F1,F2,... a Cauchy sequence if for all 7 < 8 we have 3 F ,F 2 7. To connect to the type-2 theory of computabil- ( 7 8) ≤ − ity developed above, the Cauchy-representation XX of the space is deﬁned in a natural way. y

It can be shown that as a representation of X˜, it is equivalent to W˜ : X W˜ . X X ≡ X Examples B.1.35 1. Example A.1.37 is a constructive metric space, with either of the two (equivalent) choices for an enumerated dense set. 2. Consider the metric space of Example A.1.28.6: the Cantor space -,3 . Let ( ) A Σ be a distinguished element. For each ﬁnite sequence F Σ let us 0 ∈ ∈ ∗

163 B. Constructivity

deﬁne the inﬁnite sequence bF = FA0A0 .... The elements bF form a (naturally enumerated) dense set in the space -, turning it into a constructive metric space. y

Let us point out a property of metric spaces that we use frequently. Deﬁnition B.1.36 For balls = F ,@ , 8 = 1, 2 we will say < if 8 ( 8 8) 1 2 3 F ,F < @ @ . In words, we will say that is manifestly included in . ( 1 2) 2 − 1 1 2 This relation is useful since it is an easy way to see . y 1 ⊂ 2 The property can be generalized to some constructive topological spaces. Deﬁnition B.1.37 A constructive topological space -,V,a with basis V has the ( ) manifest inclusion property if there is a relation 1 < 1′ among its basis elements with the following properties. a) 1 < 1 implies 1 1 . ′ ⊆ ′ b) The set of pairs 1 < 1′ is recursively enumerable.

c) For every point F, and pair of basis elements 1, 1′ containing F there is a basis element 1′′ containing F with 1′′ < 1, 1′. We express this relation by saying that 1 is manifestly included in 1′. In such a space, a sequence 1 > 1 > ... with 1 = F is called a manifest represen- 1 2 7 7 { } tation of F. y Ñ Note that if the space has the manifest inclusion property then for every pair F 1 there is a manifest representation of F beginning with 1. ∈ A constructive metric space has the manifest inclusion property as a topological space, and Cauchy representations are manifest. Similarly to the definition of a computable sequence of computable functions in Subsection B.1.4, we can define the notion of a computable sequence of bounded computable functions, or the computable sequence 57 of computable Lipschitz functions: the bound and the Lipschitz constant of 57 are required to be computable from 7. The following statement shows, in an effective form, that a function is lower semicomputable if and only if it is the supremum of a computable sequence of computable functions. Proposition B.1.38 Let X be a computable metric space. There is a computable mapping that to each name of a nonnegative lower semicomputable function 5 assigns a name of a computable sequence of computable bounded Lipschitz functions 57 whose supremum is 5 .

164 B.1. Computable topology

Proof sketch. Show that 5 is the supremum of a computable sequence of functions 271 C ,@ where C7 and 27,@7 > 0 are rational. Clearly, each indicator ( 7 7) ∈ function 1 C ,@ is the supremum of a computable sequence of computable func- ( 7 7) tions 67,8. We have 5 = sup< 5< where 5< = max7 < 2767,<. It is easy to see that ≤ the bounds on the functions 5< are computable from < and that they all are in Lip for a V that is computable from <. V< < The following is also worth noting. Proposition B.1.39 In a computable metric space, the intersection of two constructive open sets is constructive open. Proof. Let V = F,@ : F ,@ Q be a basis of our space. For a pair F,@ { ( ) ∈ ∈ } ( ) with F , @ Q, let ∈ ∈ Γ F,@ = G, A : G , A Q, 3 F,G A<@ . ( ) { ( ) ∈ ∈ ( )+ } If * is a constructive open set, then there is a computably enumerable set ( * ⊂ Q with = . Let = Γ : , then we * F,@ (* F,@ (*′ F,@ F,@ (* × ( ) ∈ ( ) { ( ) ( ) ∈ } have * = F,@ ( F,@ . Now, it is easy to see ( ) ∈Ð*′ ( ) Ð Ð * + = F,@ . ∩ ( ) F,@ ( ( ( )Ø ∈ *′ ∩ +′

The following theorem is very useful. Theorem B.1.1 A computable metric space - is eﬀectively compact if and only if from each (rational) Y one can compute a ﬁnite set of Y-balls covering -.

Proof. Suppose first that the space is effectively compact. For each Y, let 1,2,... be a list of all canonical balls with radius Y. This sequence covers the space, so already some 1,...,< covers the space, and this will be recognized. Suppose now that for every rational Y one can find a finite set of Y-balls covering the space. Let S V be a finite set of basis elements (balls) covering ⊆ the space. For each element = C, @ S, let = C, @ Y be its Y- ( ) ∈ Y ( − ) interior, and S = : S . Then = Y > 0 , and - = S . Y { Y ∈ } Y Y>0 Y Compactness implies that there is an Y > 0 such that already S4>A covers the space. Let ,..., be a finite set of Y 2-ballsÐ = 2 ,@ coveringÐ the spaceÐ 1 < / 7 ( 7 7) that can be computed from Y. Each of these balls 7 intersects one of the the sets = C, @ Y , 3 C, 2 @ Y 2. But then C, @ in a recognizeable Y ( − ) ( 7) ≤ − / 7 ⊆ ( ) way. Once all the relations 3 C, 2 @ Y 2 will be recognized we will also ( 7) ≤ − / recognize that S covers the space.

165 B. Constructivity

We can strengthen now Example A.1.37: Example B.1.40 (Prove!) Let - be an effectively compact computable metric space. Then the metric space - with the dense set of functions E intro- ( ) ( ) duced in Definition A.1.32 is a computable metric space. y The structure of a constructive metric space will be inherited on certain subsets: Definition B.1.41 Let X = -, d,,U be a constructive metric space, and ( ) ⊂ - a constructive open subset. Then there is an enumeration U of the set ∩ that creates a constructive metric space G = , d,,U . y ( ) Remark B.1.42 An arbitrary subset of a constructive metric space will inherit the constructive topology. It will also inherit the metric, but not necessarily the structure of a constructive metric space. Indeed, first of all it is not necessarily a complete metric space. It also does not inherit an enumerated dense subset. y

B.2 Constructive measure theory

The basic concepts and results of measure theory are recalled in Section A.2. For the theory of measures over metric spaces, see Subsection A.2.6. We introduce a certain ﬁxed, enumerated sequence of Lipschitz functions that will be used frequently. Let E be the set of functions introduced in Deﬁnition A.1.32. The following construction will prove useful later. Proposition B.2.1 All bounded continuous functions can be obtained as the limit of an increasing sequence of functions from the enumerated countable set E of bounded computable Lipschitz functions introduced in (A.1.3). The proof is routine.

B.2.1 Space of measures Let X = -,3,,U be a computable metric space. In Subsection A.2.6, the space ( ) M X of measures over X is defined, along with a natural enumeration a = aM for ( ) a subbasis f = fM of the weak topology. This is a constructive topological space M which can be metrized by introducing, as in A.2.6, the Prokhorov distance d `, a . Recall that we defined as the set of those probability measures that ( ) M are concentrated on finitely many points of and assign rational values to them. Let UM be a natural enumeration of M. Then M, d, ,U (B.2.1) ( M M) 166 B.2. Constructive measure theory is a computable metric space whose constructive topology is equivalent to M. Let * = F,@ be one of the balls in X, where F , @ Q. The function ( ) ∈ X ∈ ` ` * is typically not computable, since as mentioned in (A.2.6), it is not 7→ ( ) even continuous. The situation is better with ` ` 5 . The following theorem 7→ is the computable strengthening of part of Proposition A.2.37: Proposition B.2.2 Let X = -,3,,U be a computable metric space, and let ( ) M = M X ,f,a be the constructive topological space of probability measures ( ( ) ) over X. If the function 5 : X R is bounded and computable then ` ` 5 is → 7→ computable.

Proof sketch. To prove the theorem for bounded Lipschitz functions, we can in- voke the Strassen coupling theorem A.2.40. The function 5 can be obtained as a limit of a computable monotone in- > creasing sequence of computable Lipschitz functions 5< , and also as a limit of a computable monotone decreasing sequence of computable Lipschitz functions < > 5< . In step < of our computation of ` 5 , we can approximate ` 5< from above to within 1 <, and ` 5 < from below to within 1 <. Let these bounds be 0> and 0<. / < / < < To approximate ` 5 to within Y, ﬁnd a stage < with 0> 0< 2 <

167 B. Constructivity

B.2.2 Computable and semicomputable measures A measure ` is called computable if it is a computable element of the space of measures. Let 6 be the set of bounded Lipschitz functions over X introduced { 7} in Definition A.1.32. Proposition B.2.7 Measure ` is computable if and only if so is the function 7 7→ `67. Proof. The “only if” part follows from Proposition B.2.2. For the “if” part, note that in order to trap ` within some Prokhorov neighborhood of size Y, it is sufficient to compute `6 within a small enough X, for all 7 < for a large enough 7 ≤ <. A non-computable density function can lead to a computable measure: Example B.2.8 Let our probability space be the set R of real numbers with its standard topology. Let 0 < 1 be two computable real numbers. Let ` be the 1 probability distribution with density function 5 F = 1 0,1 F (the uniform ( ) 1 0 [ ] ( ) distribution over the interval 0, 1 ). Function 5 F is− not computable, since it [ ] ( ) is not even continuous. However, the measure ` is computable: indeed, `67 = 1 1 6 F 3F is a computable sequence, hence Proposition B.2.7 implies that 1 0 0 7 ( ) `−is computable. y ∫ The following theorem compensates somewhat for the fact mentioned earlier, that the function ` ` * is generally not computable. 7→ ( ) Proposition B.2.9 Let ` be a finite computable measure. Then there is a computable map ℎ with the property that for every bounded computable function 5 with 5 1 with the property ` 5 1 0 = 0, if | is the name of 5 then ℎ | is the | | ≤ ( − ( )) ( ) name of a program computing the value ` F : 5 F < 0 . { ( ) } Proof. Straightforward. Can we construct a measure just using the pattern of Proposition B.2.7? Sup- pose that there is a computable function 7, 8 ; 8 with the following prop- ( ) 7→ 7 ( ) erties: a) 7 < 8 < 8 implies ; 8 ; 8 < 2 81. 1 2 | 7 ( 1)− 7 ( 2)| − b) For all <, there is a probability measure ` with ; < = ` 6 for all 7<<. < 7 ( ) < 7 Thus, the sequences converge, and for each <, all values ; < for 7 < are 7 ( ) ≤ consistent with coming from a probability measure a< assigning this value to 67. Is there a probability measure ` with the property that for each 7 we have lim ; 8 = `6 ? Not necessarily, if the space is not compact. 8 7 ( ) 7 168 B.2. Constructive measure theory

Example B.2.10 Let - = 1, 2, 3,... with the discrete topology. Define a { } probability measure `< with `<67 = 0 for 7<< and otherwise arbitrarily. Since we only posed < 1 linear conditions on a finite number of variables, it is easy − to see that such a ` exists. Then define ; < = ` 7 for all 7. < 7 ( ) < ( ) Now all the numbers ; < converge with < to 0, but ` = 0 is not a proba- 7 ( ) bility measure. y To guarantee that the sequences ; 8 indeed define a probability measure, 7 ( ) progress must be made, for example, in terms of the narrowing of Prokhorov neighborhoods.

B.2.3 Random transitions Consider random transitions now. Definition B.2.11 (Computable kernel) Let X, Y be computable metric spaces, giving rise to measurable spaces with f-algebras A, B respectively. Let Λ = _ : { F F - be a probability kernel from - to . (as defined in Subsection A.2.5). ∈ } Let 6 be the set of bounded Lipschitz functions over . introduced in Defini- { 7} tion A.1.32. To each 67, the kernel assigns a (bounded) measurable function

5 F = Λ6 F = _ G 6 G . 7 ( ) ( 7)( ) F 7 ( ) We will call the kernel Λ computable if so is the assignment 7, F 5 F . y ( ) 7→ 7 ( ) When Λ is computable, each function 5 F is of course continuous. The 7 ( ) measure Λ ` is determined by the values Λ 6 = ` Λ6 , which are computable ∗ ∗ 7 ( 7) from 7,` and so the mapping ` Λ ` is computable. ( ) 7→ ∗ The following example is the simplest case, when the transition is actually deterministic. Example B.2.12 (Computable deterministic kernel) A computable function ℎ : - . deﬁnes an operator Λ with Λ 6 = 6 ℎ (as in Example A.2.29). This → ℎ ℎ ◦ is a deterministic computable transition, in which 5 F = Λ 6 F = 6 ℎ F 7 ( ) ( ℎ 7)( ) 7 ( ( )) is, of course, computable from 7, F . We deﬁne ℎ ` = Λ `. y ( ) ∗ ℎ∗

169

Bibliography

[1] Yevgeniy A. Asarin. Individual Random Signals: a Complexity Approach. PhD thesis, Moscow State University, Moscow, Russia, 1988. In Russian. [2] Ya. M. Barzdin’. The complexity of programs to determine whether natural numbers not greater than < belong to a recursively enumerable set. Soviet Math. Doklady, 9:1251–54, 1968. [3] Ya. M. Barzdin’ and R. Freivald. On the prediction of general recursive functions. Soviet Math. Doklady, 206:1224–28, 1972. [4] Charles H. Bennett and Martin Gardner. The random number Omega bids fair to hold mysteries of the universe. Scientiﬁc American, 242(5):20–34, Nov 1979. [5] Patrick Billingsley. Convergence of Probability Measures. Wiley, 1968. First edition. Second edition 1999. [6] Manuel Blum and Silvio Micali. How to generate cryptographically strong sequences of pseudo-random bits. SIAM Journal on Computing, 13:850– 864, 1984. [7] Vasco Brattka and Gero Presser. Computability on subsets of metric spaces. Theoretical Computer Science, 305:43–76, 2003. [8] Gregory J. Chaitin. Information-theoretical limitations of formal systems. Journal of the ACM, 21:403–424, 1974. [9] Gregory J. Chaitin. Scientiﬁc American, 232(5):47–52, May 1975. [10] Gregory J. Chaitin. A theory of program size formally identical to information theory. J. Assoc. Comput. Mach., 22:329–340, 1975. [11] Gregory J. Chaitin. Algorithmic entropy of sets. Computers and Mathem. with Applications, 2:233–245, 1976.

171 Bibliography

[12] Gregory J. Chaitin. Algorithmic information theory. IBM J. Res.& Dev., 21:350–359, 1977. [13] Imre Csiszár and János Körner. Information Theory. Academic Press, New York, 1981. [14] Robert Daley. Math. Syst. Th., 9(1):83–94, 1975. [15] Pierre Simon de Laplace. A Philosophical Essay on Probabilities. Dover, 1951. [16] Terrence Fine. Theories of Probability. Academic Press, New York, 1973. [17] Peter Gács. On the symmetry of algorithmic information. Soviet Math. Dokl., 15:1477–1780, 1974. Translation of the Russian version. [18] Peter Gács. Exact expressions for some randomness tests. Z. Math. Log. Grdl. M., 26:385–394, 1980. There is an earlier, shorter conference version. [19] Peter Gács. On the relation between descriptional complexity and algorithmic probability. Theoretical Computer Science, 22:71–93, 1983. Short version: Proc. 22nd IEEE FOCS (1981) 296-303. [20] Peter Gács. Randomness and probability–complexity of description. In Samuel Kotz and Norman L. Johnson, editors, Encyclopedia of Statistical Sciences, Vol. 7, pages 551–556. Wiley, New York, 1986. [21] Peter Gács. Uniform test of algorithmic randomness over a general space. Theoretical Computer Science, 341(1-3):91–137, 2005. [22] Peter Gács and János Körner. Common information is far less than mutual information. Problems of Control and Inf. Th., 2:149–162, 1973. [23] Peter Gács, John Tromp, and Paul M. B. Vitányi. Algorithmic statistics. IEEE Transactions on Information Theory, 47:2443–2463, 2001. arXiv:math/0006233 [math.PR]. Short version with similar title in Algo- rithmic Learning Theory, LNCS 1968/2000. [24] Oded Goldreich, Shaﬁ Goldwasser, and Silvio Micali. How to construct random functions. Journal of the Association for Computing Machinery, 33(4):792–807, 1986. [25] Peter Hertling and Klaus Weihrauch. Randomness spaces. In Proc. of ICALP’98, volume 1443 of Lecture Notes in Computer Science, pages 796– 807. Springer, 1998.

172 Bibliography

[26] Mathieu Hoyrup and Cristóbal Rojas. Computability of probability measures and Martin-Löf randomness over metric spaces. Information and Computation, 207(7):830–847, 2009. [27] A.N. Kolmogorov. Three approaches to the quantitative deﬁnition of information. Problems of Inform. Transmission, 1(1):1–7, 1965. [28] Andrei N. Kolmogorov. Foundations of the Theory of Probability. Chelsea, New York, 1956. [29] Andrei N. Kolmogorov. A logical basis for information theory and probability theory. IEEE Transactions on Information Theory, IT-14:662–664, 1968. [30] Leonid A. Levin. On the notion of a random sequence. Soviet Math. Dokl., 14(5):1413–1416, 1973. [31] Leonid A. Levin. Laws of information conservation (nongrowth) and as- pects of the foundations of probability theory. Problems of Inform. Transm., 10(3):206–210, 1974. [32] Leonid A. Levin. Uniform tests of randomness. Soviet Math. Dokl., 17(2):337–340, 1976. [33] Leonid A. Levin. Randomness conservation inequalities: Information and independence in mathematical theories. Information and Control, 61(1):15–37, 1984. [34] Leonid A. Levin. One-way functions and pseudorandom generators. Com- binatorica, 4, 1987. [35] Leonid A. Levin and V.V. V’yugin. Invariant properties of information bulks. In Proceedings of the Math. Found. of Comp. Sci. Conf., volume 53 of Lecture Notes on Comp.Sci., pages 359–364. Springer, 1977. [36] Ming Li and Paul M. B. Vitányi. Introduction to Kolmogorov Complexity and its Applications (Third edition). Springer Verlag, New York, 2008. [37] Donald W. Loveland. A variant of the Kolmogorov concept of complexity. Information and Control, 12:510, 1969. [38] Per Martin-Löf. The deﬁnition of random sequences. Information and Con- trol, 9:602–619, 1966.

173 Bibliography

[39] Wofgang J. Paul, Joel Seiferas, and Janos Simon. An information- theoretical approach to time-bounds for on-line computation. J. Computer and System Sciences, 23:108–126, 1981. [40] David Pollard. A User’s Guide to Measure-Theoretic Probability. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, U.K., 2001. [41] Claus Peter Schnorr. Zufälligkeit und Wahrscheinlichkeit. In A. Dold and B. Eckmann, editors, Lecture Notes in Mathematics 218, pages 1–213. Springer Verlag, New York, 1971. [42] Claus Peter Schnorr. Process complexity and effective random tests. J. Comput. Syst. Sci, 7(4):376–388, 1973. Conference version: STOC 1972, pp. 168-176. [43] Claus Peter Schnorr. A review of the theory of random sequences. In Proc.5-th Int.Congr.of Logic, Meth. and Phil. of Sci., London, Ontario, 1975. [44] Claus Peter Schnorr. General random sequences and learnable sequences. J. Symb. Logic, 42:329–340, 1977. [45] Raymond J. Solomonoﬀ. A formal theory of inductive inference I. Informa- tion and Control, 7:1–22, 1964. [46] Raymond J. Solomonoﬀ. Complexity-based induction systems: Compar- isons and convergence theorems. IEEE Transactions on Information Theory, IT-24(4):422–432, July 1978. [47] Robert Solovay. Unpublished manuscript. 1976. [48] Robert Solovay. On random r.e. sets. In A. I. Arruda et al., editor, Non-Clas- sical Logic, Model Th. and Computability, pages 283–307. North Holland, 1977. [49] Tom Stoppard. Rosencrantz and Guildenstern are Dead. A play. [50] Volker Strassen. The existence of probability measures with given marginals. Annals of Mathematical Statistics, 36:423–439, 1965. [51] Cedric Villani. Optimal Transport, Old and New. Springer, New York, 2008. [52] J. Ville. Etude Critique de la Notion de Collectif. Gauthier-Villars, Paris, 1939.

174 Bibliography

[53] Richard von Mises and H. Geiringer. The Mathematical Theory of Probability and Statistics. Academic Press, New York, 1964. [54] Vladimir G. Vovk and V. V. Vyugin. On the empirical validity of the Bayesian method. Journal of the Royal Statistical Society B, 55(1):253–266, 1993. [55] V. V. V’yugin. On Turing-invariant sets. Soviet Math. Doklady, 229(4):1090–1094, 1976. [56] V. V. V’yugin. Ergodic theorems for individual random sequences. Theo- retical Computer Science, 207(2):343–361, 1998. [57] Klaus Weihrauch. Computable Analysis. Springer, 2000. [58] D. G. Willis. Computational complexity and probability constructions. J. ACM, pages 241–259, April 1970. [59] Andrew C. Yao. Theory and applications of trapdoor functions. In IEEE Symp. on Foundations of Computer Science, pages 80–91, 1982. [60] Ann Yasuhara. Recursive Function Theory and Logic. Academic Press, New York, 1971. [61] Alexander K. Zvonkin and Leonid A. Levin. The complexity of ﬁnite objects and the development of the concepts of information and randomness by means of the theory of algorithms. Russian Math. Surveys, 25(6):83–124, 1970.

175 arXiv:2105.04704v1 [cs.IT] 10 May 2021 opeiya ela t rfi eso.W hwhwi ep understa helps it inferen how inductive show and We theory, d version. information of randomness, prefix definition of its theory Kolmogorov’s as well define w as We as complexity machines, Turing theory. and probability computability of elementary concepts basic with familiar Textbooks. w.sb.d/fcly as aes ait-notes.ps.Z papers/ gacs/ faculty/ www.cs.bu.edu/ pigr n edition. 2nd Springer, orefr7lcue f9 iue ah tasmsta h au the that assumes It each. minutes 90 of lectures 7 for course A Li-Vitányi, ee Gács,Peter 7 plctosi nutv inference inductive in Applications (7) Entropy (a) properties (6) simple its complexity, Prefix (a) (5) incompressibility using Proofs (a) problem (4) halting the of Complexity (a) (3) 2 a ipeproperties Simple (a) (2) 1 a Introduction (a) (1) b h D principle MDL The (b) identities Information-theoretical (b) theorem coding The (b) measure universal distributions, computable for Randomness distribution (b) uniform the for theory randomness Martin-Lof’s (b) b loihi properties Algorithmic (b) b ento fcmlxt,invariance complexity, of Definition (b) a oooo’ theory Solomonoff’s (a) c eaint oe’ Theorem Godel’s to Relation (c) c oe n pe bounds upper and Lower (c) OREO OMGRVCOMPLEXITY KOLMOGOROV ON COURSE A nItouto oKlooo opeiyAdisApplicati its And Complexity Kolmogorov to Introduction An etr oe nDsrpinlCmlxt n Randomness and Complexity Descriptional on Notes Lecture OTNUNIVERSITY BOSTON EE G PETER 1 ACS ´ ce. l swith as ell escription dn the nding ineis dience . ons .