Lecture Notes on Descriptional Complexity and Randomness

Lecture notes on descriptional complexity and randomness

Peter Gács Boston University [email protected]

A didactical survey of the foundations of Algorithmic Information The- ory. These notes introduce some of the main techniques and concepts of the ﬁeld. They have evolved over many years and have been used and ref- erenced widely. “Version history” below gives some information on what has changed when.

Contents

Contents iii

1 Complexity1 1.1 Introduction ...... 1 1.1.1 Formal results ...... 3 1.1.2 Applications ...... 6 1.1.3 History of the problem...... 8 1.2 Notation ...... 10 1.3 Kolmogorov complexity ...... 11 1.3.1 Invariance ...... 11 1.3.2 Simple quantitative estimates ...... 14 1.4 Simple properties of information ...... 16 1.5 Algorithmic properties of complexity ...... 19 1.6 The coding theorem...... 24 1.6.1 Self-delimiting complexity...... 24 1.6.2 Universal semimeasure...... 27 1.6.3 Preﬁx codes ...... 28 1.6.4 The coding theorem for 퐾(푥) ...... 30 1.6.5 Algorithmic probability...... 31 1.7 The statistics of description length ...... 32

2 Randomness 37 2.1 Uniform distribution...... 37 2.2 Computable distributions...... 40 2.2.1 Two kinds of test...... 40 2.2.2 Randomness via complexity...... 41 2.2.3 Conservation of randomness...... 43 2.3 Inﬁnite sequences ...... 46

iii Contents

2.3.1 Null sets ...... 47 2.3.2 Probability space...... 52 2.3.3 Computability ...... 54 2.3.4 Integral...... 54 2.3.5 Randomness tests ...... 55 2.3.6 Randomness and complexity...... 56 2.3.7 Universal semimeasure, algorithmic probability . . . . . 58 2.3.8 Randomness via algorithmic probability...... 60

3 Information 63 3.1 Information-theoretic relations...... 63 3.1.1 The information-theoretic identity...... 63 3.1.2 Information non-increase...... 72 3.2 The complexity of decidable and enumerable sets ...... 74 3.3 The complexity of complexity...... 78 3.3.1 Complexity is sometimes complex...... 78 3.3.2 Complexity is rarely complex ...... 79

4 Generalizations 83 4.1 Continuous spaces, noncomputable measures...... 83 4.1.1 Introduction ...... 83 4.1.2 Uniform tests...... 85 4.1.3 Sequences ...... 87 4.1.4 Conservation of randomness...... 88 4.2 Test for a class of measures ...... 90 4.2.1 From a uniform test ...... 90 4.2.2 Typicality and class tests...... 91 4.2.3 Martin-Löf’s approach...... 94 4.3 Neutral measure...... 97 4.4 Monotonicity, quasi-convexity/concavity ...... 101 4.5 Algorithmic entropy ...... 103 4.5.1 Entropy...... 104 4.5.2 Algorithmic entropy ...... 105 4.5.3 Addition theorem...... 106 4.5.4 Information...... 110 4.6 Randomness and complexity...... 111 4.6.1 Discrete space ...... 111 4.6.2 Non-discrete spaces ...... 113

iv Contents

4.6.3 Inﬁnite sequences ...... 114 4.6.4 Bernoulli tests ...... 116 4.7 Cells ...... 118 4.7.1 Partitions...... 119 4.7.2 Computable probability spaces ...... 122

5 Exercises and problems 125

A Background from mathematics 131 A.1 Topology...... 131 A.1.1 Topological spaces...... 131 A.1.2 Continuity ...... 133 A.1.3 Semicontinuity...... 134 A.1.4 Compactness...... 135 A.1.5 Metric spaces...... 136 A.2 Measures...... 141 A.2.1 Set algebras ...... 141 A.2.2 Measures...... 141 A.2.3 Integral...... 144 A.2.4 Density...... 145 A.2.5 Random transitions...... 146 A.2.6 Probability measures over a metric space ...... 147

B Constructivity 155 B.1 Computable topology ...... 155 B.1.1 Representations ...... 155 B.1.2 Constructive topological space ...... 156 B.1.3 Computable functions ...... 158 B.1.4 Computable elements and sequences ...... 159 B.1.5 Semicomputability...... 160 B.1.6 Eﬀective compactness ...... 161 B.1.7 Computable metric space ...... 163 B.2 Constructive measure theory...... 166 B.2.1 Space of measures ...... 166 B.2.2 Computable and semicomputable measures ...... 168 B.2.3 Random transitions...... 169

Bibliography 171

v Contents

Version history

May 2021: changed the notation to the now widely accepted one: Kolmogorov’s original complexity is denoted 퐶(푥) (instead of the earlier 퐾(푥)), while the prefix complexity is denoted 퐾(푥) instead of the earlier 퐻(푥). Corrected a remark on the notation 푎표. June 2013: corrected a slight mistake in the section on the section on randomness via algorithmic probability. February 2010: chapters introduced, and recast using the memoir class. April 2009: besides various corrections, a section is added on infinite sequences. This is not new material, just places the most classical results on randomness of infinite sequences before the more abstract theory. January 2008: major rewrite. • Added formal definitions throughout. • Made corrections in the part on uniform tests and generalized complexity, based on remarks of Hoyrup, Rojas and Shen. • Rearranged material. • Incorporated material on uniform tests from the work of Hoyrup-Rojas. • Added material on class tests.

vi 1 Complexity

Je n’ai fait celle-ci plus longue que parce que je n’ai pas eu le loisir de la faire plus courte.

Pascal

Ainsi, au jeu de croix ou pile, l’arrivée de croix cent fois de suite, nous paraît extraordinaire; parce que le nombre presque inﬁni des combinaisons qui peuvent arriver en cent coups étant partagé en séries regulières, ou dans lesquelles nous voyons régner un ordre facile à saisir, et en séries irregulières; celles-ci sont incomparablement plus nombreuses.

Laplace

1.1 Introduction

The present section can be read as an independent survey on the problems of randomness. It serves as some motivation for the dryer stuﬀ to follow. If we toss a coin 100 times and it shows each time Head, we feel lifted to a land of wonders, like Rosencrantz and Guildenstern in [49]. The argument that 100 heads are just as probable as any other outcome convinces us only that the axioms of Probability Theory, as developed in [28], do not solve

1 1. Complexity all mysteries they are sometimes supposed to. We feel that the sequence consisting of 100 heads is not random, though others with the same probability are. Due to the somewhat philosophical character of this paradox, its history is marked by an amount of controversy unusual for ordinary mathematics. Before the reader hastes to propose a solution, let us consider a less trivial example, due to L. A. Levin. Suppose that in some country, the share of votes for the ruling party in 30 consecutive elections formed a sequence 0.99푥푖 where for every even 푖, the number 푥푖 is the 푖-th digit of 휋 = 3.1415 .... Though many of us would feel that the election results were manipulated, it turns out to be surprisingly difficult to prove this by referring to some general principle. In a sequence of 푛 fair elections, every sequence 휔 of 푛 digits has approxi- −푛 mately the probability 푄푛 (휔) = 10 to appear as the actual sequence of third digits. Let us fix 푛. We are given a particular sequence 휔 and want to test the validity of the government’s claim that the elections were fair. We interpret the assertion “휔 is random with respect to 푄푛” as a synonym for “there is no reason to challenge the claim that 휔 arose from the distribution 푄푛”. How can such a claim be challenged at all? The government, just like the weather forecaster who announces 30% chance of rain, does not guarantee any particular set of outcomes. However, to stand behind its claim, it must agree to any bet based on the announced distribution. Let us call a payoff function with Í respect the distribution 푃 any nonnegative function 푡(휔) with 휔 푃(휔)푡(휔) ≤ 1. If a “nonprofit” gambling casino asks 1 dollar for a game and claims that each outcome has probability 푃(휔) then it must agree to pay 푡(휔) dollars on outcome 휔. We would propose to the government the following payoff function 푡0 with 푛/2 respect to 푄푛: let 푡0 (휔) = 10 for all sequences 휔 whose even digits are given by 휋, and 0 otherwise. This bet would cost the government 10푛/2 − 1 dollars. Unfortunately, we must propose the bet before the elections take place and it is unlikely that we would have come up exactly with the payoff function 푡0. Is then the suspicion unjustifiable? No. Though the function 푡0 is not as natural as to guess it in advance, it is still highly “regular”. And already Laplace assumed in [15] that the number of “regular” bets is so small we can afford to make them all in advance and still win by a wide margin. Kolmogorov discovered in [27] and [29] (probably without knowing about [15] but following a four decades long controversy on von Mises’ concept of randomness, see [53]) that to make this approach work we must define “regular” or “simple” as “having a short description” (in some formal sense to

2 1.1. Introduction be specified below). There cannot be many objects having a short description because there are not many short strings of symbols to be used as descriptions. We thus come to the principle saying that on a random outcome, all sufficiently simple payoff functions must take small values. It turns out below that this can be replaced by the more elegant principle saying that a random outcome itself is not too simple. If descriptions are written in a 2-letter alphabet then a typical sequence of 푛 digits takes 푛 log 10 letters to describe (if not stated otherwise, all logarithms in these notes are to the base 2). The digits of 휋 can be generated by an algorithm whose description takes up only some constant length. Therefore the sequence 푥1 . . . 푥푛 above can be described with approximately (푛/2) log 10 letters, since every other digit comes from 휋. It is thus significantly simpler than a typical sequence and can be declared nonrandom.

1.1.1 Formal results The theory of randomness is more impressive for infinite sequences than for finite ones, since sharp distinction can be made between random and nonrandom infinite sequences. For technical simplicity, first we will confine ourselves to finite sequences, especially a discrete sample space Ω, which we identify with the set of natural numbers. Binary strings will be identified with the natural numbers they denote. Definition 1.1.1 A Turing machine is an imaginary computing device consisting of the following. A control state belonging to a finite set 퐴 of possible control states. A fixed number of infinite (or infinitely extendable) strings of cells called tapes. Each cell contains a symbol belonging to a finite tape alphabet 퐵. On each tape, there is a read-write head observing one of the tape cells. The machine’s configuration (global state) is determined at each instant by giving its control state, the contents of the tapes and the positions of the read-write heads. The “hardware program” of the machine determines its configuration in the next step as a functon of the control state and the tape symbols observed. It can change the control state, the content of the observed cells and the position of the read- write heads (the latter by one step to the left or right). Turing machines can emulate computers of any known design (see for example [60]). The tapes are used for storing programs, input data, output and as memory. y The Turing machines that we will use to interpret descriptions will be somewhat restricted. Definition 1.1.2 Consider a Turing machine 퐹 which from a binary string 푝

3 1. Complexity and a natural number 푥 computes the output 퐹( 푝, 푥) (if anything at all). We will say that 퐹 interprets 푝 as a description of 퐹( 푝, 푥) in the presence of the side information 푥. We suppose that if 퐹( 푝, 푥) is defined then 퐹(푞, 푥) is not defined for any prefix 푞 of 푝. Such machines are called self-delimiting (s.d.). The conditional complexity 퐾퐹 (푥 | 푦) of the number 푥 with respect to the number 푦 is the length of the shortest description 푝 for which 퐹( 푝, 푦) = 푥. y

Kolmogorov and Solomonoﬀ observed that the function 퐾퐹 (푥 | 푦) depends only weakly on the machine 퐹, because there are universal Turing machines capable of simulating the work of any other Turing machine whose description is supplied. More formally, the following theorem holds: Theorem 1.1.1 (Invariance Theorem) There is a s.d. Turnig machine 푇 such that for any s.d. machine 퐹 a constant 푐퐹 exists such that for all 푥, 푦 we have 퐾푇 (푥 | 푦) ≤ 퐾퐹 (푥 | 푦) + 푐퐹. This theorem motivates the following deﬁnition.

Definition 1.1.3 Let us fix 푇 and define 퐾(푥 | 푦) = 퐾푇 (푥 | 푦) and 퐾(푥) = 퐾(푥 | 0). y The function 퐾(푥 | 푦) is not computable. We can compute a nonincreasing, convergent sequence of approximations to 퐾(푥) (it is semicomputable from above), but will not know how far to go in this sequence for some prescribed accuracy. If 푥 is a binary string of length 푛 then 퐾(푥) ≤ 푛 + 2 log 푛 + 푐 for some constant 푐. The description of 푥 with this length gives 푥 bit-for-bit, along with some information of length 2 log 푛 which helps the s.d. machine find the end of the description. For most binary strings of lenght 푛, no significantly shorter description exists, since the number of short descriptions is small. Below, this example is generalized and sharpened for the case when instead of counting the number of simple sequences, we measure their probability. Definition 1.1.4 Denote by 푥∗ the first one among the shortest descriptions of 푥. y The correspondence 푥 → 푥∗ is a code in which no codeword is the prefix of another one. This implies by an argument well-known in Information Theory the inequality ∑︁ 2−퐾 (푥 | 푦) ≤ 1, (1.1.1) 푥 hence only a few objects 푥 can have small complexity. The converse of the same argument goes as follows. Let 휇 be a computable probability distribution, one for

4 1.1. Introduction which there is a binary program computing 휇(휔) for each 휔 to any given degree of accuracy. Let 퐾(휇) be the length of the shortest one of these programs. Then

퐾(휔) ≤ − log 휇(휔) + 퐾(휇) + 푐. (1.1.2)

Here 푐 is a universal constant. These two inequalities are the key to the estimates of complexity and the characterization of randomness by complexity. Denote 푑휇 (휔) = − log 휇(휔) − 퐾(휔). Inequality (1.1.1) implies 푑휇 (휔) 푡휇 (휔) = 2 can be viewed as a payoﬀ function. Now we are in a position to solve the election paradox. We propose the payoﬀ function

2− log 푄푛 (휔)−퐾 (휔 | 푛) to the government. (We used the conditional complexity 퐾(휔 | 푛) because the uniform distribution 푄푛 depends on 푛.) If every other digit of the outcome 푥 푡(푥) comes from 휋 then 퐾(푥 | 푛) ≤ (푛/2) log 10 + 푐0 hence we win a sum 2 ≥ 푛/2 푐110 from the government (for some constants 푐0, 푐1 > 0), even though the bet does not contain any reference to the number 휋. The fact that 푡휇 (휔) is a payoﬀ function implies by Markov’s Inequality for any 푘 > 0 휇{휔 : 퐾(휔) < − log 휇(휔) − 푘} < 2−푘. (1.1.3) Inequalities (1.1.2) and (1.1.3) say that with large probability, the complexity 퐾(휔) of a random outcome 휔 is close to its upper bound − log 휇(휔)+퐾(휇). This law occupies distinguished place among the “laws of probability”, because if the outcome 휔 violates any such law, the complexity falls far below the upper bound. Indeed, a proof of some “law of probability” (like the law of large numbers, the law of iterated logarithm, etc.) always gives rise to some simple computable payoﬀ function 푡(휔) taking large values on the outcomes violating the law, just as in the election example. Let 푚 be some large number, suppose that 푡(휔) has 푚 complexity < 푚/2, and that 푡(휔0) > 2 . Then inequality (1.1.2) can be applied to 휈(휔) = 휇(휔)푡(휔), and we get

퐾(휔) ≤ − log 휇(휔) − 푚 + 퐾(휈) + 푐0

≤ − log 휇(휔) − 푚/2 + 퐾(휇) + 푐1 for some constants 푐0, 푐1.

5 1. Complexity

More generally, the payoff function 푡휇 (휔) is maximal (up to a multiplicative constant) among all payoff functions that are semicomputable (from below). Hence the quantity − log 휇(휔) − 퐾(휔) is a universal test of randomness. Its value measures the deficiency of randomness in the outcome 휔 with respect to the distribution 휇, or the extent of justified suspicion against the hypothesis 휇 given the outcome 휔.

1.1.2 Applications Algorithmic Information Theory (AIT) justifies the intuition of random sequences as nonstandard analysis justifies infinitely small quantities. Any statement of classical probability theory is provable without the notion of randomness, but some of them are easier to find using this notion. Due to the incomputability of the universal randomness test, only its approximations can be used in practice. Pseudorandom sequences are sequences generated by some algorithm, with some randomness properties with respect to the coin-tossing distribution. They have very low complexity (depending on the strength of the tests they withstand, see for example [14]), hence are not random. Useful pseudorandom sequences can be defined using the notion of computational complexity, for example the number of steps needed by a Turing machine to compute a certain function. The existence of such sequences can be proved using some difficult unproven (but plausible) assumptions of computation theory. See [6], [59], [24], [34].

Inductive inference The incomputable “distribution” m(휔) = 2−퐾 (휔) has the remarkable property that, the test 푑(휔 | m), shows all outcomes 휔 “random” with respect to it. Rela- tions (1.1.2) and (1.1.3) can be read as saying that if the real distribution is 휇 then 휇(휔) and m(휔) are close to each other with large probability. Therefore if we know that 휔 comes from some unknown simple distribution 휇 then we can use m(휔) as an estimate of 휇(휔). This suggests to call m the “apriori probability” (but we will not use this term much). The randomness test 푑휇 (휔) can be interpreted in the framework of hypothesis testing: it is the likelihood ratio between the hypothesis 휇 and the ﬁxed alternative hypothesis m. In ordinary statistical hypothesis testing, some properties of the unknown distribution 휇 are taken for granted. The sample 휔 is generally a large independent sample: 휇 is supposed to be a product distribution. Under these conditions, the universal test could probably be reduced to some of the tests used in statistical practice. However, these conditions do not hold in other applications: for

6 1.1. Introduction example testing for independence in a proposed random sequence, predicting some time series of economics, or pattern recognition. If the apriori probability m is a good estimate of the actual probability then we can use the conditional apriori probability for prediction, without reference to the unknown distribution 휇. For this purpose, one first has to define apriori probability for the set of infinite sequences of natural numbers, as done in [61]. We denote this function by 푀. For any finite sequence 푥, the number 푀(푥) is the apriori probability that the outcome is some extension of 푥. Let 푥, 푦 be finite sequences. The formula 푀(푥 푦) (1.1.4) 푀(푥) is an estimate of the conditional probability that the next terms of the outcome will be given by 푦 provided that the first terms are given by 푥. It converges to the actual conditional probability 휇(푥 푦)/휇(푥) with 휇-probability 1 for any computable distribution 휇 (see for example [46]). To understand the surprising generality of the formula (1.1.4), suppose that some infinite sequence 푧(푖) is given in the following way. Its even terms are the subsequent digits of 휋, its odd terms are uniformly distributed, independently drawn random digits. Let 푧(1 : 푛) = 푧(1)··· 푧(푛). Then 푀(푧(1 : 2푖)푎/푀(푧(1 : 2푖)) converges to 0.1 for 푎 = 0,..., 9, while 푀(푧(1 : 2푖 + 1)푎)/푀(푧(1 : 2푖 + 1)) converges to 1 if 푎 is the 푖-th digit of 휋, and to 0 otherwise. The inductive inference formula using conditional apriori probability can be viewed as a mathematical form of “Occam’s Razor”: the advice to predict by the simplest rule fitting the data. It can also viewed as a realization of Bayes’Rule, with a universally applicable apriori distribution. Since the distribution 푀 is incomputable, we view the main open problem of inductive inference to find maximally efficient approximations to it. Sometimes, even a simple approximation gives nontrivial results (see [3]).

Information theory Since with large probability, 퐾(휔) is close to − log 휇(휔), the entropy Í − 휔 휇(휔) log 휇(휔) of the distribution 휇 is close to the average complexity Í 휔 휇(휔)퐾(휔). The complexity 퐾(푥) of an object 푥 can indeed be interpreted as the distribution-free deﬁnition of information content. The correspondence 푥 ↦→ 푥∗ is a sort of universal code: its average (even individual) “rate”, or codeword length is almost equally good for any simple computable distribution.

7 1. Complexity

It is of great conceptual help to students of statistical physics that entropy can be defined now not only for ensembles (probability distributions), but for individual states of a physical system. The notion of an individual incompressible sequence, a sequence whose complexity is maximal, proved also extremely useful in finding information-theoretic lower bounds on the computing speed of certain Turing machines (see [39]). The conditional complexity 퐾(푥 | 푦) obeys identities analogous to the information-theoretical identities for conditional entropy, but these identities are less trivial to prove in AIT. The information 퐼(푥 : 푦) = 퐾(푥) + 퐾( 푦) − 퐾(푥, 푦) has several interpretations. It is known that 퐼(푥 : 푦) is equal, to within an additive constant, to 퐾( 푦) − 퐾( 푦 | 푥∗), the amount by which the object 푥∗ (as defined in the previous section) decreases our uncertainty about 푦. But it can be written as − log m2 (푥, 푦) − 퐾(푥, 푦) = 푑((푥, 푦) | m2) where m2 = m × m is the product distribution of m with itself. It is thus the deficiency of randomness of the pair (푥, 푦) with respect to this product distribution. Since any object is random with respect to m, we can view the randomness of the pair (푥, 푦) with respect to the product m2 as the independence of 푥 and 푦 from each other. Thus “information” measures the “deficiency of independence”.

Logic Some theorems of Mathematical Logic (in particular, Gödel’s theorem) have a strong quantitative form in AIT, with new philosophical implications (see [8], [9], [31], [33]). Levin based a new system of intuitionistic analysis on his Inde- pendence Principle (see below) in [33].

1.1.3 History of the problem P. S. Laplace thought that the number of “regular” sequences (whatever “regular” means) is much smaller than the number of irregular ones (see [15]). In the first attempt at formalization hundred years later, R. von Mises defined an infinite binary sequence as random (a “Kollektiv”) if the relative frequencies converge in any subsequence selected according to some (non-anticipating) “rule” (whatever “rule” means, see [53]). As pointed out by A. Wald and others, Mises’s definitions are sound only if a countable set of possible rules is fixed. The logician A. Church, in accordance with his famous thesis, proposed to understand “rule” here as “recursive (computable) function”. The Mises selection rules can be considered as special randomness tests. In the ingenious work [52], J. Ville proved that they do not capture all relevant

8 1.1. Introduction properties of random sequences. In particular, a Kollektiv can violate the law of iterated logarithm. He proposed to consider arbitrary payoff functions (a countable set of them), as defined on the set of infinite sequences—these are more commonly known as martingales. For the solution of the problem of inductive inference, R. Solomonoff introduced complexity and apriori probability in [45] and proved the Invariance The- orem. A. N. Kolmogorov independently introduced complexity as a measure of individual information content and randomness, and proved the Invariance The- orem (see [27] and [29]). The incomputability properties of complexity have noteworthy philosophical implications (see [4], [8], [9]. P. Martin-Löf defined in [38] randomness for infinite sequences. His concept is essentially the synthesis of Ville and Church (as noticed in [41]). He recognized the existence of a universal test, and pointed out the close connection between the randomness of an infinite sequence and the complexity of its initial segments. L. A. Levin defined the apriori probability 푀 as a maximal (to within a multiplicative constant) semicomputable measure. With the help of a modi- fied complexity definition, he gave a simple and general characterization of random sequences by the behavior of the complexity of their initial segments (see [61], [30]). In [17] and [31], the information-theoretical properties of the self-delimiting complexity (as defined above) are exactly described. See also [10], [42] and [58]. In [33], Levin defined a deficiency of randomness 푑(휔 | 휇) in a uniform manner for all (computable or incomputable) measures 휇. He proved that all outcomes are random with respect to the apriori probability 푀. In this and earlier papers, he also proved the Law of Information Conservation, stating that the information 퐼(훼; 훽) in a sequence 훼 about a sequence 훽 cannot be significantly increased by any algorithmic processing of 훼 (even using random number generators). He derived this law from a so-called Law of Randomness Conservation via the definition of information 퐼(훼; 훽) as deficiency of randomness with respect to the product distribution 푀2. Levin suggested the Independence Principle saying that any sequence 훼 arising in nature contains only finite information 퐼(훼; 훽) about any sequence 훽 defined by mathematical means. With this principle, he showed that the use of more powerful notions of definability in randomness tests (or the notion of complexity) does not lead to fewer random sequences among those arising in nature. The monograph [16] is very useful as background information on the various attempts in this century at solving the paradoxes of probability theory. The

9 1. Complexity work [33] is quite comprehensive but very dense; it can be recommended only to devoted readers. The work [61] is comprehensive and readable but not quite up-to-date. The surveys [12], [41] and [43] can be used to complement it. The most up-to-date and complete survey, which subsumes most of these notes, is [36]. AIT created many interesting problems of its own. See for example [10], [11], [17], [18], [19], [20], [35], [33], [37], [41], [44], [47], and the technically diﬃcult results [48] and [55].

Acknowledgment The form of exposition of the results in these notes and the general point of view represented in them were greatly inﬂuenced by Leonid Levin. More recent com- munication with Paul Vitányi, Mathieu Hoyrup, Cristóbal Rojas and Alexander Shen has also been very important.

1.2 Notation

When not stated otherwise, log means base 2 logarithm. The cardinality of a set 퐴 will be denoted by |퐴|. (Occasionally there will be inconsistency, sometimes denoting it by |퐴|, sometimes by #퐴.) If 퐴 is a set then 1퐴 (푥) is its indicator function: ( 1 if 푥 ∈ 퐴, 1퐴 (푥) = 0 otherwise.

The empty string is denoted by Λ. The set 퐴∗ is the set of all finite strings of elements of 퐴, including the empty string. Let 푙(푥) denote the length of string 푥. (Occasionally there will be inconsistency, sometimes denoting it by |푥|, sometimes by 푙(푥)). For sequences 푥 and 푦, let 푥 v 푦 denote that 푥 is a prefix of 푦. For a string 푥 and a (finite or infinite) sequence 푦, we denote their concatenation by 푥 푦. For a sequence 푥 and 푛 ≤ 푙(푥), the 푛-th element of 푥 is 푥(푛), and 푥(푖 : 푗) = 푥(푖)··· 푥( 푗). Sometimes we will also write

푥 ≤푛 = 푥(1 : 푛).

10 1.3. Kolmogorov complexity

The string 푥1 ··· 푥푛 will sometimes be written also as (푥1, . . . , 푥푛). For natural number 푚 let 훽(푚) be the binary sequence denoting 푚 in the binary notation. We denote by 푋N the set of all inﬁnite sequences of elements of 푋. The sets of natural numbers, integers, rational numbers, real numbers and complex numbers will be denoted respectively by N, Z, Q, R, C. The set of nonnegative real numbers will be denoted by R+. The set of real numbers with −∞, ∞ added (with the appropriate topology making it compact) will be denoted by R. Let

∗ ∗ S푟 = {0, . . . , 푟 − 1} , S = N , B = {0, 1}.

We use ∧ and ∨ to denote min and max, further

|푥|+ = 푥 ∨ 0, |푥|− = | − 푥|+ for real numbers 푥. Let h·i be some standard one-to-one encoding of N∗ to N, with partial inverses [·]푖 where [h푥i]푖 = 푥(푖) for 푖 ≤ 푙(푥). For example, we can have 1 h푖, 푗i = (푖 + 1)(푖 + 푗 + 1) + 푗, h푛1, . . . , 푛푘+1i = hh푛1, . . . , 푛푘i, 푛푘+1i. 2 We will use h·, ·i in particular as a pairing function over N. Similar pairing functions will be assumed over other domains, and they will also be denoted by h·, ·i. Another use of the notation (·, ·) may arise when the usual notation (푥, 푦) of an ordered pair of real numbers could be confused with the same notation of an open interval. In this case, we will use (푥, 푦) to denote the pair. The relations + ∗ 푓 < 푔, 푓 < 푔 mean inequality to within an additive constant and multiplicative constant respectively. The ﬁrst is equivalent to 푓 ≤ 푔 + 푂(1), the second to 푓 = 푂(푔). The ∗ ∗ relation 푓 =∗ 푔 means 푓 < 푔 and 푓 > 푔.

1.3 Kolmogorov complexity

1.3.1 Invariance It is natural to try to deﬁne the information content of some text as the size of the smallest string (code) from which it can be reproduced by some decoder,

11 1. Complexity interpreter. We do not want too much information “to be hidden in the decoder” we want it to be a “simple” function. Therefore, unless stated otherwise, we require that our interpreters be computable, that is partial recursive functions.

Definition 1.3.1 A partial recursive function 퐴 from S2 × S to S will be called a (binary) interpreter, or decoder. We will often refer to its first argument as the program, or code. y Partial recursive functions are relatively simple; on the other hand, the class of partial recursive functions has some convenient closure properties. Definition 1.3.2 For any binary interpreter 퐴 and strings 푥, 푦 ∈ S, the number

퐶퐴 (푥 | 푦) = min{푙( 푝) : 퐴( 푝, 푦) = 푥 } is called the conditional Kolmogorov-complexity of 푥 given 푦, with respect to the interpreter 퐴 . (If the set after the “min” is empty, then the minimum is ∞). We write 퐶퐴 (푥) = 퐶퐴 (푥 | Λ). y

The number 퐶퐴 (푥) measures the length of the shortest description for 푥 when the algorithm 퐴 is used to interpret the descriptions. The value 퐶퐴 (푥) depends, of course, on the underlying function 퐴. But, as Theorem 1.3.1 shows, if we restrict ourselves to sufficiently powerful interpreters 퐴, then switching between them can change the complexity function only by amounts bounded by some additive constant. Therefore complexity can be considered an intrinsic characteristic of finite objects. Definition 1.3.3 A binary p.r. interpreter 푈 is called optimal if for any binary p.r. interpreter 퐴 there is a constant 푐 < ∞ such that for all 푥, 푦 we have

퐶푈 (푥 | 푦) ≤ 퐶퐴 (푥 | 푦) + 푐. (1.3.1)

y Theorem 1.3.1 (Invariance Theorem) There is an optimal p.r. binary interpreter.

Proof. The idea is to use interpreters that come from universal partial recursive functions. However, not all such functions can be used for universal interpreters. Let us introduce an appropriate pairing function. For 푥 ∈ B푛, let

푥표 = 푥(1)0푥(2)0 . . . 푥(푛 − 1)0푥(푛)1 where 푥표 = 푥1 for 푙(푥) = 1. Any binary sequence 푥 having at least one 1 in an even position can be represented, uniquely, in the form 푥 = 푎표푏.

12 1.3. Kolmogorov complexity

We know there is a p.r. function 푉 : S2 × S2 × S → S that is universal: for any p.r. binary interpreter 퐴, there is a string 푎 such that for all 푝, 푥, we have 퐴( 푝, 푥) = 푉 (푎, 푝, 푥). Let us deﬁne the function 푈( 푝, 푥) as follows. We represent the string 푝 in the form 푝 = 푢표푣 and deﬁne 푈( 푝, 푥) = 푉 (푢, 푣, 푥). Then 푈 is a p.r. interpreter. Let us verify that it is optimal. Let 퐴 be a p.r. binary interpreter, 푎 a binary string such that 퐴( 푝, 푥) = 푈(푎, 푝, 푥) for all 푝, 푥. Let 푥, 푦 be two strings. If 퐶퐴 (푥 | 푦) = ∞, then (1.3.1) holds trivially. Otherwise, let 푝 be a binary string of length 퐶퐴 (푥 | 푦) with 퐴( 푝, 푦) = 푥. Then we have

푈(푎표 푝, 푦) = 푉 (푎, 푝, 푦) = 퐴( 푝, 푦) = 푥, and 퐶푈 (푥 | 푦) ≤ 2푙(푎) + 퐶퐴 (푥 | 푦). The constant 2푙(푎) is in the above proof a bound on the complexity of description of the interpreter 퐴 for the optimal interpreter 푈. Let us note that for any two optimal interpreters 푈 (1) , 푈 (2) , there is a constant 푐 such that for all 푥, 푦, we have

|퐶푈 (1) (푥 | 푦) − 퐶푈 (2) (푥 | 푦)| < 푐. (1.3.2)

Hence the complexity 퐶푈 (푥) of description of an object 푥 does not depend strongly on the interpreter 푈. Still, for every string 푥, there is an optimal interpreter 푈 with 퐶푈 (푥) = 0. Imposing a universal bound on the table size of the Turing machines used to implement optimal interpreters, we can get a universal bound on the constants in (1.3.2). The theorem motivates the following definition. Definition 1.3.4 We fix an optimal binary p.r. interpreter 푈 and write 퐶(푥 | 푦) for 퐶푈 (푥 | 푦). y Theorem 1.3.1 (as well as other invariance theorems) is used in AIT for much more than just to show that 퐶퐴 (푥) is a proper concept. It is the principal tool to find upper bounds on 퐶(푥); this is why most such upper bounds are proved to hold only to within an additive constant. The optimal interpreter 푈( 푝, 푥) defined in the proof of Theorem 1.3.1 is obviously a universal partial recursive function. Because of its convenient properties we will use it from now on as our standard universal p.r. function, and we will refer to an arbitrary p.r. function as 푈푝 (푥) = 푈( 푝, 푥).

13 1. Complexity

Definition 1.3.5 We define 푈푝 (푥1, . . . , 푥푘) = 푈푝 (h푥1 . . . , 푥푘i). Similarly, we will refer to an arbitrary computably enumerable set as the range 푊푝 of some 푈푝. We often do not need the second argument of the function 푈( 푝, 푥). We therefore define 푈( 푝) = 푈( 푝, Λ). y It is of no consequence that we chose binary strings as as descriptions. It is rather easy to define, (Exercise) for any two natural numbers 푟, 푠, a standard 푟 encoding cnv푠 of base 푟 strings 푥 into base 푠 strings with the property log 푟 푙(cnv푟 (푥)) ≤ 푙(푥) + 1. 푠 log 푠

Now, with 푟-ary strings as descriptions, we must define 퐶퐴 (푥) as the minimum of 푙( 푝) log 푟 over all programs 푝 in S푟 with 퐴( 푝) = 푥. The equivalence of the definitions for different values of 푟 is left as an exercise.

1.3.2 Simple quantitative estimates We found it meaningful to speak about the information content, complexity of a ﬁnite object. But how to compute this quantity? It turns out that complexity is not a computable function, but much is known about its statistical behavior. The following notation will be useful in the future. + Deﬁnition 1.3.6 The relation 푓 < 푔 means inequality to within an additive constant, that there is a constant 푐 such that for all 푥, 푓 (푥) ≤ 푔(푥). We can write this also as 푓 ≤ 푔 + 푂(1). We also say that 푔 additively dominates 푓 . The + + ∗ relation 푓 =+ 푔 means 푓 < 푔 and 푓 > 푔. The relation 푓 < 푔 among nonnegative functions means inequality to within a multiplicative constant. It is equivalent + to log 푓 < log 푔 and 푓 = 푂(푔). We also say that 푔 multiplicatively dominates 푓 . ∗ ∗ The relation 푓 =∗ 푔 means 푓 < 푔 and 푓 > 푔. y With the above notation, here are the simple properties. Theorem 1.3.2 The following simple upper bound holds. a) For any natural number 푚, we have

+ 퐶(푚) < log 푚. (1.3.3)

b) For any positive real number 푣 and string 푦, every ﬁnite set 퐸 of size 푚 has at least 푚(1 − 2−푣+1) elements 푥 with 퐶(푥 | 푦) ≥ log 푚 − 푣.

14 1.3. Kolmogorov complexity

Corollary 1.3.7 lim푛→∞ 퐶(푛) = ∞. Theorem 1.3.2 suggests that if there are so few strings of low complexity, then 퐶(푛) converges fast to ∞. In fact this convergence is extremely slow, as shown in a later section. Proof of Theorem 1.3.2. First we prove (a). Let the interpreter 퐴 be deﬁned such + that if 훽(푚) = 푝 then 퐴( 푝, 푦) = 푚. Since |훽(푚)| = dlog 푚e, we have 퐶(푚) < 퐶퐴 (푚) < log 푚 + 1. Part (b) says that the trivial estimate (1.3.3) is quite sharp for most numbers under 푚. The reason for this is that since there are only few short programs, there are only few objects of low complexity. For any string 푦, and any positive real natural number 푢, we have

|{푥 : 퐶(푥 | 푦) ≤ 푢}| < 2푢+1. (1.3.4)

To see this, let 푛 = blog 푢c. The number 2푛+1 − 1 of diﬀerent binary strings of length ≤ 푛 is an upper bound on the number of diﬀerent shortest programs of length ≤ 푢. Now (b) follows immediately from (1.3.4). The three properties of complexity contained in Theorems 1.3.1 and 1.3.2 are the ones responsible for the applicabiliy of the complexity concept. Several important later results can be considered transformations, generalizations or analogons of these. Let us elaborate on the upper bound (1.3.3). For any string 푥 ∈ S푟, we have

+ 퐶(푥) < 푛 log 푟 + 2 log 푟. (1.3.5) In particular, for any binary string 푥, we have

+ 퐶(푥) < 푙(푥).

표 푟 Indeed, let the interpreter 퐴 be deﬁned such that if 푝 = 훽(푟) cnv2 (푥) then 푙(훽 (푟)) 퐴( 푝, 푦) = 푥. We have 퐶퐴 (푥) ≤ 2 + 푛 log 푟 + 1 ≤ (푛 + 2) log 푟 + 3. Since + 퐶(푥) < 퐶퐴 (푥), we are done. We will have many more upper bound proofs of this form, and will not always + explicitly write out the reference to 퐶(푥 | 푦) < 퐶퐴 (푥 | 푦), needed for the last step. Apart from the conversion, inequality (1.3.5) says that since the beginning of a program can command the optimal interpreter to copy the rest, the complexity of a binary sequence 푥 of length 푛 is not larger than 푛. Inequality (1.3.5) contains the term 2 log 푟 instead of log 푟 since we have to apply the encoding 푤표 to the string 푤 = 훽(푟); otherwise the interpreter cannot

15 1. Complexity detach it from the program 푝. We could use the code 훽(|푤|)표푤, which is also a prefix code, since the first, detachable part of the codeword tells its length. For binary strings 푥, natural numbers 푛 and real numbers 푢 ≥ 1 we define

퐽(푢) = 푢 + 2 log 푢, 휄(푥) = 훽(|푥|)표푥, 휄(푛) = 휄(훽(푛)).

+ + We have 푙(휄(푥)) < 퐽(푙(푥)) for binary sequences 푥, and 푙(휄(푟)) < 퐽(log 푟) for numbers 푟. Therefore (1.3.5) is true with 퐽(log 푟) in place of 2 log 푟. Of course, 퐽(푥) could be replaced by still smaller functions, for example 푥 + 퐽(log 푥). We return to this topic later.

1.4 Simple properties of information

If we transform a string 푥 by applying to it a p.r. function, then we cannot gain information over the amount contained in 푥 plus the amount needed to describe the transformation. The string 푥 becomes easier to describe, but it helps less in describing other strings 푦.

Theorem 1.4.1 For any partial recursive function 푈푞, over strings, we have

+ 퐶(푈푞 (푥) | 푦) < 퐶(푥 | 푦) + 퐽(푙(푞)), + 퐶( 푦 | 푈푞 (푥)) > 퐶( 푦 | 푥) − 퐽(푙(푞)).

+ Proof. To define an interpreter 퐴 with 퐶퐴 (푈푞 (푥) | 푦) < 퐶(푥 | 푦) + 퐽(푙(푞)), we + define 퐴(휄(푞) 푝, 푦) = 푈푞 (푈( 푝, 푦)). To define an interpreter 퐵 with 퐶퐵 ( 푦 | 푥) < 퐶( 푦 | 푈푞 (푥)) + 퐽(푞), we define 퐵(휄(푞) 푝, 푥) = 푈( 푝, 푈푞 (푥)). The following notation is natural. Definition 1.4.1 The definition of conditional complexity is extended to pairs, etc. by

퐶(푥1, . . . , 푥푚 | 푦1, . . . , 푦푛) = 퐶(h푥1, . . . , 푥푚i | h푦1, . . . , 푦푛i).

y With the new notation, here are some new results.

16 1.4. Simple properties of information

Corollary 1.4.2 For any one-to-one p.r. function 푈푝, we have

+ |퐶(푥) − 퐶(푈푝 (푥))| < 퐽(푙( 푝)).

Further,

We made several times implicit use of a basic additivity property of complexity which makes it possible to estimate the joint complexity of a pair by the complexities of its constituents. As expressed in Theorem 1.4.2, it says essentially that to describe the pair of strings, it is enough to know the description of the first member and of a method to find the second member using our knowledge of the first one. Theorem 1.4.2 + 퐶(푥, 푦) < 퐽(퐶(푥)) + 퐶( 푦 | 푥). Proof. We define an interpreter 퐴 as follows. We decompose any binary string 푝 into the form 푝 = 휄(푤)푞, and let 퐴( 푝, 푧) = 푈(푞, 푈(푤)). Then 퐶퐴 (푥, 푦) is bounded, to within an additive constant, by the right-hand side of the above inequality.

Corollary 1.4.3 For any p.r. function 푈푝 over pairs of strings, we have

+ 퐶(푈푝 (푥, 푦)) < 퐽(퐶(푥)) + 퐶( 푦 | 푥) + 퐽(푙( 푝)) + < 퐽(퐶(푥)) + 퐶( 푦) + 퐽(푙( 푝)), and in particular, + 퐶(푥, 푦) < 퐽(퐶(푥)) + 퐶( 푦).

17 1. Complexity

This corollary implies the following continuity property of the function 퐶(푛): for any natural numbers 푛, ℎ, we have

+ |퐶(푛 + ℎ) − 퐶(푛)| < 퐽(log ℎ). (1.4.3)

Indeed, 푛 + ℎ is a recursive function of 푛 and ℎ. The term 2 log 퐶(푥) making up the diﬀerence between 퐶(푥) and 퐽(퐶(푥)) in Theorem 1.4.2 is attributable to the fact that minimal descriptions cannot be concatenated without loosing an “endmarker”. It cannot be eliminated, since there is a constant 푐 such that for all 푛, there are binary strings 푥, 푦 of length ≤ 푛 with

퐶(푥) + 퐶( 푦) + log 푛 < 퐶(푥, 푦) + 푐.

Indeed, there are 푛2푛 pairs of binary strings whose sum of lengths is 푛. Hence by Theorem 1.3.2b, there will be a pair (푥, 푦) of binary strings whose sum of lengths is 푛, with 퐶(푥, 푦) > 푛 + log 푛 − 1. For these strings, inequality (1.3.5) + + implies 퐶(푥) + 퐶( 푦) < 푙(푥) + 푙( 푦). Hence 퐶(푥) + 퐶( 푦) + log 푛 < 푛 + log 푛 < 퐶(푥, 푦) + 1. Regularities in a string will, in general, decrease its complexity radically. If the whole string 푥 of length 푛 is given by some rule, that is we have 푥(푘) = 푈푝 (푘), for some recursive function 푈푝, then

+ 퐶(푥) < 퐶(푛) + 퐽(푙( 푝)).

Indeed, let us deﬁne the p.r. function 푉 (푞, 푘) = 푈푞 (1) . . . 푈푞 (푘). Then 푥 = 푉 ( 푝, 푛) and the above estimate follows from Corollary 1.4.2. For another example, let 푥 = 푦1 푦1 푦2 푦2 ··· 푦푛 푦푛, and 푦 = 푦1 푦2 ··· 푦푛. Then 퐶(푥) =+ 퐶( 푦) even though the string 푥 is twice longer. This follows from Corol- lary 1.4.2 since 푥 and 푦 can be obtained from each other by a simple recursive operation. Not all “regularities” decrease the complexity of a string, only those which distinguish it from the mass of all strings, putting it into some class which is both small and algorithmically deﬁnable. For a binary string√ of length 푛, it is nothing unusual to have its number of 0-s between 푛/2 − 푛 and 푛/2. Therefore such strings can have maximal or almost maximal complexity. If 푥 has 푘 zeroes then the inequality + 푛 퐶(푥) < log + 퐽(log 푛) + 퐽(log 푘) 푘 follows from Theorems 1.3.2 and 1.4.3.

18 1.5. Algorithmic properties of complexity

Theorem 1.4.3 Let 퐸 = 푊푝 be an enumerable set of pairs of strings deﬁned enumerated with the help of the program 푝 for example as

푊푝 = {푈( 푝, 푥) : 푥 ∈ N}. (1.4.4)

We deﬁne the section 퐸푎 = {푥 : h푎, 푥i ∈ 퐸}. (1.4.5) Then for all 푎 and 푥 ∈ 퐸푎, we have

+ 퐶(푥 | 푎) < log |퐸푎| + 퐽(푙( 푝)).

Proof. We define an interpreter 퐴(푞, 푏) as follows. We decompose 푞 into 푞 = 휄( 푝)훽(푡). From a standard enumeration (푎(푘), 푥(푘))(푘 = 1, 2,...) of the set 푊푝, we produce an enumeration 푥(푘, 푏)(푘 = 1, 2,...), without repetition, of 푏 the set 푊푝 for each 푏, and define 퐴(푞, 푏) = 푥(푡, 푏). For this interpreter 퐴, we + 푎 get 푘퐴 (푥 | 푎) < 퐽(푙( 푝)) + log |퐸 |. It follows from Theorem 1.3.2 that whenever the set 퐸푎 is finite, the estimate of Theorem 1.4.3 is sharp for most of its elements 푥. The shortest descriptions of a string 푥 seem to carry some extra information in them, above the description of 푥. This is suggested by the following strange identities. Theorem 1.4.4 We have 퐶(푥, 퐶(푥)) =+ 퐶(푥).

+ + Proof. The inequality > follows from (1.4.1). To prove <, let 퐴( 푝) = h푈( 푝, 푙( 푝))i. Then 퐶퐴 (푥, 퐶(푥)) ≤ 퐶(푥). More generally, we have the following inequality. Theorem 1.4.5 + 퐶( 푦 | 푥, 푖 − 퐶( 푦 | 푥, 푖)) < 퐶( 푦 | 푥, 푖). The proof is exercise. The above inequalities are in some sense “pathological”, and do not necessarily hold for all “reasonable” deﬁnitions of descriptional complexity.

1.5 Algorithmic properties of complexity

The function 퐶(푥) is not computable, as it will be shown in this section. However, it has a property closely related to computability. Let Q be the set of rational numbers.

19 1. Complexity

Deﬁnition 1.5.1 (Computability) Let 푓 : S → R be a function. It is computable if there is a recursive function 푔(푥, 푛) with rational values, and | 푓 (푥)−푔(푥, 푛)| < 1/푛. y Deﬁnition 1.5.2 (Semicomputability) A function 푓 : S → (−∞, ∞] is (lower) semicomputable if the set

{(푥, 푟) : 푥 ∈ S, 푟 ∈ Q, 푟 < 푓 (푥)} is recursively enumerable. A function 푓 is called upper semicomputable if − 푓 is lower semicomputable. y It is easy to show the following: • A function 푓 : S → R is lower semicomputable iﬀ there exists a recursive function with rational values, (or, equivalently, a computable real function) 푔(푥, 푛) nondecreasing in 푛, with 푓 (푥) = lim푛→∞ 푔(푥, 푛). • A function 푓 : S → R is computable if it is both lower and upper semicomputable. The notion of semicomputibility is naturally extendable over other discrete domains, like S × S. It is also useful to extend it to functions R → R. Let I denote the set of open rational intervals of R, that is

훽 = {(푝, 푞) : 푝, 푞 ∈ Q, 푝 < 푞}.

Deﬁnition 1.5.3 (Computability for real functions) Let 푓 : R → R be a function. It is computable if there is a recursively enumerable set F ⊆ 훽2 such that denoting F퐽 = { 퐼 : (퐼, 퐽) ∈ F} we have Ø 푓 −1 (퐽) = 퐼.

퐼 ∈F퐽 y

This says that if 푓 (푥) ∈ 퐼 then sooner or later we will ﬁnd an interval 퐼 ∈ F퐽 with the property that 푓 (푧) ∈ 퐽 for all 푧 ∈ 퐼. Note that computability implies continuity. Deﬁnition 1.5.4 We say that 푓 : R → R is lower semicomputable if there is a recursively enumerable set G ⊆ Q × 훽 such that denoting G푞 = { 퐼 : (퐼, 푞) ∈ G} we have Ø 푓 −1 ((푞, ∞)) = 퐼.

퐼 ∈F퐽 y

20 1.5. Algorithmic properties of complexity

This says that if 푓 (푥) > 푟 then sooner or later we will ﬁnd an interval 퐼 ∈ F퐽 with the property that 푓 (푧) > 푞 for all 푧 ∈ 퐼. The following facts are useful and simple to verify: Proposition 1.5.5 a) The function d·e : R → R is lower semicomputable. b) If 푓 , 푔 : R → R are computable then their composition 푓 (푔(·)) is, too. c) If 푓 : R → R is lower semicomputable and 푔 : R → R (or S → R) is computable then 푓 (푔(·)) is lower semicomputable. d) If 푓 : R → R is lower semicomputable and monotonic, and 푔 : R → R (or S → R) is lower semicomputable then 푓 (푔(·)) is lower semicomputable. Deﬁnition 1.5.6 We introduce a universal lower semicomputable function. For every binary string 푝, and 푥 ∈ S, let

푆푝 (푥) = sup{ 푦/푧 : 푦, 푧 ∈ 푍, 푧 ≠ 0, hh푥i, 푦, 푧i ∈ 푊푝 } where 푊푝 was deﬁned in (1.4.4). y

The function 푆푝 (푥) is lower semicomputable as a function of the pair ( 푝, 푥), and for diﬀerent values of 푝, it enumerates all semicomputable functions of 푥.

Definition 1.5.7 Let 푆푝 (푥1, . . . , 푥푘) = 푆푝 (h푥1, . . . , 푥푘i). For any lower semicomputable function 푓 , we call any binary string 푝 with 푆푝 = 푓 a Gödel number of 푓 . There is no universal computable function, but for any computable function 푔, we call any number hh푝i, h푞ii a Gödel number of 푔 if 푔 = 푆푝 = −푆푞. y With this notion, we claim the following. Theorem 1.5.1 The function 퐶(푥 | 푦) is upper semicomputable. Proof. By Theorem 1.3.2 there is a constant 푐 such that for any strings 푥, 푦, we have 퐶(푥 | 푦) < logh푥i+푐. Let some Turing machine compute our optimal binary p.r. interpreter. Let 푈푡 ( 푝, 푦) be defined as 푈( 푝, 푦) if this machine, when started on input ( 푝, 푦), gives an output in 푡 steps, undefined otherwise. Let 퐾푡 (푥 | 푦) be the smaller of logh푥i + 푐 and

min{푙( 푝) ≤ 푡 : 푈푡 ( 푝, 푦) = 푥 }.

푡 푡 Then the function 퐾 (푥 | 푦) is computable, monotonic in 푡 and lim푡→∞ 퐾 (푥 | 푦) = 퐶(푥 | 푦). The function 퐶(푛) is not computable. Moreover, it has no nontrivial partial recursive lower bound.

21 1. Complexity

Theorem 1.5.2 Let 푈푝 (푛) be a partial recursive function whose values are numbers smaller than 퐶(푛) whenever deﬁned. Then

+ 푈푝 (푛) < 퐽(푙( 푝)).

Proof. The proof of this theorem resembles “Berry’s paradox ”, which says: “The least number that cannot be defined in less than 100 words” is a definition of that number in 12 words. The paradox can be used to prove, whenever we agree on a formal notion of “definition”, that the sentence in quotes is not a definition. Let 푥( 푝, 푘) for 푘 = 1, 2,... be an enumeration of the domain of 푈푝. For any natural number 푚 in the range of 푈푝, let 푓 ( 푝, 푚) be equal to the first 푥( 푝, 푘) with 푈푝 (푥( 푝, 푘)) = 푚. Then by definition, 푚 < 퐶( 푓 ( 푝, 푚)). On the other hand, 푓 ( 푝, 푚) is a p.r. function, hence applying Corollary 1.4.3 and (1.3.5) we get

+ 푚 < 퐶( 푓 ( 푝, 푚)) < 퐶( 푝) + 퐽(퐶(푚)) + < 푙( 푝) + 1.5 log 푚.

Church’s first example of an undecidable recursively enumerable set used universal partial recursive functions and the diagonal technique of Gödel and Cantor, which is in close connection to Russel’s paradox. The set he constructed is complete in the sense of “many-one reducibility”. The first example of sets not complete in this sense were Post’s simple sets. Definition 1.5.8 A computably enumerable set is simple if its complement is infinite but does not contain any infinite computably enumerable subsets. y The paradox used in the above proof is not equivalent in any trivial way to Russell’s paradox, since the undecidable computably enumerable sets we get from Theorem 1.5.2 are simple. Let 푓 (푛) ≤ log 푛 be any recursive function with lim푛→∞ 푓 (푛) = ∞. We show that the set 퐸 = {푛 : 퐶(푛) ≤ 푓 (푛)} is simple. It follows from Theorem 1.5.1 that 퐸 is recursively enumerable, and from Theorem 1.3.2 that its complement is infinite. Let 퐴 be any computably enumerable set disjoint from 퐸. The restriction 푓퐴 (푛) of the function 푓 (푛) to 퐴 is a p.r. lower bound for 퐶(푛). It follows from Theorem 1.5.2 that 푓 (퐴) is bounded. Since 푓 (푛) → ∞, this is possible only if 퐴 is finite. Gödel used the diagonal technique to prove the incompleteness of any sufficiently rich logical theory with a recursively enumerable axiom system. His technique provides for any sufficiently rich computably enumerable theory a concrete example of an undecidable sentence. Theorem 1.5.2 leads to a new proof of this

22 1.5. Algorithmic properties of complexity result, with essentially different examples of undecidable propositions. Let us consider any first-order language 퐿 containing the language of standard first- order arithmetic. Let h·i be a standard encoding of the formulas of this theory into natural numbers. Let the computably enumerable theory 푇푝 be the theory for which the set of codes hΦi of its axioms Φ is the computably enumerable set 푊푝 of natural numbers. Corollary 1.5.9 There is a constant 푐 < ∞ such that for any 푝, if any sentences with the meaning “푚 < 퐶(푛)” for 푚 > 퐽(푙( 푝)) + 푐 are provable in theory 푇푝 then some of these sentences are false.

Proof. For some 푝, suppose that all sentences “푚 < 퐶(푛)” provable in 푇푝 are true. For a sentence Φ, let 푇푝 ` Φ denote that Φ is a theorem of the theory 푇푝. Let us deﬁne the function

퐴( 푝, 푛) = max{푚 : 푇푝 ` “푚 < 퐶(푛)”}.

This function is semicomputable since the set {( 푝, hΦi) : 푇푝 ` Φ} is recursively enumerable. Therefore there is a binary string 푎 such that 퐴( 푝, 푛) = 푆(푎, h푝, 푛i). It is easy to see that we have a constant string 푏 such that

푆(푎, h푝, 푛i) = 푆(푞, 푛) where 푞 = 푏푎푝. (Formally, this statement follows for example from the so-called 푛 푆푚-theorem.) The function 푆푞 (푛) is a lower bound on the complexity function 퐶(푛). By an immediate generalization of Theorem 1.5.2 for semicomputable + + functions), we have 푆(푞, 푛) < 퐽(푙(푞)) < 퐽(푙( 푝)). We have seen that the complexity function 퐶(푥 | 푦) is not computable, and does not have any nontrivial recursive lower bounds. It has many interesting upper bounds, however, for example the ones in Theorems 1.3.2 and 1.4.3. The following theorem characterizes all upper semicomputable upper bounds (majorants) of 퐶(푥 | 푦). Theorem 1.5.3 (Levin) Let 퐹(푥, 푦) be a function of strings semicomputable from + above. The relation 퐶(푥 | 푦) < 퐹(푥, 푦) holds for all 푥, 푦 if and only if we have

+ log |{푥 : 퐹(푥, 푦) < 푚}| < 푚 (1.5.1) for all strings 푦 and natural numbers 푚.

+ Proof. Suppose that 퐶(푥 | 푦) < 퐹(푥, 푦) holds. Then (1.5.1) follows fromb of Theorem 1.3.2.

23 1. Complexity

Now suppose that (1.5.1) holds for all 푦, 푚. Then let 퐸 be the computably enumerable set of triples (푥, 푦, 푚) with 퐹(푥, 푦) < 푚. It follows from (1.5.1) + and Theorem 1.4.3 that for all (푥, 푦, 푚) ∈ 퐸 we have 퐶(푥 | 푦, 푚) < 푚. By + Theorem 1.4.5 then 퐶(푥 | 푦) < 푚. The minimum of any ﬁnite number of majorants is again a majorant. Thus, we can combine diﬀerent heuristics for recognizing patterns of sequences.

1.6 The coding theorem

1.6.1 Self-delimiting complexity Some theorems on the addition of complexity do not have as simple a form as desirable. For example, Theorem 1.4.2 implies

+ 퐶(푥, 푦) < 퐽(퐶(푥)) + 퐶( 푦).

+ We would like to see just 퐶(푥) on the right-hand side, but the inequality 퐶(푥, 푦) < 퐶(푥) + 퐶( 푦) does not always hold (see Exercise5). The problem is that if we compose a program for (푥, 푦) from those of 푥 and 푦 then we have to separate them from each other somehow. In this section, we introduce a variant of Kol- mogorov’s complexity discovered by Levin and independently by Chaitin and Schnorr, which has the property that “programs” are self-delimiting: this will free us of the necessity to separate them from each other. Definition 1.6.1 A set of strings is called prefix-free if for any pair 푥, 푦 of elements in it, 푥 is not a prefix of 푦. A one-to-one function into a set of strings is a prefix code, or instantaneous code if its domain of definition is prefix free. An interpreter 푓 ( 푝, 푥) is self-delimiting (s.d.) if for each 푥, the set 퐷푥 of strings 푝 for which 푓 ( 푝, 푥) is defined is a prefix-free set. y Example 1.6.2 The mapping 푝 → 푝표 is a prefix code. y A self-delimiting p.r. function 푓 ( 푝) can be imagined as a function computable on a special self-delimiting Turing machine. Definition 1.6.3 A Turing machine T is self-delimiting if it has no input tape, but can ask any time for a new input symbol. After some computation and a few such requests (if ever) the machine decides to write the output and stop. y The essential difference between a self-delimiting machine T and an ordinary Turing machine is that T does not know in advance how many input symbols

24 1.6. The coding theorem suﬃce; she must compute this information from the input symbols themselves, without the help of an “endmarker”. Requiring an interpreter 퐴( 푝, 푥) to be self-delimiting in 푝 has many advan- tages. First of all, when we concatenate descriptions, the interpreter will be able to separate them without endmarkers. Lost endmarkers were responsible for the additional logarithmic terms and the use of the function 퐽(푛) in Theorem 1.4.2 and several other formulas for 퐾. Deﬁnition 1.6.4 A self-delimiting partial recursive interpreter 푇 is called optimal if for any other s.d. p.r. interpreter 퐹, we have

+ 퐶푇 < 퐶퐹 . (1.6.1) y Theorem 1.6.1 There is an optimal s.d. interpreter. Proof. The proof of this theorem is similar to the proof of Theorem 1.3.1. We take the universal partial recursive function 푉 (푎, 푝, 푥). We transform each function 푉푎 ( 푝, 푥) = 푉 (푎, 푝, 푥) into a self-delimiting function 푊푎 ( 푝, 푥) = 푊(푎, 푝, 푥), but so as not to change the functions 푉푎 which are already self-delimiting. Then we form 푇 from 푊 just as we formed 푈 from 푉. It is easy to check that the function 푇 thus defined has the desired properties. Therefore we are done if we construct 푊. Let some Turing machine M compute 푦 = 푉푎 ( 푝, 푥) in 푡 steps. We define 푊푎 ( 푝, 푥) = 푦 if 푙( 푝) ≤ 푡 and if M does not compute in 푡 or fewer steps any output 푉푎 (푞, 푥) from any extension or prefix 푞 of length ≤ 푡 of 푝. If 푉푎 is self- delimiting then 푊푎 = 푉푎. But 푊푎 is always self-delimiting. Indeed, suppose that 푝0 v 푝1 and that M computes 푉푎 ( 푝푖, 푥) in 푡푖 steps. The value 푊푎 ( 푝푖, 푥) can be defined only if 푙( 푝푖) ≤ 푡푖. The definition of 푊푎 guarantees that if 푡0 ≤ 푡1 then 푊푎 ( 푝1, 푥) is undefined, otherwise 푊푎 ( 푝0, 푥) is undefined. Definition 1.6.5 We fix an optimal self-delimiting p.r. interpreter 푇 ( 푝, 푥) and write 퐾(푥 | 푦) = 퐶푇 (푥 | 푦). We call 퐾( 푦 | 푥) the (self-delimiting) conditional complexity of 푥 with respect to 푦. y We will use the expression “self-delimiting” only if confusion may arise. Oth- erwise, under complexity, we generally understand the self-delimiting complexity. Let 푇 ( 푝) = 푇 ( 푝, Λ). All definitions for unconditional complexity, joint complexity etc. are automatically in force for 퐻 = 퐶푇 .

25 1. Complexity

Let us remark that the s.d. interpreter deﬁned in Theorem 1.6.1 has the following stronger property. For any other s.d. interpreter 퐺 there is a string 푔 such that we have 퐺(푥) = 푇 (푔푥) (1.6.2) for all 푥. We could say that the interpreter 푇 is universal. Let us show that the functions 퐾 and 퐻 are asymptotically equal. Theorem 1.6.2 + + 퐾 < 퐻 < 퐽(퐾). (1.6.3)

+ Proof. Obviously, 퐾 < 퐻. We deﬁne a self-delimiting p.r. function 퐹( 푝, 푥). The machine computing 퐹 tries to decompose 푝 into the form 푝 = 푢표푣 such that 푢 is the number 푙(푣) in binary notation. If it succeeds, it outputs 푈(푣, 푥). We have

+ + 퐾( 푦 | 푥) < 퐶퐹 ( 푦 | 푥) < 퐶( 푦 | 푥) + 2 log 퐶( 푦 | 푥).

Let us review some of the relations proved in Sections 1.3 and 1.4. Instead of + the simple estimate 퐶(푥) < 푛 for a binary string 푥, of length 푛, only the obvious consequence of Theorem 1.6.2 holds that is

+ 퐾(푥) < 퐽(푛).

We must similarly change Theorem 1.4.3. We will use the universal p.r. function 푇푝 (푥) = 푇 ( 푝, 푥). The r.e. set 푉푝 is the range of the function 푇푝. Let 퐸 = 푉푝 be an enumerable set of pairs of strings. The section 퐸푎 is deﬁned as in (1.4.5). Then for all 푥 ∈ 퐸푎, we have

+ 퐾(푥 | 푎) < 퐽(log |퐸푎|) + 푙( 푝). (1.6.4)

We do not need 퐽(푙( 푝)) since the function 푇 is self-delimiting. However, the real analogon of Theorem 1.4.3 is the Coding Theorem, to be proved later in this section. Partb of Theorem 1.3.2 holds for 퐻, but is not the strongest what can be said. The counterpart of this theorem is the inequality (1.6.10) below. Theorem 1.4.1 and its corollary hold without change for 퐻. The additivity relations will be proved in a sharper form for 퐻 in the next section.

26 1.6. The coding theorem

1.6.2 Universal semimeasure Self-delimiting complexity has an interesting characterization that is very useful in applications. Definition 1.6.6 A function 푤 : 푆 → [0, 1] is called a semimeasure over the space 푆 if ∑︁ 푤(푥) ≤ 1. (1.6.5) 푥 It is called a measure (a probability distribution) if equality holds here. A semimeasure is called constructive if it is lower semicomputable. A constructive semimeasure which multiplicatively dominates every other constructive semimeasure is called universal. y Remark 1.6.7 Just as in an earlier remark, we warn that semimeasures will later be defined over the space NN. They will be characterized by a nonnegative function 푤 over S = N∗ but with a condition different from (1.6.5). y The following remark is useful. Proposition 1.6.8 If a constructive semimeasure 푤 is also a measure then it is computable.

Proof. If we compute an approximation 푤푡 of the function 푤 from below for Í which 1 − 휀 < 푥 푤푡 (푥) then we have |푤(푥) − 푤푡 (푥)| < 휀 for all 푥. Theorem 1.6.3 There is a universal constructive semimeasure.

Proof. Every computable family 푤푖 (푖 = 1, 2,... ) of semimeasures is dominated Í −푖 by the semimeasure 푖 2 푤푖. Since there are only countably many constructive semimeasures there exists a semimeasure dominating them all. The only point to prove is that the dominating semimeasure can be made constructive. Below, we construct a function 휇 푝 (푥) lower semicomputable in 푝, 푥 such that 휇 푝 as a function of 푥 is a constructive semimeasure for all 푝 and all constructive semimeasures occur among the 푤푝. Once we have 휇 푝 we are done because Í for any positive computable function 훿( 푝) with 푝 훿( 푝) ≤ 1 the semimeasure Í 푝 훿( 푝)휇 푝 is obviously lower semicomputable and dominates all the 휇 푝. Let 푆푝 (푥) be the lower semicomputable function with Gödel number 푝 and 푡 let 푆푝 (푥) be a function recursive in 푝, 푡, 푥 with rational values, dondecreasing in 푡 푡 푡, such that lim푡 푆푝 (푥) = max{0, 푆푝 (푥)} and for each 푡, 푝, the function 푆푝 (푥) is 푡 different from 0 only for 푥 ≤ 푡. Let us define 휇 푝 recursively in 푡 as follows. Let 0 푡 푡+1 휇 푝 (푥) = 0. Suppose that 휇 푝 is already defined. If 푆푝 is a semimeasure then we

27 1. Complexity

푡+1 푡+1 푡+1 푡 푡 define 휇 푝 = 푆푝 , otherwise 휇 푝 = 휇 푝. Let 휇 푝 = lim푡 휇 푝. The function 휇 푝 (푥) is, by its definition, lower semicomputable in 푝, 푥 and a semimeasure for each fixed 푝. It is equal to 푆푝 whenever 푆푝 is a semimeasure, hence it enumerates all constructive semimeasures. The above theorem justifies the following notation. Definition 1.6.9 We choose a fixed universal constructive semimeasure and call it m(푥). y With this notation, we can make it a little more precise in which sense this measure dominates all other constructive semimeasures. Definition 1.6.10 For an arbitrary constructive semimeasure 휈 let us define ∑︁ m(휈) = {m( 푝) : the (self-delimiting) program 푝 computes 휈}. (1.6.6)

Similar notation will be used for other objects: for example now m( 푓 ) make sense for a recursive function 푓 . y Let us apply the above concept to some interesting cases. Theorem 1.6.4 For all constructive semimeasures 휈 and for all strings 푥, we have

∗ m(휈)휈(푥) < m(푥). (1.6.7)

Proof. Let us repeat the proof of Theorem 1.6.3, using m( 푝) in place of 훿( 푝). We obtain a new constructive semimeasure m0(푥) with the property m(휈)휈(푥) ≤ m0(푥) for all 푥. Noting m0(푥) =∗ m(푥) ﬁnishes the proof.

1.6.3 Prefix codes The main theorem of this section says 퐾(푥) =+ − log m(푥). Before proving it, we relate descriptional complexity to the classical coding problem of information theory. Definition 1.6.11 A (binary) code is any mapping from a set 퐸 of binary sequences to a set of objects. This mapping is the decoding function of the code. If the mapping is one-to-one, then sometimes the set 퐸 itself is also called a code. A code is a prefix code if 퐸 is a prefix-free set. y The interpreter 푈( 푝) is a decoding function. Definition 1.6.12 We call string 푝 the first shortest description of 푥 if it is the lexicographically first binary word 푞 with |푞| = 퐶(푥) and 푈(푞) = 푥. y

28 1.6. The coding theorem

The set of ﬁrst shortest descriptions is a code. This imposes the implicit lower bound stated in (1.5.1) on 퐶(푥):

|{푥 : 퐶(푥) = 푛}| ≤ 2푛. (1.6.8)

The least shortest descriptions in the deﬁnition of 퐾(푥) form moreover a preﬁx code. We recall a classical result of information theory.

Lemma 1.6.13 (Kraft’s Inequality, see [13]) For any sequence 푙1, 푙2,... of natural numbers, there is a prefix code with exactly these numbers as codeword lengths, if and only if ∑︁ 2−푙푖 ≤ 1. (1.6.9) 푖 Proof. Recall the standard correspondence 푥 ↔ [푥] between binary strings and binary subintervals of the interval [0, 1]. A prefix code corresponds to a set of disjoint intervals, and the length of the interval [푥] is 2−푙(푥) . This proves that (1.6.9) holds for the lengths of codewords of a prefix code. Suppose that 푙푖 is given and (1.6.9) holds. We can assume that the sequence 푙푖 is nondecreasing. Chop disjoint, adjacent intervals 퐼1, 퐼2,... of length −푙1 −푙2 2 , 2 ,... from the left end of the interval [0, 1]. The right end of 퐼푘 is Í푘 −푙 푗 푗=1 2 . Since the sequence 푙 푗 is nondecreasing, all intervals 퐼 푗 are binary intervals. Take the binary string corresponding to 퐼 푗 as the 푗-th codeword.

+ Corollary 1.6.14 We have − log m(푥) < 퐾(푥). Proof. The lemma implies ∑︁ 2−퐾 ( 푦 | 푥) ≤ 1. (1.6.10) 푦

Since 퐾(푥) is an upper semicomputable function, the function 2−퐾 (푥) is lower + semicomputable. Hence it is a constructive semimeasure, and we have − log m(푥) < 퐾(푥). The construction in the second part of the proof of Lemma 1.6.13 has some disadvantages. We do not always want to rearrange the numbers 푙 푗, for example because we want that the order of the codewords reﬂect the order of our original objects. Without rearrangement, we can still achieve a code with only slightly longer codewords.

29 1. Complexity

Lemma 1.6.15 (Shannon-Fano code) (see [13]) Let 푤1, 푤2,... be positive num- Í bers with 푗 푤푗 ≤ 1. There is a binary preﬁx code 푝1, 푝2,... where the codewords are in lexicographical order, such that

| 푝푗 | ≤ − log 푤푗 + 2. (1.6.11)

Proof. We follow the construction above and cut oﬀ disjoint, adjacent (not necessarily binary) intervals 퐼 푗 of length 푤푗 from the left end of [0, 1]. Let 푣푗 be the length of the longest binary intervals contained in 퐼 푗. Let 푝푗 be the binary word corresponding to the ﬁrst one of these. Four or fewer intervals of length 푣푗 cover 퐼 푗. Therefore (1.6.11) holds.

1.6.4 The coding theorem for 퐾(푥) The above results suggest 퐾(푥) ≤ − log m(푥). But in the proof, we have to deal with the problem that m is not computable. Theorem 1.6.5 (Coding Theorem, see [31, 17, 10])

퐾(푥) =+ − log m(푥) (1.6.12)

Proof. We construct a self-delimiting p.r. function 퐹( 푝) with the property that 퐶퐹 (푥) ≤ − log m(푥) + 4. The function 퐹 to be constructed is the decoding function of a preﬁx code hence the code-construction of Lemma 1.6.13 proposes itself. But since the function m(푥) is only lower semicomputable, it is given only by a sequence converging to it from below. Let {(푧푡, 푘푡) : 푡 = 1, 2,...} be a recursive enumeration of the set {(푥, 푘) : 푘 < m(푥)} without repetition. Then ∑︁ ∑︁ ∑︁ ∑︁ 2−푘푡 = 2−푘푡 ≤ 2m(푥) < 2.

푡 푥 푧푡=푥 푥

−푘푡−1 Let us cut off consecutive adjacent, disjoint intervals 퐼푡 of length 2 from the left side of the interval [0, 1]. We define 퐹 as follows. If [푝] is a largest binary subinterval of some 퐼푡 then 퐹( 푝) = 푧푡. Otherwise 퐹( 푝) is undefined. The function 퐹 is obviously self-delimiting and partial recursive. It follows from the construction that for every 푥 there is a 푡 with 푧푡 = 푥 and 0.5m(푥) < 2−푘푡 . Therefore, for every 푥 there is a 푝 such that 퐹( 푝) = 푥 and | 푝| ≤ − log m(푥)+ 4.

30 1.6. The coding theorem

The Coding Theorem can be straightforwardly generalized as follows. Let 푓 (푥, 푦) be a lower semicomputable nonnegative function. Then we have + 퐾( 푦 | 푥) < − log 푓 (푥, 푦) (1.6.13) Í for all 푥 with 푦 푓 (푥, 푦) ≤ 1. The proof is the same.

1.6.5 Algorithmic probability We can interpret the Coding Theorem as follows. It is known in classical information theory that for any probability distribution 푤(푥)(푥 ∈ 푆) a binary preﬁx code 푓푤 (푥) can be constructed such that 푙( 푓푤 (푥)) ≤ − log 푤(푥) + 1. We learned that for computable, moreover, even for contstructive distributions 푤 there is a universal code with a self-delimiting partial-recursive decoding function 푇 independent of 푤 such that for the codeword length 퐾(푥) we have

퐾(푥) ≤ − log 푤(푥) + 푐푤.

Here, only the additive constant 푐푤 depends on the distribution 푤. Let us imagine the self-delimiting Turing machine T computing our interpreter 푇 ( 푝). Every time the machine asks for a new bit of the description, we could toss a coin to decide whether to give 0 or 1.

Deﬁnition 1.6.16 Let 푃푇 (푥) be the probability that the self-delimiting machine T gives out result 푥 after receiving random bits as inputs. y Í −푙( 푝) We can write 푃푇 (푥) = 푊푇 (푥) where 푊푇 (푥) is the set {2 : 푇 ( 푝) = 푥 }. −퐾 (푥) Since 2 = max 푊푇 (푥), we have − log 푃푇 (푥) ≤ 퐾(푥). The semimeasure 푃푇 ∗ is constructive, hence 푃 < m, hence + + 퐻 < − log m < − log 푃푇 ≤ 퐻, hence + + − log 푃푇 = − log m = 퐻.

Hence the sum 푃푇 (푥) of the set 푊푇 (푥) is at most a constant times larger than its maximal element 2−퐾 (푥) . This relation was not obvious in advance. The outcome 푥 might have high probability because it has many long descriptions. But we found that then it must have a short description too. In what follows it will be convenient to fix the definition of m(푥) as follows. Definition 1.6.17 From now on let us define

m(푥) = 푃푇 (푥). (1.6.14) y

31 1. Complexity

1.7 The statistics of description length

+ We can informally summarize the relation 퐻 = − log 푃푇 saying that if an object has many long descriptions then it has a short one. But how many descriptions does an object really have? With the Coding Theorem, we can answer several questions of this kind. The reader can view this section as a series of exercises in the technique acquired up to now. Theorem 1.7.1 Let 푓 (푥, 푛) be the number of binary strings 푝 of length 푛 with 푇 ( 푝) = 푥 for the universal s.d. interpreter 푇 deﬁned in Theorem 1.6.1. Then for every 푛 ≥ 퐾(푥), we have

log 푓 (푥, 푛) =+ 푛 − 퐾(푥, 푛). (1.7.1)

+ Using 퐾(푥) < 퐾(푥, 푛), which holds just as (1.4.1), and substituting 푛 = 퐾(푥) + we obtain log 푓 (푥, 퐾(푥)) =+ 퐾(푥) − 퐾(푥, 퐾(푥)) < 0, implying the following. Corollary 1.7.1 The number of shortest descriptions of any object is bounded by a universal constant. Since the number log 푓 (푥, 퐾(푥)) is nonnegative, we also derived the identity

퐾(푥, 퐾(푥)) =+ 퐾(푥) (1.7.2) which could have been proven also in the same way as Theorem 1.4.4.

Proof of Theorem 1.7.1. It is more convenient to prove equation (1.7.1) in the form 2−푛 푓 (푥, 푛) =∗ m(푥, 푛) (1.7.3) ∗ where m(푥, 푦) = m(h푥, 푦i). First we prove <. Let us deﬁne a p.r. self-delimiting function 퐹 by 퐹( 푝) = h푇 ( 푝), 푙( 푝)i. Applying 퐹 to a coin-tossing argument, 2−푛 푓 (푥, 푛) is the probability that the pair h푥, 푛i is obtained. Therefore the left- hand side of (1.7.3), as a function of h푥, 푛i, is a constructive semimeasure, dominated by m(푥, 푛). ∗ Now we prove >. We deﬁne a self-delimiting p.r. function 퐺 as follows. The machine computing 퐺( 푝) tries to decompose 푝 into three segments 푝 = 훽(푐)표푣푤 in such a way that 푇 (푣) is a pair h푥, 푙( 푝) + 푐i. If it succeeds then it outputs 푥. By the universality of 푇, there is a binary string 푔 such that 푇 (푔푝) = 퐺( 푝) for all 푝. Let 푟 = 푙(푔). For an arbitrary pair 푥, 푛, let 푞 be a shortest binary string with 푇 (푞) = h푥, 푛i, and 푤 an arbitrary string of length

푙 = 푛 − 푙(푞) − 푟 − 푙(훽(푟)표).

32 1.7. The statistics of description length

Then 퐺(훽(푟)표푞푤) = 푇 (푔훽(푟)표푞푤) = 푥. Since 푤 is arbitrary here, there are 2푙 binary strings 푝 of length 푛 with 푇 ( 푝) = 푥. How many objects are there with a given complexity 푛? We can answer this question with a good approximation.

Definition 1.7.2 Let 푔푇 (푛) be the number of objects 푥 ∈ S with 퐾(푥) = 푛, and 퐷푛 the set of binary strings 푝 of length 푛 for which 푇 ( 푝) is defined. Let us define the moving average 푐 1 ∑︁ ℎ푇 (푛, 푐) = 푔푇 (푛 + 푖). 2푐 + 1 푖=−푐 y Here is an estimation of these numbers. Theorem 1.7.2 ([47]) There is a natural number 푐 such that

+ log |퐷푛| = 푛 − 퐾(푛), (1.7.4) + log ℎ푇 (푛, 푐) = 푛 − 퐾(푛). (1.7.5) Since we are interested in general only in accuracy up to additive constants, we could omit the normalizing factor 1/(2푐 + 1) from the moving average ℎ푇 . We do not know whether this average can be replaced by 푔푇 (푛). This might depend on the special universal partial recursive function we use to construct the optimal s.d. interpreter 푇 ( 푝). But equation (1.7.5) is true for any optimal s.d. interpreter 푇 (such that the inequality (1.6.1) holds for all 퐹) while there is an optimal s.d. interpreter 퐹 for which 푔퐹 (푛) = 0 for every odd 푛. Indeed, let 퐹(00푝) = 푇 ( 푝) if 푙( 푝) is even, 퐹(1푝) = 푇 ( 푝) if 푙( 푝) is odd and let 퐹 be undeﬁned in all other cases. Then 퐹 is deﬁned only for inputs of even length, while 퐶퐹 ≤ 퐶푇 + 2. Lemma 1.7.3 ∑︁ ∗ m(푥, 푦) = m(푥). 푦 Proof. The left-hand side is a constructive semimeasure therefore it is dominated ∗ by the right side. To show >, note that by equation (1.4.2) as applied to 퐻, we + ∗ Í have 퐾(푥, 푥) = 퐾(푥). Therefore m(푥) = m(푥, 푥) < 푦 m(푥, 푦).

Proof of Theorem 1.7.2: Let 푑푛 = |퐷푛|. Using Lemma 1.7.3 and Theorem 1.7.1 we have ∑︁ ∗ 푛 ∑︁ ∗ 푛 푔푇 (푛) ≤ 푑푛 = 푓 (푥, 푛) = 2 m(푥, 푛) = 2 m(푛). 푥 푥

33 1. Complexity

Since the complexity 퐻 has the same continuity property (1.4.3) as 퐾, we can 푛 푐 2 ∗ derive from here ℎ푇 (푛, 푐) ≤ 2 m(푛)푂(2 푐 ). To prove ℎ푇 (푛, 푐) > 푑푛 for an appropriate 푐, we will prove that the complexity of at least half of the elements of 퐷푛 is near 푛. For some constant 푐0, we have 퐾( 푝) ≤ 푛 + 푐0 for all 푝 ∈ 퐷푛. Indeed, if the s.d. p.r. interpreter 퐹( 푝) is equal to 푝 where 푇 ( 푝) is deﬁned and is undeﬁned + otherwise then 퐶퐹 ( 푝) ≤ 푛 for 푝 ∈ 퐷푛. By 퐾( 푝, 푙( 푝)) = 퐾( 푝), and a variant of Lemma 1.7.3, we have

∑︁ ∗ ∑︁ ∗ m( 푝) = m( 푝, 푛) = m(푛).

푝∈퐷푛 푝∈퐷푛

Using the expression (1.7.4) for 푑푛, we see that there is a constant 푐1 such that

−1 ∑︁ −퐾 ( 푝) −푛+푐1 푑푛 2 ≤ 2 . 푝∈퐷푛

Using Markov’s Inequality, hence the number of elements 푝 of 퐷푛 with 퐾( 푝) ≥ 푛 − 푐1 − 1 is at least 푑푛/2. We ﬁnish the proof making 푐 to be the maximium of 푐0 and 푐1 + 1. What can be said about the complexity of a binary string of length 푛? We + mentioned earlier the estimate 퐾(푥) < 푛 + 2 log 푛. The same argument gives the stronger estimate + 퐾(푥) < 푙(푥) + 퐾(푙(푥)). (1.7.6) By a calculation similar to the proof of Theorem 1.7.2, we can show that the estimate (1.7.6) is exact for most binary strings. First, it follows just as in Lemma 1.7.3 that there is a constant 푐 such that ∑︁ 2−푛 2−퐾 ( 푝) ≤ 푐2−푛m(푛). 푙( 푝)=푛

From this, Markov’s Inequality gives that the number of strings 푝 ∈ B푛 with 퐾( 푝) < 푛 − 푘 is at most 푐2푛−푘. There is a more polished way to express this result. Deﬁnition 1.7.4 Let us introduce the following function of natural numbers:

퐾+(푛) = max 퐾(푘). 푘≤푛 y

34 1.7. The statistics of description length

This is the smallest monotonic function above 퐾(푛). It is upper semicom- + putable just as 퐻 is, hence 2−퐾 is universal among the monotonically decreasing constructive semimeasures. The following theorem expresses 퐾+ more directly in terms of 퐾. Theorem 1.7.3 We have 퐾+(푛) =+ log 푛 + 퐾(blog 푛c).

+ Proof. To prove <, we construct a s.d. interpreter as follows. The machine computing 퐹( 푝) ﬁnds a decomposition 푢푣 of 푝 such that 푇 (푢) = 푙(푣), then outputs the number whose binary representation (with possible leading zeros) is the binary string 푣. With this 퐹, we have

+ 퐾(푘) < 퐶퐹 (푘) ≤ log 푛 + 퐻(blog 푛c) (1.7.7) for all 푘 ≤ 푛. The numbers between 푛/2 and 푛 are the ones whose binary representation has length blog 푛c + 1. Therefore if the bound (1.7.6) is sharp for most 푥 then the bound (1.7.7) is sharp for most 푘. The sharp monotonic estimate log 푛 + 퐾(blog 푛c) for 퐾(푛) is less satisfactory than the sharp estimate log 푛 for Kolmogorov’s complexity 퐶(푛), because it is not a computable function. We can derive several computable upper bounds from it, for example log 푛 + 2 log log 푛, log 푛 + log log 푛 + 2 log log log 푛, etc. but none of these is sharp for large 푛. We can still hope to find computable upper bounds of 퐻 which are sharp infinitely often. The next theorem provides one. Theorem 1.7.4 (see [47]) There is a computable upper bound 퐺(푛) of the function 퐾(푛) with the property that 퐺(푛) = 퐾(푛) holds for infinitely many 푛. For the proof, we make use of a s.d. Turing machine T computing the function 푇. Similarly to the proof of Theorem 1.5.1, we introduce a time-restricted complexity. Definition 1.7.5 We know that for some constant 푐, the function 퐾(푛) is bounded by 2 log 푛 + 푐. Let us fix such a 푐. For any number 푛 and s.d. Turing machine M, let 퐾M (푛; 푡) be the minimum of 2 log 푛 + 푐 and the lengths of descriptions from which M computes 푛 in 푡 or fewer steps. y At the time we proved the Invariance Theorem we were not interested in exactly how the universal p.r. function 푉 was to be computed. Now we need to be more specific. Definition 1.7.6 Let us agree now that the universal p.r. function is computed by a universal Turing machine V with the property that for any Turing machine

35 1. Complexity

M there is a constant 푚 such that the number of steps needed to simulate 푡 steps of M takes no more than 푚푡 steps on V. Let T be the Turing machine constructed from V computing the optimal s.d. interpreter 푇, and let us write

퐾(푛, 푡) = 퐾T (푛, 푡).

y With these deﬁnitions, for each s.d. Turing machine M there are constants 푚0, 푚1 with 퐾(푛, 푚0푡) ≤ 퐾M (푛, 푡) + 푚1. (1.7.8) The function 퐾(푛; 푡) is nonincreasing in 푡.

Proof of Theorem 1.7.4. First we prove that there are constants 푐0, 푐1 such that 퐾(푛; 푡) < 퐾(푛; 푡 − 1) implies

퐾(h푛, 푡i; 푐0푡) ≤ 퐾(푛; 푡) + 푐1. (1.7.9)

Indeed, we can construct a Turing machine M simulating the work of T such that if T outputs a number 푛 in 푡 steps then M outputs the pair h푛, 푡i in 2푡 steps:

퐾M (h푛, 푡i; 2푡) ≤ 퐾(푛, 푡).

Combining this with the inequality (1.7.8) gives the inequality (1.7.9). We define now a recursive function 퐹 as follows. To compute 퐹(푥), the s.d. Turing machine first tries to find 푛, 푡 such that 푥 = h푛, 푡i. (It stops even if it did not find them.) Then it outputs 퐾(푥; 푐0푡). Since 퐹(푥) is the length of some description of 푥, it is an upper bound for 퐾(푥). On the other hand, suppose that 푥 is one of the infinitely many integers of the form h푛, 푡i with

퐾(푛) = 퐾(푛; 푡) < 퐾(푛; 푡 − 1).

+ Then the inequality (1.7.9) implies 퐹(푥) ≤ 퐾(푛) + 푐1 while 퐾(푛) < 퐾(푥) is known (it is the equivalent of 1.4.4 for 퐻), so 퐹(푥) ≤ 퐾(푛) + 푐 for some constant 푐. Now if 퐹 would not be equal to 퐻 at inﬁnitely many places, only close to it (closer than 푐) then we can decrease it by some additive constant less than 푐 and modify it at ﬁnitely many places to obtain the desired function 퐺.

36 2 Randomness

2.1 Uniform distribution

Speaking of a “random binary string”, one generally understands randomness with respect to the coin-toss distribution (when the probability 푃(푥) of a binary sequence 푥 of length 푛 is 2−푛). This is the only distribution considered in the present section. One can hope to distinguish sharply between random and nonrandom infinite sequences. Indeed, an infinite binary sequence whose elements are 0 with only finitely many exceptions, can be considered nonrandom without qualifi- cation. This section defines randomness for finite strings. For a sequence 푥 of length 푛, it would be unnatural to fix some number 푘, declare 푥 nonrandom when its first 푘 elements are 0, but permit it to be random if only the first 푘 − 1 elements are 0. So, we will just declare some finite strings less random than others. For this, we introduce a certain real-valued function 푑(푥) ≥ 0 measuring the deficiency of randomness in the string 푥. The occurrence of a nonrandom sequence is considered an exceptional event, therefore the function 푑(푥) can assume large values only with small probability. What is “large” and “small” depends only on our “unit of measurement”. We require for all 푛, 푘 ∑︁ { 푃(푥) : 푥 ∈ B푛, 푑(푥) > 푘} < 2−푘, (2.1.1) saying that there be at most 2푛−푘 binary sequences 푥 of length 푛 with 푑(푥) > 푘. Under this condition, we even allow 푑(푥) to take the value ∞. To avoid arbitrariness in the distinction between random and nonrandom, the function 푑(푥) must be simple. We assume therefore that the set {(푛, 푘, 푥) : 푙(푥) = 푛, 푑(푥) > 푘} is recursively enumerable, or, which is the same, that the function

d : Ω → (−∞, ∞]

37 2. Randomness is lower semicomputable. Remark 2.1.1 We do not assume that the set of strings 푥 ∈ B푛 with small deﬁ- ciency of randomness is enumerable, because then it would be easy to construct a “random” sequence. y

Deﬁnition 2.1.2 A lower semicomputable function 푑 : S2 → R (where R is the set of real numbers) is called a Martin-Löf test (ML-test), or a probability-bounded test if it satisﬁes (2.1.1). A ML-test 푑0 (푥) is universal if it additively dominates all other ML-tests: for any other ML-test 푑(푥) there is a constant 푐 < ∞ such that for all 푥 we have 푑(푥) < 푑0 (푥) + 푐. y

If a test 푑0 is universal then any other test 푑 of randomness can discover at most by a constant amount more deficiency of randomness in any sequence 푥 than 푑0 (푥). Obviously, the difference of any two universal ML-tests is bounded by some constant. The following theorem reveals a simple connnection between descriptional complexity and a certain randomness property. Definition 2.1.3 Let us define the following function for binary strings 푥:

푑0 (푥) = 푙(푥) − 퐶(푥 | 푙(푥)). (2.1.2)

Theorem 2.1.1 (Martin-Löf) The function 푑0 (푥) is a universal Martin-Löf test.

Proof. Since 퐶(푥 | 푦) is semicomputable from above, 푑0 is semicomputable from below. The property of semicomputability holds for 푑0 as a straightforward consequence of Theorem 1.5.1 Therefore 푑0 is a ML-test. We must show that it is larger (to within an additive constant) than any other ML-test. Let 푑 be a ML- test. Let us deﬁne the function 퐹(푥, 푦) to be 푦 − 푑(푥) for 푦 = 푙(푥), and ∞ otherwise. Then 퐹 is upper semicomputable, and satisﬁes (1.5.1). Therefore by + Theorem 1.5.3, we have 퐶(푥 | 푙(푥)) < 푙(푥) − 푑(푥). Theorem 2.1.1 says that under very general assumptions about randomness, those strings 푥 are random whose descriptional complexity is close to its maximum, 푙(푥). The more “regularities” are discoverable in a sequence, the less random it is. However, this is true only of regularities which decrease the descriptional complexity. “Laws of randomness” are regularities whose probability is high, for example the law of large numbers, the law of iterated logarithm, the arcsine law, etc. A random sequence will satisfy all such laws.

38 2.1. Uniform distribution

Example 2.1.4 The law of large numbers says that in most binary sequences of length 푛, the number of 0’s is close to the numer of 1’s. We prove this law of probability theory by constructing a ML-test 푑(푥) taking large values on sequences in which the number of 0’s is far from the number of 1’s. Instead of requiring (2.1.1) we require a somewhat stronger property of 푑(푥): for all 푛, ∑︁ 푃(푥)2푑 (푥) ≤ 1. (2.1.3) 푥 ∈B푛 From this, the inequality (2.1.1) follows by the following well-known inequality called the Markov Inequality which says the following. Let 푃 be any probability distribution, 푓 any nonnegative function, with expected value Í 퐸푃 ( 푓 ) = 푥 푃(푥) 푓 (푥). For all 휆 ≥ 0 we have ∑︁ { 푃(푥) : 푓 (푥) > 휆퐸푃 ( 푓 ) } ≤ 휆. (2.1.4)

For any string 푥 ∈ S, and natural number 푖, let 푁 (푖 | 푥) denote the number of occurrences of 푖 in 푥. For a binary string 푥 of length 푛 deﬁne 푝푥 = 푁 (1 | 푥)/푛, and

푁 (1 | 푦) 푁 (0 | 푦) 푃푥 ( 푦) = 푝푥 (1 − 푝푥)

푑(푥) = log 푃푥 (푥) + 푛 − log(푛 + 1).

We show that 푑(푥) is a ML-test. It is obviously computable. We prove (2.1.3). We have

∑︁ 푑 (푥) ∑︁ −푛 푛 1 1 ∑︁ 푃(푥)2 = 2 2 푃푥 (푥) = 푃푥 (푥) 푛 + 1 푛 + 1 푥 푥 푥 푛 푛 1 ∑︁ 푛 푘 푘 1 ∑︁ = ( )푘 (1 − )푛−푘 < 1 = 1. 푛 + 1 푘 푛 푛 푛 + 1 푘=0 푘=0

The test 푑(푥) expresses the (weak) law of large numbers in a rather strong version. We rewrite 푑(푥) as

푑(푥) = 푛(1 − ℎ( 푝푥)) − log(푛 + 1) where ℎ( 푝) = −푝 log 푝 − (1 − 푝) log(1 − 푝). The entropy function ℎ( 푝) achieves its maximum 1 at 푝 = 1/2. Therefore the test 푑(푥) tells us that the probability of sequences 푥 with 푝푥 < 푝 < 1/2 for some constant 푝 is bounded by the

39 2. Randomness exponentially decreasing quantitiy (푛 + 1)2−푛(1−ℎ( 푝)) . We also see that since for some constant 푐 we have

1 − ℎ( 푝) > 푐( 푝 − 1/2)2, therefore if the diﬀerence | 푝푥 − 1/2| is much larger than √︂ log 푛 푐푛 then the sequence 푥 is “not random”, and hence, by Theorem 2.1.1 its complexity is signiﬁcantly smaller than 푛. y

2.2 Computable distributions

Let us generalize the notion of randomness to arbitrary discrete computable probability distributions.

2.2.1 Two kinds of test Let 푃(푥) be a probability distribution over some discrete countable space Ω, which we will identify for simplicity with the set S of ﬁnite strings of natural numbers. Thus, 푃 is a nonnegative function with ∑︁ 푃(푥) = 1. 푥

Later we will consider probability distributions 푃 over the space NN, and they will also be characterized by a nonnegative function 푃(푥) over S. However, the above condition will be replaced by a different one. We assume 푃 to be computable, with some Gödel number 푒. We want to define a test 푑(푥) of randomness with respect to 푃. It will measure how justified is the assumption that 푥 is the outcome of an experiment with distribution 푃. Definition 2.2.1 A function 푑 : S → R is an integrable test, or expectation- bounded test of randomness with respect to 푃 if it is lower semicomputable and satisfies the condition ∑︁ 푃(푥)2푑 (푥) ≤ 1. (2.2.1) 푥 It is universal if it dominates all other integrable tests to within an additive constant. y

40 2.2. Computable distributions

(A similar terminology is used in [36] in order to distinguish tests satisfying condition (2.2.1) from Martin-Löf tests.) Remark 2.2.2 We must allow 푑(푥) to take negative values, even if only, say values ≥ −1, since otherwise condition (2.2.1) could only be satisfied with 푑(푥) = 0 for all 푥 with 푃(푥) > 0. y Proposition 2.2.3 Let 푐 = log(2휋2/6). If a function 푑(푥) satisfies (2.2.1) then it satisfies (2.1.1). If 푑(푥) satisfies (2.1.1) then 푑 − 2 log 푑 − 푐 satisfies (2.2.1). Proof. The first statement follows by Markov’s Inequality (2.1.4). The second one can be checked by immediate computation. Thus, condition 2.2.1 is nearly equivalent to condition (2.1.1): to each test 푑 according to one of them, there is a test 푑0 asymptotically equal to 푑 satisfying the other. But condition (2.2.1) has the advantage of being just one inequality instead of the infinitely many ones.

2.2.2 Randomness via complexity We want to find a universal integrable test. Let us introduce the function 푤(푥) = 푃(푥)2푑 (푥) . The above conditions on 푑(푥) imply that 푤(푥) is semicomputable Í from below and satisfies 푥 푤(푥) ≤ 1: so, it is a constructive semimeasure. With the universal constructive semimeasure of Theorem 1.6.3, we also obtain a universal test for any computable probability distribution 푃. Definition 2.2.4 For an arbitrary measure 푃 over a discrete space Ω, let us denote m(푥) d푃 (푥) = log = − log 푃(푥) − 퐾(푥). (2.2.2) 푃(푥) y The following theorem shows that randomness can be tested by checking how close is the complexity 퐾(푥) to its upper bound − log 푃(푥).

Theorem 2.2.1 The function d푃 (푥) = − log 푃(푥) − 퐾(푥) is a universal integrable test for any ﬁxed computable probability distribution 푃. More exactly, it is lower semicomputable, satisﬁes (2.2.1) and for all integrable tests 푑(푥) for 푃, we have

+ 푑(푥) < d푃 (푥) + 퐾(푑) + 퐾(푃).

Proof. Let 푑(푥) be an integrable test for 푃. Then 휈(푥) = 2푑 (푥) 푃(푥) is a construc- + tive semimeasure, and it has a self-delimiting program of length < 퐾(푑) + 퐾(푃).

41 2. Randomness

∗ It follows (using the definition (1.6.6)) that 푚(휈) < 2퐾 (푑)+퐾 (푃) ; hence inequality (1.6.7) gives ∗ 휈(푥)2퐾 (푑)+퐾 (푃) < m(푥). Taking logarithms finishes the proof. The simple form of our universal test suggests remarkable interpretations. It says that the outcome 푥 is random with respect to the distribution 푃, if the latter assigns to 푥 a large enough probability—however, not in absolute terms, only relatively to the universal semimeasure m(푥). The relativization is essential, since otherwise we could not distinguish between random and nonrandom outcomes for the uniform distribution 푃푛 defined in Section 2.1. For any (not necessarily computable) distribution 푃, the 푃-expected value of the function m(푥)/푃(푥) is at most 1, therefore the relation

m(푥) ≤ 푘푃(푥) holds with probability not smaller than 1 − 1/푘. If 푃 is computable then the inequality ∗ m(푃)푃(푥) < m(푥) holds for all 푥. Therefore the following applications are obtained. • If we assume 푥 to be the outcome of an experiment with some simple computable probability distribution 푃 then m(푥) is a good estimate of 푃(푥). The goodness depends on how simple 푃 is to define and how random 푥 is with respect to 푃: how justified is the assumption. Of course, we cannot compute m(푥) but it is nevertheless well defined. • If we trust that a given object 푥 is random with respect to a given distribution 푃 then we can use 푃(푥) as an estimate of m(푥). The degree of approximation depends on the same two factors. Let us illustrate the behavior of the universal semimeasure m(푥) when 푥 runs over the set of natural numbers. We define the semimeasures 푣 and 푤 as follows. 푘 Let 푣(푛) = 푐/푛2 for an appropriate constant 푐, let 푤(푛) = 푣(푥) if 푛 = 22 , and 0 otherwise. Then m(푛) dominates both 푣(푛) and 푤(푛). The function m(푛) dominates 1/푛 log2 푛, but jumps at many places higher than that. Indeed, it is easy to see that m(푛) converges to 0 slower than any positive computable function converging to 0: in particular, it is not computable. Hence it cannot Í be a measure, that is 푥 m(푥) < 1. We feel that we can make m(푥) “large” whenever 푥 is “simple”.

42 2.2. Computable distributions

We cannot compare the universal test defined in the present section directly with the one defined in Section 2.1. There, we investigated tests for a whole 푛 family 푃푛 of distributions, where 푃푛 is the uniform distribution on the set B . That the domain of these distributions is a different one for each 푛 is not essential since we can identify the set B푛 with the set {2푛,..., 2푛+1 − 1}. Let us rewrite the function 푑0 (푥) defined in (2.1.2) as

푑0 (푥) = 푛 − 퐶(푥 | 푛) for 푥 ∈ B푛, and set it ∞ for 푥 outside B푛. Now the similarity to the expression

d푃 (푥) = − log 푃(푥) − log m(푥) is striking because 푛 = log 푃푛 (푥). Together with the remark made in the previous paragraph, this observation suggests that the value of − log m(푥) is close to the complexity 퐶(푥).

2.2.3 Conservation of randomness The idea of conservation of randomness has been expressed first by Levin, who has published some papers culminating in [33] developing a general theory. In this section, we address the issue in its simplest form only. Suppose that we want to talk about the randomness of integers. One and the same integer 푥 can be represented in decimal, binary, or in some other notation: this way, every time a different string will represent the same number. Still, the form of the representation should not affect the question of randomness of 푥 significantly. How to express this formally? Let 푃 be a computable measure and let 푓 : S → S be a computable function. We expect that under certain conditions, the randomness of 푥 should imply the randomness of 푓 (푥)—but, for which distribution? For example, if 푥 is a binary string and 푓 (푥) is the decimal notation for the number expressed by 1푥 then we expect 푓 (푥) to be random not with respect to 푃, but with respect to the measure 푓 ∗푃 that is the image of 푃 under 푓 . This measure is defined by the formula ∑︁ ( 푓 ∗푃)( 푦) = 푃( 푓 −1 ( 푦)) = 푃(푥). (2.2.3) 푓 (푥)=푦

Proposition 2.2.5 If 푓 is a computable function and 푃 is a computable measure then 푓 ∗푃 is a computable measure.

43 2. Randomness

Proof. It is sufficient to show that 푓 ∗푃 is lower semicomputable, since we have seen that a lower semicomputable measure is computable. But, the semicomputability can be seen immediately from the definition. Let us see that randomness is indeed preserved for the image. Similarly to (1.6.6), let us define for an arbitrary computable function 푓 : ∑︁ m( 푓 ) = {m( 푝) : program 푝 computes 푓 }. (2.2.4)

Theorem 2.2.2 Let 푓 be a computable function. Between the randomness test for 푃 and the test for the image of 푃, the following relation holds, for all 푥:

+ d 푓 ∗푃 ( 푓 (푥)) < d푃 (푥) + 퐾( 푓 ) + 퐾(푃). (2.2.5)

Proof. Let us denote the function on the left-hand side of (2.2.5) by

m( 푓 (푥)) 푑푃 (푥) = d 푓 ∗푃 ( 푓 (푥)) = log . ( 푓 ∗푃)( 푓 (푥))

It is lower semicomputable, with the help of a program of length 퐾( 푓 ) + 퐾(푃), by its very deﬁnition. Let us check that it is an integrable test, so it satisﬁes the inequality (2.2.1). We have

∑︁ ( ) ∑︁ ∗ ∗ ( ) ∑︁ 푃(푥)2푑푃 푥 = ( 푓 푃)( 푦)2d 푓 푃 푦 = m( 푦) ≤ 1. 푥 푦 푦

+ Hence 푑푃 (푥) is a test, and hence it is < d푃 (푥)+퐾( 푓 )+퐾(푃), by the universality of the test d푃 (푥). (Strictly speaking, we must modify the proof of Theorem 2.2.1 and see that a program of length 퐾( 푓 ) + 퐾(푃) is also suﬃcient to deﬁne the semimeasure 푃(푥)푑(푥 | 푃).) The appearance of the term 퐾( 푓 ) on the right-hand side is understandable: using some very complex function 푓 , we can certainly turn any string into any other one, also a random string into a much less random one. On the other hand, the term 퐾(푃) appears only due to the imperfection of our concepts. It will disappear in the more advanced theory presented in later sections, where we develop uniform tests. Example 2.2.6 For each string 푥 of length 푛 ≥ 1, let

2−푛 푃(푥) = . 푛(푛 + 1)

44 2.2. Computable distributions

This distribution is uniform on strings of equal length. Let 푓 (푥) be the function that erases the first 푘 bits. Then for 푛 ≥ 1 and for any string 푥 of length 푛 we have 2−푛 ( 푓 ∗푃)(푥) = . (푛 + 푘)(푛 + 푘 + 1) (The empty string has the rest of the weight of 푓 ∗푃.) The new distribution is still uniform on strings of equal length, though the total probability of strings of length 푛 has changed somewhat. We certainly expect randomnes conservation here: after erasing the first 푘 bits of a random string of length 푛 + 푘, we should still get a random string of length 푛. y The computable function 푓 applied to a string 푥 can be viewed as some kind of transformation 푥 ↦→ 푓 (푥). As we have seen, it does not make 푥 less random. Suppose now that we introduce randomization into the transformation process itself: for example, the machine computing 푓 (푥) can also toss some coins. We want to say that randomness is also conserved under such more general transformations, but first of all, how to express such a transformation mathematically? We will describe it by a “matrix”: a computable probability transition function 푇 (푥, 푦) ≥ 0 with the property that ∑︁ 푇 (푥, 푦) = 1. 푦

Now, the image of the distribution 푃 under this transformation can be written as 푇∗푃, and deﬁned as follows: ∑︁ (푇∗푃)( 푦) = 푃(푥)푇 (푥, 푦). 푥 How to express now randomness conservation? It is certainly not true that every possible outcome 푦 is as random with respect to 푇∗푃 as 푥 is with respect to 푃. We can only expect that for each 푥, the conditional probability (in terms of the transition 푇 (푥, 푦)) of those 푦 whose non-randomness is larger than that of 푥, is small. This will indeed be expressed by the corollary below. To get there, we ∗ ( ) upperbound the 푇 (푥, ·)-expected value of 2d푇 푃 푦 . Let us go over to exponential notation: Deﬁnition 2.2.7 Denote m(푥) d푃 (푥) t푃 (푥) = 2 = . (2.2.6) 푃(푥) y

45 2. Randomness

Theorem 2.2.3 We have ∑︁ + d푇∗푃 ( 푦) log 푇 (푥, 푦)2 < d푃 (푥) + 퐾(푇) + 퐾(푃). (2.2.7) 푦

Proof. Let us denote the function on the left-hand side of (2.2.7) by 푡푃 (푥). It is lower semicomputable by its very construction, using a program of length + Í < 퐾(푇) + 퐾(푃). Let us check that it satisﬁes 푥 푃(푥)푡푃 (푥) ≤ 1 which, in this notation, corresponds to inequality (2.2.1). We have ∑︁ ∑︁ ∑︁ 푃(푥)푡푃 (푥) = 푃(푥) 푇 (푥, 푦)푡푇∗푃 ( 푦) 푥 푥 푦 ∑︁ ∑︁ m( 푦) ∑︁ = 푃(푥) 푇 (푥, 푦) = m( 푦) ≤ 1. (푇∗푃)( 푦) 푥 푦 푦

+ It follows that 푑푃 (푥) = log 푡푃 (푥) is an integrable test and hence 푑푃 (푥) < d푃 (푥) + 퐾(푇) + 퐾(푃). (See the remark at the end of the proof of Theorem 2.2.2.)

Corollary 2.2.8 There is a constant 푐0 such that for every integer 푘 ≥ 0, for all 푥 we have

∑︁ −푘+푐0 {푇 (푥, 푦) : d푇∗푃 ( 푦) − d푃 (푥) > 푘 + 퐾(푇) + 퐾(푃) } ≤ 2 . Proof. The theorem says

∑︁ ∗ 퐾 (푇)+퐾 (푃) 푇 (푥, 푦)t푇∗푃 ( 푦) < 푡푃 (푥)2 . 푦

∗ ( ) Thus, it upperbounds the expected value of the function 2d푇 푃) 푦 according to the distribution 푇 (푥, ·). Applying Markov’s inequality (2.1.4) to this function yields the desired result.

2.3 Inﬁnite sequences

These lecture notes treat the theory of randomness over continuous spaces mostly separately, starting in Section 4.1. But the case of computable measures over infinite sequences is particularly simple and appealing, so we give some of its results here, even if most follow from the more general results given later. In this section, we will fix a finite or countable alphabet Σ = {푠1, 푠2,...}, and consider probability distributions over the set

푋 = ΣN

46 2.3. Infinite sequences of infinite sequences with members in Σ. An alternative way of speaking of this is to consider a sequence of random variables 푋1, 푋2,..., where 푋푖 ∈ Σ, with a joint distribution. The two interesting extreme special cases are Σ = B = {0, 1}, giving the set of infinite 0-1 sequences, and Σ = N, giving the set sequences of natural numbers.

2.3.1 Null sets Our goal is here to illustrate the measure theory developed in Section A.2, through Subsection A.2.3, on the concrete example of the set of infinite sequences, and then to develop the theory of randomness in it. We will distinguish a few simple kinds of subsets of the set 푆 of sequences, those for which probability can be defined especially easily. Definition 2.3.1 • For string 푧 ∈ Σ∗, we will denote by 푧푋 the set of elements of 푋 with prefix 푧. Such sets will be called cylinder sets. • A subset of 푋 is open if it is the union of any number (not necessarily finite) of cylinder sets. It is called closed if its complement is open. • An open set is called constructive if it is the union of a recursively enumerable set of cylinder sets.

• A set is called 퐺훿 if it is the intersection of a sequence of open sets. It is called 퐹휎 if it is the union of a sequence of closed sets. • We say that a set 퐸 ⊆ 푋 is ﬁnitely determined if there is an 푛 and an E ⊆ Σ푛 such that Ø 퐸 = E푋 = 푠푋. 푠∈E

Let F be the class of all finitely determined subsets of 푋. • A class of subsets of 푋 is called an algebra if it is closed with respect to finite intersections and complements (and then of course, also with respect to finite unions). It is called a 휎-algebra (sigma-algebra) when it is also closed with respect to countable intersections (and then of course, also with respect to countable unions). y Example 2.3.2 An example open set that is not finitely determined, is the set 퐸 of all sequences that contain a substring 11. y

47 2. Randomness

The following observations are easy to prove. Proposition 2.3.3 a) Every open set 퐺 can be represented as the union of a sequence of disjoint cylinder sets. b) The set of open sets is closed with respect to finite intersection and arbitrarily large union. c) Each finitely determined set is both open and closed. d) The class F of finitely determined sets forms an algebra. Probability is generally defined as a nonnegative, monotonic function (a “measure”, see later) on some subsets of the event space (in our case the set 푋) that has a certain additivity property. We do not need immediately the complete definition of measures, but let us say what we mean by an additive set function. Definition 2.3.4 Consider a nonnegative, function 휇 is defined over some subsets of 푋. that is also monotonic, that is 퐴 ⊆ 퐵 implies 휇(퐴) ⊆ 휇(퐵). We say that Ð푛 it is additive if, whenever it is defined on the sets 퐸1, . . . , 퐸푛, and on 퐸 = 푖=1 퐸푖 and the sets 퐸푖 are mutually disjoint, then we have 휇(퐸) = 휇(퐸1) + · · · + 휇(퐸푛). We say that 휇 is countably additive, or 휎-additive (sigma-additive), if in ad- Ð∞ dition whenever it is defined on the sets 퐸1, 퐸2,..., and on 퐸 = 푖=1 퐸푖 and the Í∞ sets 퐸푖 are mutually disjoint, then we have 휇(퐸) = 푖=1 휇(퐸푖). y First we consider only measures defined on cylinder sets. N ∗ Definition 2.3.5 (Measure over Σ ) Let 휇 : Σ → R+ be a function assigning a nonnegative number to each string in Σ∗. We will call 휇 a measure if it satisfies the condition ∑︁ 휇(푥) = 휇(푥푠). (2.3.1) 푠∈Σ

We will take the liberty to also write

휇(푠푋) = 휇(푠) for all 푠 ∈ Σ∗, and 휇(∅) = 0. We call 휇 a probability measure if 휇(Λ) = 1 and correspondingly, 휇(푋) = 1. ∗ A measure is called computable if it is computable as a function 휇 : Σ → R+ according to Deﬁnition 1.5.2. y The following observation lets us extend measures.

48 2.3. Inﬁnite sequences

Lemma 2.3.6 Let 휇 be a measure deﬁned as above. If a cylinder set 푠푋 is the Í disjoint union 푥푋 = 푥1푋 ∪푥2푋 ∪... of cylinder sets, then we have 휇(푧) = 푖 휇(푥푖). Proof. Let us call an arbitrary 푦 ∈ Σ∗ good if ∑︁ 휇(푧푦) = 휇(푧푦푋 ∩ 푥푖 푋) 푖 holds. We have to show that the empty string Λ is good. It is easy to see that 푦 is good if 푧푦푋 ⊆ 푥푖 푋 for some 푖. Also, if 푦푠 is good for all 푠 ∈ Σ then 푦 is good. Now assume that Λ is not good. Then there is also an 푠1 ∈ Σ that is not good. But then there is also an 푠2 ∈ Σ such that 푠1푠2 is not good. And so on, we obtain an inﬁnite sequence 푠1푠2 ··· such that 푠1 ··· 푠푛 is Ð not good for any 푛. But then the sequence 푧푠1푠2 ··· is not contained in 푖 푥푖 푋, contrary to the assumption . Corollary 2.3.7 Two disjoint union representations of the same open set Ø Ø 푥푖 푋 = 푦푖 푋 푖 푖 Í Í imply 푖 휇(푥푖) = 푖 휇( 푦푖). Ð Proof. The set 푖 푥푖 푋 can also be written as Ø Ø 푥푖 푋 = 푥푖 푋 ∩ 푦푗 푋. 푖 푖, 푗

An element 푥푖 푋∩푦푗 푋 is nonempty only if one of the strings 푥푖, 푦푗 is a continuation of the other. Calling this string 푧푖 푗 we have 푥푖 푋 ∩ 푦푗 푋 = 푧푖 푗 푋. Now Lemma 2.3.6 Ð is applicable to the union 푥푖 푋 = 푗 (푥푖 푋 ∩ 푦푗 푋), giving ∑︁ 휇(푥푖) = 휇(푥푖 푋 ∩ 푦푗 푋), 푗 ∑︁ ∑︁ 휇(푥푖) = 휇(푥푖 푋 ∩ 푦푗 푋). 푖 푖, 푗 Í The right-hand side is also equal similarly to 푗 휇( 푦푗). The above corollary allows the following deﬁnition: Deﬁnition 2.3.8 For an open set 퐺 given as a disjoint union of cylinder sets Í 푥푖 푋 let 휇(퐺) = 푖 휇(푥푖). For a closed set 퐹 = 푋 \ 퐺 where 퐺 is open, let 휇(퐹) = 휇(푋) − 휇(퐺). y

49 2. Randomness

The following is easy to check. Proposition 2.3.9 The measure as defined on open sets is monotonic and countably additive. In particular, it is countably additive on the algebra of finitely determined sets. Before extending measures to a wider range of sets, consider an important special case. With infinite sequences, there are certain events that are not impossible, but still have probability 0. Definition 2.3.10 Given a measure 휇, a set 푁 ⊆ 푋 is called a null set with respect to 휇 if there is a sequence 퐺1, 퐺2,... of open sets with the property Ñ −푚 푁 ⊆ 푚 퐺푚, and 휇(퐺푚) ≤ 2 . We will say that the sequence 퐺푚 witnesses the fact that 푁 is a null set. y Examples 2.3.11 Let Σ = {0, 1}, let us define the measure 휆 by 휆(푥) = 2−푛 for all 푛 and for all 푥 ∈ Σ푛. 1. For every infinite sequence 휉 ∈ 푋, the one-element set {휉} is a null set with respect to 휆. Indeed, for each natural number 푛, let 퐻푛 = {휉 ∈ 푋 : 휉(0) = −푛−1 휉(0), 휉(1) = 휉(1), . . . , 휉(푛) = 휉(푛)}. Then 휆(퐻푛) = 2 , and {푠} = Ñ 푛 퐻푛. 2. For a less trivial example, consider the set 퐸 of those elements 푡 ∈ 푋 that are 1 in each positive even position, that is

퐸 = {푡 ∈ 푋 : 푡(2) = 1, 푡(4) = 1, 푡(6) = 1 ...}.

Then 퐸 is a null set. Indeed, for each natural number 푛, let 퐺푛 = {푡 ∈ 푋 : Ñ 푡(2) = 1, 푡(4) = 1, . . . , 푡(2푛) = 1}. This helps expressing 퐸 as 퐸 = 푛 퐺푛, −푛 where 휆(퐺푛) = 2 . y

Proposition 2.3.12 Let 푁1, 푁2,... be a sequence of null sets with respect to a Ð measure 휇. Their union 푁 = 푖 푁푖 is also a null set.

Proof. Let 퐺푖,1, 퐺푖,2,... be the inﬁnite sequence witnessing the fact that 푁푖 is a Ð∞ null set. Let 퐻푚 = 푖=1 퐺푖,푚+푖. Then the sequence 퐻푚 of open sets witnesses the fact that 푁 is a null set. We would like to extend our measure to null sets 푁 and say 휇(푁) = 0. The proposition we have just proved shows that such an extension would be countably additive on the null sets. But we do not need this extension for the moment,

50 2.3. Infinite sequences so we postpone it. Still, given a probability measure 푃 over 푋, it becomes meaningful to say that a certain property holds with probability 1. What this means is that the set of those sequences that do not have this property is a null set. Following the idea of Martin-Löf, we would like to call a sequence nonrandom, if it is contained in some simple null set; that is it has some easily definable property with probability 0. As we have seen in part1 of Example 2.3.11, it is important to insist on simplicity, otherwise (say with respect to the measure 휆) every sequence might be contained in a null set, namely the one-element set consisting of itself. But most of these sets are not defined simply at all. An example of a simply defined null set is given is part2 of Example 2.3.11. These reflections justify the following definition, in which “simple” is specified as “constructive”. Definition 2.3.13 Let 휇 be a computable measure. A set 푁 ⊆ 푋 is called a constructive null set if there is recursively enumerable set Γ ⊆ N × Σ∗ with the = { ( ) ∈ } = Ð property that denoting Γ푚 푥 : 푚, 푥 Γ and 퐺푚 푥 ∈Γ푚 푥푋 we have Ñ −푚 푁 ⊆ 푚 퐺푚, and 휇(퐺푚) ≤ 2 . y In words, the difference between the definition of null sets and constructive = Ð null sets is that the sets 퐺푚 푥 ∈Γ푚 푥푋 here are required to be constructive open, moreover, in such a way that from 푚 one can compute the program gen- erating 퐺푚. In even looser words, a set is a constructive null set if for any 휀 > 0 one can construct effectively a union of cylinder sets containing it, with total measure ≤ 휀. Now we are in a position to define random infinite sequences. Definition 2.3.14 (Random sequence) An infinite sequence 휉 is random with respect to a probability measure 푃 if and only if 휉 is not contained in any constructive null set with respect to 푃. y It is easy to see that the set of nonrandom sequences is a null set. Indeed, there is only a countable number or constructive null sets, so even their union is a null set. The following theorem strengthens this observation significantly. Theorem 2.3.1 Let us fix a computable probability measure 푃. The set of all nonrandom sequences is a constructive null set. Thus, there is a universal constructive null set, containing all other constructive null sets. A sequence is random when it does not belong to this set.

Proof of Theorem 2.3.1. The proof uses the projection technique that has appeared several times in this book, for example in proving the existence of a universal lower semicomputable semimeasure.

51 2. Randomness

We know that it is possible to list all recursively enumerable subsets of the set N × Σ∗ into a sequence, namely that there is a recursively enumerable set Δ ⊆ N2 × Σ∗ with the property that for every recursively enumerable set Γ ⊆ N × Σ∗ there is an 푒 with Γ = {(푚, 푥) : (푒, 푚, 푥) ∈ Δ}. We will write

Δ푒,푚 = {푥 : (푒, 푚, 푥) ∈ Δ}, Ø 퐷푒,푚 = 푥푋.

푥 ∈Δ푒,푚

We transform the set Δ into another set Δ0 with the following property. Í −푚+1 • For each 푒, 푚 we have (푒,푚,푥) ∈Δ0 휇(푥) ≤ 2 . Í −푚 • If for some 푒 we have (푒,푚,푥) ∈Δ 휇(푥) ≤ 2 for all 푚 then for all 푚 we have {푥 : (푒, 푚, 푥) ∈ Δ0 } = {푥 : (푒, 푚, 푥) ∈ Δ}. This transformation is routine, so we leave it to the reader. By the construction of Δ0, for every constructive null set 푁 there is an 푒 with

Ù 0 푁 ⊆ 퐷푒,푚. 푚

Deﬁne the recursively enumerable set

Γˆ = {(푚, 푥) : ∃ 푒 (푒, 푚 + 푒 + 2, 푥) ∈ Δ0 }.

ˆ Ð 0 Then 퐺푚 = 푒 퐷푒,푚+푒+2. For all 푚 we have ∑︁ ∑︁ ∑︁ ∑︁ 휇(푥) = 휇(푥) ≤ 2−푚−푒−1 = 2−푚. 푒 0 푒 푥 ∈Γˆ푚 (푒,푚+푒+2,푥) ∈Δ

This shows that Γˆ deﬁnes a constructive null set. Let Γ be any other recursively enumerable subset of N × Σ∗ that deﬁnes a constructive null set. Then there is 0 an 푒 such that for all 푚 we have 퐺푚 = 퐷푒,푚. The universality follows now from

Ù Ù 0 Ù 0 Ù ˆ 퐺푚 = 퐷푒,푚 ⊆ 퐷푒,푚+푒+2 ⊆ 퐺푚. 푚 푚 푚 푚

2.3.2 Probability space Now we are ready to extend measure to a much wider range of subsets of 푋.

52 2.3. Inﬁnite sequences

Definition 2.3.15 Elements of the smallest 휎-algebra A containing the cylinder sets of 푋 are called the Borel sets of 푋. The pair (푋, A) is an example of a measureable space, where the Borel sets are called the measureable sets. A nonnegative sigma-additive function 휇 over A is called a measure. It is called a probability measure if 휇(푋) = 1. If 휇 is fixed then the triple (푋, A, 휇) is called a measure space. If it is a probability measure then the space is called a probability space. y In the Appendix we cited a central theorem of measure theory, Caratheodory’s extension theorem. It implies that if a measure is defined on an algebra L in a sigma-additive way then it can be extended uniquely to the 휎-algebra generated by L, that is the smallest 휎-algebra containing L. We defined a measure 휇 over ∗ 푋 as a nonnegative function 휇 : Σ → R+ satisfying the equality (2.3.1). Then we defined 휇(푥푋) = 휇(푥), and further extended 휇 in a 휎-additive way to all elements of the algebra F. Now Caratheodory’s theorem allows us to extend it uniquely to all Borel sets, and thus to define a measureable space (푋, A, 휇). Of course, all null sets in A get measure 0. Open, closed and measureable sets can also be defined in the set of real numbers. Definition 2.3.16 A subset 퐺 ⊆ R is open if it is the union of a set of open intervals (푎푖, 푏푖). It is closed if its complement is open. The set B of Borel sets of B is defined as the smallest 휎-algebra containing all open sets. y The pair (R, B) is another example of a measureable space. Definition 2.3.17 (Lebesgue measure) Consider the set left-closed intervals of the line (including intervals of the form (−∞, 푎). Let L be the set of finite disjoint unions of such intervals. This is an algebra. We define the function 휆 over L as Ð Í follows: 휆( 푖 [푎푖, 푏푖)) = 푖 푏푖 − 푎푖. It is easy to check that this is a 휎-additive measure and therefore by Caratheodory’s theorem can be extended to the set B of all Borel sets. This function is called the Lebesgue measure over R, giving us the measureable space (R, B, 휆). y Finally, we can define the notion of a measureable function over 푋. Definition 2.3.18 (Measureable functions) A function 푓 : 푋 → R is called measureable if and only if 푓 −1 (퐸) ∈ A for all 퐸 ∈ B. y The following is easy to prove. Proposition 2.3.19 Function 푓 : 푋 → R is measureable if and only if all sets of

53 2. Randomness the form 푓 −1 ((푟, ∞)) = {푥 : 푓 (푥) > 푟} are measureable, where 푟 is a rational number.

2.3.3 Computability Measureable functions are quite general; it is worth introducing some more resticted kinds of function over the set of sequences. Definition 2.3.20 A function 푓 : 푋 → R is continuous if and only if for every 푥 ∈ 푋, for every 휀 > 0 there is a cylinder set 퐶 3 푥 such that | 푓 ( 푦) − 푓 (푥)| < 휀 for all 푦 ∈ 퐶. A function 푓 : 푋 → R is lower semicontinuous if for every 푟 ∈ R the set {푥 ∈ 푋 : 푓 (푥) > 푟} is open. It is upper semicontinuous if − 푓 is lower semicontinuous. y The following is easy to verify. Proposition 2.3.21 A function 푓 : 푋 → R is continuous if and only if it is both upper and lower semicontinuous. Definition 2.3.22 A function 푓 : 푋 → R is computable if for every open rational interval (푢, 푣) the set 푓 −1 ((푢, 푣)) is a constructive open set of 푋, uniformly in 푢, 푣. y Informally this means that if for the infinite sequence 휉 ∈ 푋 we have 푢 < 푓 (휉) < 푣 then from 푢, 푣 sooner or later we will find a prefix 푥 of 휉 with the property 푢 < 푓 (푥푋) < 푣. Definition 2.3.23 A function 푓 : 푋 → R is lower semicomputable if for every rational 푟 the set {푠 ∈ 푋 : 푓 (푠) > 푟} is a constructive open set of 푋, uniformly in 푟. It is upper semicomputable if − 푓 is lower semicomputable. y The following is easy to verify. Proposition 2.3.24 A function 푋 → R is computable if and only if it is both lower and upper semicomputable.

2.3.4 Integral The definition of integral over a measure space is given in the Appendix, in Sub- section A.2.3. Here, we give an exposition specialized to infinite sequences. Definition 2.3.25 A measurable function 푓 : 푋 → R is called a step function if its range is finite. The set of step functions will be called E.

54 2.3. Inﬁnite sequences

Given a step function 푓 which takes values 푥푖 on sets 퐴푖, and a finite measure 휇, we define ∫ ∫ ∑︁ 휇( 푓 ) = 휇 푓 = 푓 푑휇 = 푓 (푥)휇(푑푥) = 푥푖휇(퐴푖). 푖 y Proposition A.2.14, when specified to our situation here, says the following. Proposition 2.3.26 The functional 휇 defined above on step functions can be extended to the set E+ of monotonic limits of nonnegative elements of E, by continuity. The set E+ is the set of all nonnegative measurable functions. Now we extend the notion of integral to a wider class of functions. Definition 2.3.27 A measurable function 푓 is called integrable with respect to a finite measure 휇 if 휇| 푓 |+ < ∞ and 휇| 푓 |− < ∞. In this case, we define 휇 푓 = 휇| 푓 |+ − 휇| 푓 |−. y It is easy to see that the mapping 푓 ↦→ 휇 푓 is linear when 푓 runs through the set of measureable functions with 휇| 푓 | < ∞. The following is also easy to check. Proposition 2.3.28 Let 휇 be a computable measure. a) If 푓 is computable then a program to compute 휇 푓 can be found from the program to compute 푓 . b) If 푓 is lower semicomputable then a program to lower semicompute 휇 푓 can be found from a program to lower semicompute 푓 .

2.3.5 Randomness tests We can now define randomness tests similarly to Section 2.2. Definition 2.3.29 A function 푑 : 푋 → R is an integrable test, or expectation- bounded test of randomness with respect to the probability measure 푃 if it is lower semicomputable and satisfies the condition ∫ 2푑 (푥) 푃(푑푥) ≤ 1. (2.3.2)

It is called a Martin-Löf test, or probability-bounded test, if instead of the latter condition only the weaker one is satisﬁed saying 푃(푑(푥) > 푚) < 2−푚 for each positive integer 푚.

55 2. Randomness

It is universal if it dominates all other integrable tests to within an additive constant. Universal Martin-Löf tests are deﬁned in the same way. y Randomness tests and constructive null sets are closely related. Theorem 2.3.2 Let 푃 be a computable probability measure over the set 푆 = Σ∞. There is a correspondence between constructive null sets and randomness tests: a) For every randomness tests 푑, the set 푁 = {휉 : 푑(휉) = ∞} is a constructive null set. b) For every constructive null set 푁 there is a randomness test 푑 with 푁 = {휉 : 푑(휉) = ∞}.

Proof. Let 푑 be a randomness test: then for each 푘 the set 퐺푘 = {휉 : 푑(휉) > 푘} is a constructive open set, and by Markov’s inequality (wich is proved in the −푘 continuous case just as in the discrete case) we have 푃(퐺푘) ≤ 2 . The sets 퐺푘 witness that 푁 is a constructive null set. Ñ∞ Let 푁 be a constructive null set with 푁 ⊆ 푘=1 퐺푘, where 퐺푘 is a uniform −푘 sequence of constructive open sets with 푃(퐺푘) = 2 . Without loss of generality assume that the sequence 퐺푘 is decreasing. Then the function 푑(휉) = sup{푘 : −푘 휉 ∈ 퐺푘 } is lower semicomputable and satisﬁes 푃{휉 : 푑(휉) ≥ 푘} ≤ 2 , so it is a Martin-Löf test. Just as in Proposition 2.2.3, it is easy to check that 푑(푥) − 2 log 푑(푥) − 푐 is an integrable test for some constant 푐. Just as there is a universal constructive null set, there are universal randomness tests. Theorem 2.3.3 For a computable measure 푃, there is a universal integrable test d푃 (휉). More exactly, the function 휉 ↦→ d푃 (휉) is lower semicomputable, satis- ﬁes (2.2.1) and for all integrable tests 푑(휉) for 푃, we have

+ 푑(휉) < d푃 (휉) + 퐾(푑) + 퐾(푃).

The proof is similar to the proof Theorem 1.6.3 on the existence of a universal constructive semimeasure. It uses the projection technique and a weighted sum.

2.3.6 Randomness and complexity We hope for a result connecting complexity with randomness similar to Theo- rem 2.2.1. Somewhat surprisingly, there is indeed such a result. This is a sur- prise since in an inﬁnite sequence, arbitrarily large oscillations of any quantity depending on the preﬁxes are actually to be expected.

56 2.3. Inﬁnite sequences

Theorem 2.3.4 Let 푋 = ΣN be the set of inﬁnite sequences. For all computable measures 휇 over 푋, we have

+ d휇 (휉) = sup (− log 휇(휉1:푛) − 퐾(휉1:푛)). (2.3.3) 푛

Here, the constant in =+ depends on the computable measure 휇. Corollary 2.3.30 Let 휆 be the uniform distribution over the set of inﬁnite binary sequences. Then + d휆 (휉) = sup (푛 − 퐾(휉1:푛)). (2.3.4) 푛 In other words, an inﬁnite binary sequence is random (with respect to the uniform distribution) if and only if the complexity 퐾(휉1:푛) of its initial segments of length 푛 never decreases below 푛 by more than an additive constant.

+ Proof. To prove <, deﬁne the function

∑︁ m(푠 | 휇) ∑︁ m(휉1:푛 | 휇) m(휉1:푛 | 휇) 푓휇 (휉) = 1푠푋 (휉) = ≥ sup . 휇(푠) 휇(휉 ) 휇(휉 ) 푠 푛 1:푛 푛 1:푛

휉 The function 휉 ↦→ 푓휇 (휉) is lower semicomputable with 휇 푓휇 (휉) ≤ 1, and hence

+ + d휇 (휉) > log 푓 (휉) > sup (− log 휇(휉1:푛) − 퐾(휉1:푛 | 휇)). 푛

+ The proof of < reproduces the proof of Theorem 5.2 of [18]. Let us abbrevi- ate: 1푦 (휉) = 1푦푋 (휉). Since d푑(휉)e only takes integer values and is lower semicomputable, there are ∗ computable sequences 푦푖 ∈ Σ and 푘푖 ∈ N with 1 d푑 (휉) e 푘푖 ∑︁ 푘푖 2 = sup 2 1푦 (휉) ≥ 2 1푦 (휉) 푖 2 푖 푖 푖 with the property that if 푖 < 푗 and 1푦푖 (휉) = 1푦푗 (휉) = 1 then 푘푖 < 푘 푗. The inequality follows from the fact that for any ﬁnite sequence 푛1 < 푛2 < . . ., Í 푛 푗 ≤ 푛 푗 ( ) = Í 푘푖 푗 2 2 max 푗 2 . The function 훾 푦 푦푖=푦 2 is lower semicomputable. With it, we have

∑︁ 푘푖 ∑︁ 2 1푦푖 (휉) = 1푦 (휉)훾( 푦). 푖 푦 ∈Σ∗

57 2. Randomness

d푑e Í ∗ Since 휇2 ≤ 2, we have 푦 휇( 푦)훾( 푦) ≤ 2, hence 휇( 푦)훾( 푦) < m( 푦), that is ∗ 훾( 푦) < m( 푦)/휇( 푦). It follows that

푑 (휉) ∗ m( 푦) m(휉1:푛) 2 < sup 1푦 (휉) = sup . 푦 ∈Σ∗ 휇( 푦) 푛 휇(휉1:푛)

+ Taking logarithms, we obtain the < part of the theorem.

2.3.7 Universal semimeasure, algorithmic probability Universal semimeasures exist over infinite sequences just as in the discrete space. ∗ Definition 2.3.31 A function 휇 : Σ → R+ is a semimeasure if it satisfies ∑︁ 휇(푥) ≥ 휇(푥푠), 푠∈Σ 휇(Λ) ≤ 1.

If it is lower semicomputable we will call the semimeasure constructive. y These requirements imply, just as for measures, the following generalization to an arbitrary prefix-free set 퐴 ⊆ Σ∗: ∑︁ 휇(푥) ≤ 1. 푥 ∈퐴 Standard technique gives the following: Theorem 2.3.5 (Universal semimeasure) There is a universal constructive semimeasure 휇, that is a constructive semimeasure with the property that for every other constructive semimeasure 휈 there is a constant 푐휈 > 0 with 휇(푥) ≥ 푐휈휈(푥). Definition 2.3.32 Let us fix a universal constructive semimeasure over 푋 and denote it by 푀(푥). y There is a graphical way to represent a universal semimeasure, via monotonic Turing machines, which can be viewed a generalization of self-delimiting machines. Definition 2.3.33 A Turing machine T is monotonic if it has no input tape or output tape, but can ask repeatedly for input symbols and can emit repeatedly output symbols. The input is assumed to be an infinite binary string, but it is not assumed that T will ask for all of its symbols, even in infinite time. If the finite or infinite input string is 푝 = 푝1 푝2 ···, the (finite or infinite) output string is written as 푇 ( 푝) ∈ Σ∗ ∪ ΣN. y

58 2.3. Inﬁnite sequences

Just as we can generalize self-delimiting machines to obtain the notion of monotonic machines, we can generalize algorithmic probability, defined for a discrete space, to a version defined for infinite sequences. Imagine the monotonic Turing machine T computing the function 푇 ( 푝) as in the definition, and assume that its input symbols come from coin-tossing. Definition 2.3.34 (Algorithmic probability over infinite sequences) For a string ∗ 푥 ∈ Σ , let 푃푇 (푥) be the probability of 푇 (휋) w 푥, when 휋 = 휋1휋2 ··· is the infinite coin-tossing sequence. y We can compute

∑︁ −푙( 푝) 푃푇 (푥) = 2 , 푇 ( 푝) w푥,푝 minimal where “minimal” means that no prefix 푝0 of 푝 gives 푇 ( 푝0) w 푥. This expression shows that 푃푇 (푥) is lower semicomputable. It is also easy to check that it is a semimeasure, so we have a constructive semimeasure. The following theorem is obtained using a standard construction: Theorem 2.3.6 Every constructive semimeasure 휇(푥) can be represented as 휇(푥) = 푃푇 (푥) with the help of an appropriate a monotonic machine T. This theorem justifies the following. Definition 2.3.35 From now on we will call the universal semimeasure 푀(푥) also the algorithmic probability. A monotonic Turing machine T giving 푃푇 (푥) = 푀(푥) will be called an optimal machine. y We will see that − log 푀(푥) should also be considered a kind of description complexity: Definition 2.3.36 Denote 퐾푀(푥) = − log 푀(푥). y How does 퐾푀(푥) compare to 퐾(푥)? Proposition 2.3.37 We have the bounds + + 퐾푀(푥) < 퐾(푥) < 퐾푀(푥) + 퐾(푙(푥)).

+ Proof. To prove 퐾푀(푥) < 퐾(푥) deﬁne the lower semicomputable function ∑︁ 휇(푥) = sup m(푥 푦), 푆 푦 ∈푆

59 2. Randomness

By its deﬁnition this is a constructive semimeasure and it is ≥ m(푥). This implies ∗ + 푀(푥) > m(푥), and thus 퐾푀(푥) < 퐾(푥). For the other direction, deﬁne the following lower semicomputable function 휈 over Σ∗:

휈(푥) = m(푙(푥))· 푀(푥).

To show that it is a semimeasure compute: ∑︁ ∑︁ ∑︁ ∑︁ 휈(푥) = m(푛) 푀(푥) ≤ m(푛) ≤ 1. 푥 ∈Σ∗ 푛 푥 ∈Σ푛 푛

∗ + It follows that m(푥) > 휈(푥) and hence 퐾(푥) < 퐾푀(푥) + 퐾(푙(푥)).

2.3.8 Randomness via algorithmic probability Fix a computable measure 푃(푥) over the space 푋. Just as in the discrete case, the universal semimeasure gives rise to a Martin-Löf test. Deﬁnition 2.3.38 Denote

0 푀(휉1:푛) d푃 (휉) = log sup . 푛 푃(휉1:푛) y ∗ By Proposition 2.3.37, this function is is > d푃 (휉), defined in Theorem 2.3.3. It can be shown that it is no longer expectation-bounded. But the theorem below shows that it is still a Martin-Löf (probability-bounded) test, so it defines the same infinite random sequences. 0 Theorem 2.3.7 The function d푃 (휉) is a Martin-Löf test. Proof. Lower semicomputability follows from the form of definition. By standard methods, produce a list of finite strings 푦1, 푦2,... with: 푚 a) 푀( 푦푖)/푃( 푦푖) > 2

b) The cylinder sets 푦푖 푋 cover of 퐺푚, and are disjoint (thus the set {푦1, 푦2,... } is preﬁx-free). We have

∑︁ −푚 ∑︁ −푚 푃(퐺푚) = 푃( 푦푖) < 2 푀( 푦푖) ≤ 2 . 푖 푖

60 2.3. Inﬁnite sequences

Just as in the discrete case, the universality of 푀(푥) implies

+ 퐾푀(푥) < − log 푃(푥) + 퐾(푃).

On the other hand,

sup − log 푃(휉1:푛) − 퐾푀(휉1:푛) 푛 is a randomness test. So for random 휉, the value 퐾푀(휉1:푛) remains within a constant of − log 푃(푥): it does not oscillate much.

3 Information

3.1 Information-theoretic relations

In this subsection, we use some material from [23].

3.1.1 The information-theoretic identity Information-theoretic entropy is related to complexity in both a formal and a more substantial way.

Classical entropy and classical coding Entropy is used in information theory in order to characterize the compressibility of a statistical source. Deﬁnition 3.1.1 The entropy of a discrete probability distribution 푃 is deﬁned as ∑︁ H(푃) = − 푃(푥) log 푃(푥). 푥 y A simple argument shows that this quantity is non-negative. It is instructive to recall a more general inequality, taken from [13], implying it: Í Í Theorem 3.1.1 Let 푎푖, 푏푖 > 0, 푎 = 푖 푎푖, 푏 = 푖 푏푖. Then we have

∑︁ 푎푖 푎 푎푖 log ≥ 푎 log , (3.1.1) 푏 푏 푖 푖 Í Í with equality only if 푎푖/푏푖 is constant. In particular, if 푖 푎푖 = 1 and 푖 푏푖 ≤ 1 Í 푎푖 then we have 푎푖 log ≥ 0. 푖 푏푖 Proof. Exercise, an application of Jensen’s inequality to the concave function log 푥.

63 3. Information

Í If 푃 is a probability distribution and 푄(푥) ≥ 0, 푥 푄(푥) ≤ 1 then this inequality implies ∑︁ 퐾(푃) ≤ 푃(푥) log 푄(푥). 푥 We can interpret this inequality as follows. We have seen in Subsection 1.6.3 that if 푥 ↦→ 푧푥 is a preﬁx code, mapping the objects 푥 into binary strings 푧푥 that are Í −푙(푧푥 ) −푙(푧푥 ) not preﬁxes of each other then 푥 2 ≤ 1. Thus, substituting 푄(푥) = 2 above we obtain Shannon’s result:

Theorem 3.1.2 (Shannon) If 푥 ↦→ 푧푥 is a preﬁx code then for its expected codeword length we have the lower bound: ∑︁ 푃(푥)푙(푧푥) ≥ 퐾(푃). 푥

Entropy as expected complexity

Applying Shannon’s theorem 3.1.2 to the code obtained by taking 푧푥 as the shortest self-delimiting description of 푥, we obtain the inequality ∑︁ 푃(푥)퐾(푥) ≥ H(푃) 푥

∗ On the other hand, since m(푥) is a universal semimeasure, we have m(푥) > + 푃(푥): more precisely, 퐾(푥) < − log 푃(푥) + 퐾(푃), leading to the following theorem. Theorem 3.1.3 For a computable distribution 푃 we have

∑︁ + H(푃) ≤ 푃(푥)퐾(푥) < H(푃) + 퐾(푃). (3.1.2) 푥 Note that H(푃) is the entropy of the distribution 푃 while 퐾(푃) is just the description complexity of the function 푥 ↦→ 푃(푥), it has nothing to do directly with the magnitudes of the values 푃(푥) for each 푥 as real numbers. These two relations give + ∑︁ 퐾(푃) = 푃(푥)퐾(푥) − 퐾(푃), 푥 the entropy is within an additive constant equal to the expected complexity. Our intended interpretation of 퐾(푥) as information content of the individual object 푥 is thus supported by a tight quantitative relationship to Shannon’s statistical concept.

64 3.1. Information-theoretic relations

Identities, inequalities The statistical entropy obeys meaningful identities which immediately follow from the definition of conditional entropy. Definition 3.1.2 Let 푋 and 푌 be two discrete random variables with a joint distribution. The conditional entropy of 푌 with respect to 푋 is defined as ∑︁ ∑︁ H(푌 | 푋) = − 푃[푋 = 푥] 푃[푌 = 푦 | 푋 = 푥] log 푃[푌 = 푦 | 푋 = 푥]. 푥 푦 The joint entropy of 푋 and 푌 is defined as the entropy of the pair (푋, 푌). y The following additivity property is then verifiable by simple calculation:

H(푋, 푌) = H(푋) + H(푌 | 푋). (3.1.3)

Its meaning is that the amount of information in the pair (푋, 푌) is equal to the amount of information in 푋 plus the amount of our residual uncertainty about 푌 once the value of 푋 is known. Though intuitively expected, the following identity needs proof:

H(푌 | 푋) ≤ H(푌).

We will prove it after we define the difference below. Definition 3.1.3 The information in 푋 about 푌 is defined by

I(푋 : 푌) = H(푌) − H(푌 | 푋).

y This is the amount by which our knowledge of 푋 decreases our uncertainty about 푌. The identity below comes from equation (3.1.3).

I(푋 : 푌) = I(푌 : 푋) = H(푋) + H(푌) − H(푋, 푌)

The meaning of the symmetry relation is that the quantity of information in 푌 about 푋 is equal to the quantity of information in 푋 about 푌. Despite its simple derivation, this fact is not very intuitive; especially since only the quantities are equal, the actual contents are generally not (see [22]). We also call I(푋 : 푌) the mutual information of 푋 and 푌. We can also write ∑︁ 푃[푋 = 푥, 푌 = 푦] I(푋 : 푌) = 푃[푋 = 푥, 푌 = 푦] log . (3.1.4) 푃[푋 = 푥]푃[푌 = 푦] 푥,푦

65 3. Information

Proposition 3.1.4 The information I(푋 : 푌) is nonnegative and is 0 only if 푋, 푌 are independent.

Proof. Apply Theorem 3.1.1 to (3.1.4). This is 0 only if 푃[푋 = 푥, 푌 = 푦] = 푃[푋 = 푥]푃[푌 = 푦] for all 푥, 푦.

The algorithmic addition theorem For the notion of algorithmic entropy 퐾(푥) defined in Section 1.6, it is a serious test of fitness to ask whether it has an additivity property analogous to the identity (3.1.3). Such an identity would express some deeper relationship than the trivial identity (3.1.3) since the conditional complexity is defined using the notion of conditional computation, and not by an algebraic formula involving the unconditional definition. A literal translation of the identity (3.1.3) turns out to be true for the function 퐾(푥) as well as for Kolmogorov’s complexity 퐶(푥) with good approximation. But the exact additivity property of algorithmic entropy is subtler. Theorem 3.1.4 (Addition) We have

퐾(푥, 푦) =+ 퐾(푥) + 퐾( 푦 | 푥, 퐾(푥)). (3.1.5)

+ Proof. We first prove <. Let 퐹 be the following s.d. interpreter. The machine computing 퐹( 푝) tries to parse 푝 into 푝 = 푢푣 so that 푇 (푢) is defined, then outputs the pair (푇 (푢), 푇 (푣, h푇 (푢), 푙(푢)i)). If 푢 is a shortest description of 푥 and 푣 a shortest description of 푦 given the pair (푥, 퐾(푥)) then the output of 퐹 is the pair (푥, 푦). + Now we prove >. Let 푐 be a constant (existing in force of Lemma 1.7.3) such Í −퐾 (푥,푦) −퐾 (푥)+푐 that for all 푥 we have 푦 2 ≤ 2 . Let us define the semicomputable function 푓 (푧, 푦) by 푓 ((푥, 푘), 푦) = 2푘−퐾 (푥,푦)−푐. Í Then for 푘 = 퐾(푥) we have 푦 푓 (푧, 푦) ≤ 1, hence the generalization (1.6.13) + of the coding theorem implies 퐾( 푦 | 푥, 퐾(푥)) < 퐾(푥, 푦) − 퐾(푥). Recall the definition of 푥∗ as the first shortest description of 푥. Corollary 3.1.5 We have

퐾(푥∗, 푦) =+ 퐾(푥, 푦). (3.1.6)

66 3.1. Information-theoretic relations

The Addition Theorem implies

+ + 퐾(푥, 푦) < 퐾(푥) + 퐾( 푦 | 푥) < 퐾(푥) + 퐾( 푦) while these relations do not hold for Kolmogorov’s complexity 퐶(푥), as pointed out in the discussion of Theorem 1.4.2. For an ordinary interpreter, extra information is needed to separate the descriptions of 푥 and 푦. The self-delimiting interpreter accepts only a description that provides the neccesary information to determine its end. The term 퐾(푥) cannot be omitted from the condition in 퐾( 푦 | 푥, 퐾(푥)) in the identity (3.1.5). Indeed, it follows from equation (1.7.2) that with 푦 = 퐾(푥), we have 퐾(푥, 푦) − 퐾(푥) =+ 0. But 퐾(퐾(푥) | 푥) can be quite large, as shown in the following theorem given here without proof. Theorem 3.1.5 (see [17]) There is a constant 푐 such that for all 푛 there is a binary string 푥 of length 푛 with

퐾(퐾(푥) | 푥) ≥ log 푛 − log log 푛 − 푐.

We will prove this theorem in Section 3.3. For the study of information relations, let us introduce yet another notation. Deﬁnition 3.1.6 For any functions 푓 , 푔, ℎ with values in S, let

푓 푔 mod ℎ mean that there is a constant 푐 such that 퐾( 푓 (푥) | 푔(푥), ℎ(푥)) ≤ 푐 for all 푥. We will omit modℎ if ℎ is not in the condition. The sign will mean inequality in both directions. y The meaning of 푥 푦 is that the objects 푥 and 푦 hold essentially the same information. Theorem 1.4.1 implies that if 푓 푔 then we can replace 푓 by 푔 wherever it occurs as an argument of the functions 퐻 or 퐾. As an illustration of this deﬁnition, let us prove the relation

푥∗ h푥, 퐾(푥)i which implies 퐾( 푦 | 푥, 퐾(푥)) =+ 퐾( 푦 | 푥∗), and hence

퐾(푥, 푦) − 퐾(푥) =+ 퐾( 푦 | 푥∗). (3.1.7)

The direction holds because we can compute 푥 = 푇 (푥∗) and 퐾(푥) = 푙(푥∗) from 푥∗. On the other hand, if we have 푥 and 퐾(푥) then we can enumerate the

67 3. Information set 퐸(푥) of all shortest descriptions of 푥. By the corollary of Theorem 1.7.1, the number of elements of this set is bounded by a constant. The estimate (1.6.4) implies 퐾(푥∗ | 푥, 퐾(푥)) =+ 0. Though the string 푥∗ and the pair (푥, 퐾(푥)) contain the same information, they are not equivalent under some stronger criteria on the decoder. To obtain 푥 from the shortest description may take extremely long time. But even if this time 푡 is acceptable, to ﬁnd any description of length 퐾(푥) for 푥 might require about 푡2퐾 (푥) steps of search. The Addition Theorem can be made formally analogous to its information- theoretic couterpart using Chaitin’s notational trick. Deﬁnition 3.1.7 Let 퐾∗( 푦 | 푥) = 퐾( 푦 | 푥∗). y Of course, 퐾∗(푥) = 퐾(푥). Now we can formulate the relation (3.1.7) as

퐾∗(푥, 푦) =+ 퐾∗(푥) + 퐾∗( 푦 | 푥).

However, we prefer to use the function 퐾( 푦 | 푥) since the function 퐾∗( 푦 | 푥) is not semicomputable. Indeed, if there was a function ℎ(푥, 푦) upper semicomputable in h푥, 푦i such that 퐾(푥, 푦) − 퐾(푥) =+ ℎ(푥, 푦) then by the argument of the proof + of the Addition Theorem, we would have 퐾( 푦 | 푥) < ℎ(푥, 푦) which is not true, according to remark after the proof of the Addition Theorem. Deﬁnition 3.1.8 We deﬁne the algorithmic mutual information with the same formula as classical information:

퐼(푥 : 푦) = 퐾( 푦) − 퐾( 푦 | 푥). y Since the quantity 퐼(푥 : 푦) is not quite equal to 퐾(푥) − 퐾(푥 | 푦) (in fact, Levin showed (see [17]) that they are not even asymptotically equal), we prefer to deﬁne information also by a symmetrical formula. Deﬁnition 3.1.9 The mutual information of two objects 푥 and 푦 is

퐼∗(푥 : 푦) = 퐾(푥) + 퐾( 푦) − 퐾(푥, 푦). y The Addition Theorem implies the identity

퐼∗(푥 : 푦) =+ 퐼(푥∗ : 푦) =+ 퐼( 푦∗ : 푥).

Let us show that classical information is indeed the expected value of algorithmic information, for a computable probability distribution 푝:

68 3.1. Information-theoretic relations

Lemma 3.1.10 Given a computable joint probability mass distribution 푝(푥, 푦) over (푥, 푦) we have

+ ∑︁ ∑︁ I(푋 : 푌) − 퐾( 푝) < 푝(푥, 푦)퐼∗(푥 : 푦) (3.1.8) 푥 푦 + < I(푋 : 푌) + 2퐾( 푝), where 퐾( 푝) is the length of the shortest preﬁx-free program that computes 푝(푥, 푦) from input (푥, 푦). Proof. We have

∑︁ + ∑︁ 푝(푥, 푦)퐼∗(푥 : 푦) = 푝(푥, 푦)[퐾(푥) + 퐾( 푦) − 퐾(푥, 푦)]. 푥,푦 푥,푦 Í Í Deﬁne 푦 푝(푥, 푦) = 푝1 (푥) and 푥 푝(푥, 푦) = 푝2 ( 푦) to obtain

∑︁ + ∑︁ ∑︁ ∑︁ 푝(푥, 푦)퐼(푥 : 푦) = 푝1 (푥)퐾(푥) + 푝2 ( 푦)퐾( 푦) − 푝(푥, 푦)퐾(푥, 푦). 푥,푦 푥 푦 푥,푦

The distributions 푝푖 (푖 = 1, 2) are computable. We have seen in (3.1.2) that + Í + H(푞) < 푥 푞(푥)퐾(푥) < H(푞) + 퐾(푞). + Í + + Hence, H( 푝푖) < 푥 푝푖 (푥)퐾(푥) < H( 푝푖) + 퐾( 푝푖).(푖 = 1, 2), and H( 푝) < Í + 푥,푦 푝(푥, 푦)퐾(푥, 푦) < H( 푝)+퐾( 푝). On the other hand, the probabilistic mutual information is expressed in the entropies by I(푋 : 푌) = H( 푝1) + H( 푝2) − H( 푝). + By construction of the 푞푖’s above, we have 퐾( 푝1), 퐾( 푝2) < 퐾( 푝). Since the complexities are positive, substitution establishes the lemma. Remark 3.1.11 The information 퐼∗(푥 : 푦) can be written as m(푥, 푦) 퐼∗(푥 : 푦) =+ log . m(푥)m( 푦)

∗ Formally, 퐼 (푥 : 푦) looks like dm×m ((푥, 푦) | m × m) with the function d· (·) introduced in (2.2.2). Thus, it looks like 퐼∗(푥 : 푦) measures the deficiency of randomness with respect to the distribution m × m. The latter distribution expresses our “hypothesis” that 푥, 푦 are “independent” from each other. There is a serious technical problem with this interpretation: the function d푃 (푥) was only defined for computable measures 푃. Though the definition can be extended, it is not clear at all that the expression d푃 (푥) will also play the role of universal test for arbitrary non-computable distributions. Levin’s theory culminating in [33] develops this idea, and we return to it in later parts of these notes. y

69 3. Information

Let us examine the size of the defects of naive additivity and information symmetry. Corollary 3.1.12 For the defect of additivity, we have

+ 퐾(푥) + 퐾( 푦 | 푥) − 퐾(푥, 푦) =+ 퐼(퐾(푥) : 푦 | 푥) < 퐾(퐾(푥) | 푥). (3.1.9) For information asymmetry, we have 퐼(푥 : 푦) − 퐼( 푦 : 푥) =+ 퐼(퐾( 푦) : 푥 | 푦) − 퐼(퐾(푥) : 푦 | 푥). (3.1.10) Proof. Immediate from the addition theorem. Theorem 3.1.6 (Levin) The information 퐼(푥 : 푦) is not even asymptotically symmetric. Proof. For some constant 푐, assume |푥| = 푛 and 퐾(퐾(푥) | 푥) ≥ log 푛−log log 푛− 푐. Consider 푥, 푥∗ and 퐾(푥). Then we can check immediately the relations

+ + 퐼(푥 : 퐾(푥)) < 3 log log 푛 < log 푛 − log log 푛 − 푐 < 퐼(푥∗ : 퐾(푥)). and 퐼(퐾(푥) : 푥) =+ 퐼(퐾(푥) : 푥∗). It follows that symmetry breaks down in an exponentially great measure either on the pair (푥, 퐾(푥)) or the pair (푥∗, 퐾(푥)).

Data processing The following useful identity is also classical, and is called the data processing identity: I(푍 : (푋, 푌)) = I(푍 : 푋) + I(푍 : 푌 | 푋). (3.1.11) Let us see what corresponds to this for algorithmic information. Here is the correct conditional version of the addition theorem: Theorem 3.1.7 We have 퐾(푥, 푦 | 푧) =+ 퐾(푥 | 푧) + 퐾( 푦 | 푥, 퐾(푥 | 푧), 푧). (3.1.12) Proof. The same as for the unconditional version. Remark 3.1.13 The reader may have guessed the inequality 퐾(푥, 푦 | 푧) =+ 퐾(푥 | 푧) + 퐾( 푦 | 푥, 퐾(푥), 푧), but it is incorrect: taking 푧 = 푥, 푦 = 퐾(푥), the left-hand side equals 퐾(푥∗ | 푥), and the right-hand side equals 퐾(푥 | 푥) + 퐾(퐾(푥) | 푥∗, 푥) =+ 0. y

70 3.1. Information-theoretic relations

The following inequality is a useful tool. It shows that conditional complexity is a kind of one-sided distance, and it is the algorithmic analogue of the well- known (and not diﬃcult classical inequality)

H(푋 | 푌) ≤ H(푍 | 푌) + H(푋 | 푍).

Theorem 3.1.8 (Simple triangle inequality) We have

+ + 퐾(푧 | 푥) < 퐾( 푦, 푧 | 푥) < 퐾( 푦 | 푥) + 퐾(푧 | 푦). (3.1.13)

Proof. According to (3.1.12), we have

퐾( 푦, 푧 | 푥) =+ 퐾( 푦 | 푥) + 퐾(푧 | 푦, 퐾( 푦 | 푥), 푥).

+ + The left-hand side is > 퐾(푧 | 푥), the second term of the right-hand side is < 퐾(푧 | 푦). Equation (3.1.12) justiﬁes the following. Deﬁnition 3.1.14

퐼∗(푥 : 푦 | 푧) = 퐾(푥 | 푧) + 퐾( 푦 | 푧) − 퐾(푥, 푦 | 푧). (3.1.14)

y Then we have

퐼∗(푥 : 푦 | 푧) =+ 퐾( 푦 | 푧) − 퐾( 푦 | 푥, 퐾(푥 | 푧), 푧) =+ 퐾(푥 | 푧) − 퐾(푥 | 푦, 퐾( 푦 | 푧), 푧).

Theorem 3.1.9 (Algorithmic data processing identity) We have

퐼∗(푧 : (푥, 푦)) =+ 퐼∗(푧 : 푥) + 퐼∗(푧 : 푦 | 푥∗). (3.1.15)

Proof. We have

퐼∗(푧 : (푥, 푦)) =+ 퐾(푥, 푦) + 퐾(푧) − 퐾(푥, 푦, 푧) 퐼∗(푧 : 푥) =+ 퐾(푥) + 퐾(푧) − 퐾(푥, 푧) 퐼∗(푧 : (푥, 푦)) − 퐼∗(푧 : 푥) =+ 퐾(푥, 푦) + 퐾(푥, 푧) − 퐾(푥, 푦, 푧) − 퐾(푥) =+ 퐾( 푦 | 푥∗) + 퐾(푧 | 푥∗) − 퐾( 푦, 푧 | 푥∗) =+ 퐼∗(푧 : 푦 | 푥∗), where we used additivity and the deﬁnition (3.1.14) repeatedly.

71 3. Information

Corollary 3.1.15 The information 퐼∗ is monotonic:

+ 퐼∗(푥 : ( 푦, 푧)) > 퐼∗(푥 : 푦). (3.1.16)

The following theorem is also sometimes useful. Theorem 3.1.10 (Starry triangle inequality) For all 푥, 푦, 푧, we have

+ + 퐾(푥 | 푦∗) < 퐾(푥, 푧 | 푦∗) < 퐾(푧 | 푦∗) + 퐾(푥 | 푧∗). (3.1.17)

Proof. Using the algorithmic data-processing identity (3.1.15) and the deﬁni- tions, we have

퐾(푧) − 퐾(푧 | 푦∗) + 퐾(푥 | 푧∗) − 퐾(푥 | 푦, 퐾( 푦 | 푧∗), 푧∗) =+ 퐼∗( 푦 : 푧) + 퐼∗( 푦 : 푥 | 푧∗) =+ 퐼∗( 푦 : (푥, 푧)) =+ 퐾(푥, 푧) − 퐾(푥, 푧 | 푦∗), which gives, after cancelling 퐾(푧) + 퐾(푥 | 푧∗) on the left-hand side with 퐾(푥, 푧) on the right-hand side, to which it is equal according to additivity, and changing sign: 퐾(푥, 푧 | 푦∗) =+ 퐾(푧 | 푦∗) + 퐾(푥 | 푦, 퐾( 푦 | 푧∗), 푧∗). From here, we proceed as in the proof of the simple triangle inequality (3.1.13)

3.1.2 Information non-increase In this subsection, we somewhat follow the exposition of [23]. The classical data processing identity has a simple but important application. Suppose that the random variables 푍, 푋, 푌 form a Markov chain in this order. Then 퐼(푍 : 푌 | 푋) = 0 and hence 퐼(푍 : (푋, 푌)) = 퐼(푍 : 푋). In words: all information in 푍 about 푌 is coming through 푋: a Markov transition from 푋 to 푌 cannot increase the information about 푍. Let us try to ﬁnd the algorithmic counterpart of this theorem. Here, we rigorously show that this is the case in the algorithmic statistics setting: the information in one object about another cannot be increased by any deterministic algorithmic method by more than a constant. With added randomization this holds with overwhelming probability. For more elaborate versions see [31, 33].

72 3.1. Information-theoretic relations

Suppose we want to obtain information about a certain object 푥. It does not seem to be a good policy to guess blindly. This is conﬁrmed by the following inequality. ∑︁ ∗ m( 푦)2퐼 (푥:푦) < 1 (3.1.18) 푦 which says that for each 푥, the expected value of 2퐼 (푥:푦) is small, even with respect to the universal constructive semimeasure m( 푦). The proof is immediate if we recognize that by the Coding Theorem,

2−퐾 ( 푦 | 푥) 2퐼 (푥:푦) =∗ m( 푦) and use the estimate (1.6.10). We prove a strong version of the information non-increase law under deterministic processing (later we need the attached corollary): Theorem 3.1.11 Given 푥 and 푧, let 푞 be a program computing 푧 from 푥∗. Then we have + 퐼∗( 푦 : 푧) < 퐼∗( 푦 : 푥) + 퐾(푞). Proof. By monotonicity (3.1.16) and the data processing identity (3.1.15):

+ 퐼∗( 푦 : 푧) < 퐼∗( 푦 : (푥, 푧)) =+ 퐼∗( 푦 : 푥) + 퐼∗( 푦 : 푧 | 푥∗).

By deﬁnition of information and the deﬁnition of 푞:

+ + 퐼∗( 푦 : 푧 | 푥∗) < 퐾(푧 | 푥∗) < 퐾(푞).

Randomized computation can increase information only with negligible probability. Suppose that 푧 is obtained from 푥 by some randomized computation. The probability 푝(푧 | 푥) of obtaining 푧 from 푥 is a semicomputable distribution over the 푧’s. The information increase 퐼∗(푧 : 푦) − 퐼∗(푥 : 푦) satisﬁes the theorem below. Theorem 3.1.12 For all 푥, 푦, 푧 we have

∗ ∗ ∗ m(푧 | 푥∗)2퐼 (푧:푦)−퐼 (푥:푦) < m(푧 | 푥∗, 푦, 퐾( 푦 | 푥∗)).

∗ Therefore it is upperbounded by m(푧 | 푥) < m(푧 | 푥∗).

73 3. Information

Remark 3.1.16 The theorem implies

∑︁ ∗ ∗ ∗ m(푧 | 푥∗)2퐼 (푧:푦)−퐼 (푥:푦) < 1. (3.1.19) 푧

This says that the m(· | 푥∗)-expectation of the exponential of the increase is bounded by a constant. It follows that for example, the probability of an in- ∗ crease of mutual information by the amount 푑 is < 2−푑. y Proof. We have, by the data processing inequality:

퐼∗( 푦 : (푥, 푧)) − 퐼∗( 푦 : 푥) =+ 퐼∗( 푦 : 푧 | 푥∗) =+ 퐾(푧 | 푥∗) − 퐾(푧 | 푦, 퐾( 푦 | 푥∗), 푥∗).

+ Hence, using also 퐼∗( 푦 : (푥, 푧)) > 퐼∗( 푦 : 푧) by monotonicity:

+ 퐼∗( 푦 : 푧) − 퐼∗( 푦 : 푥) − 퐾(푧 | 푥∗) < −퐾(푧 | 푦, 퐾( 푦 | 푥∗), 푥∗).

Putting both sides into the exponent gives the statement of the theorem. Remark 3.1.17 The theorem on information non-increase and its proof look similar to the theorems on randomness conservation. There is a formal connection, via the observation made in Remark 3.1.11. Due to the diﬃculties mentioned there, we will explore the connection only later in our notes. y

3.2 The complexity of decidable and enumerable sets

If 휔 = 휔(1)휔(2)··· is an inﬁnite binary sequence, then we may be interested in how the complexity of the initial segments

휔(1 : 푛) = 휔(1)··· 휔(푛) grows with 푛. We would guess that if 휔 is “random” then 퐶(휔(1 : 푛)) is approximately 푛; this question will be studied satisfactorily in later sections. Here, we want to look at some other questions. Can we determine whether the sequence 휔 is computable, by just looking at the complexity of its initial segments? It is easy to see that if 휔 is computable + + then 퐶(휔(1 : 푛)) < log 푛. But of course, if 퐶(휔(1 : 푛)) < log 푛 then it is not necessarily true yet that 휔 is computable. Maybe, we should put 푛 into + the condition. If 휔 is computable then 퐶(휔(1 : 푛) | 푛) < 0. Is it true that + 퐶(휔(1 : 푛) | 푛) < 0 for all 푛 then indeed 휔 is computable? Yes, but the proof is quite diﬃcult (see either its original, attributed to Albert Meyer in [37], or

74 3.2. The complexity of decidable and enumerable sets in [61]). The suspicion arises that when measuring the complexity of starting segments of infinite sequences, neither Kolmogorov complexity nor its prefix version are the most natural choice. Other examples will support the suspicion, so let us introduce, following Loveland’s paper [37], the following variant. Definition 3.2.1 Let us say that a program 푝 decides a string 푥 = (푥(1), 푥(2),... ) on a Turing machine 푇 up to 푛 if for all 푖 ∈ {1, . . . , 푛} we have 푇 ( 푝, 푖) = 푥(푖). Let us further define the decision complexity

퐶푇 (푥; 푛) = min{푙( 푝) : 푝 decides 푥 on 푇 up to 푛}.

y As for the Kolmogorov complexity, an invariance theorem holds, and there is an optimal machine 푇0 for this complexity. Again, we omit the index 푇, assuming that such an optimal machine has been fixed. The differences between 퐶(푥), 퐶(푥 | 푛) and 퐶(푥; 푛) are interesting to explore; intuitively, the important difference is that the program achieving the decision complexity of 푥 does not have to offer precise information about the length of 푥. We clearly have

+ + 퐶(푥 | 푛) < 퐶(푥; 푛) < 퐶(푥).

Examples that each of these inequalities can be strict, are left to exercises. Remark 3.2.2 If 휔(1 : 푛) is a segment of an infinite sequence 휔 then we can write 퐶(휔; 푛) instead of 퐶(휔(1 : 푛); 푛) without confusion, since there is only one way to understand this expression. y Decision complexity offers an easy characterization of decidable infinite sequences. Theorem 3.2.1 Let 휔 = (휔(1), 휔(2),... ) be an infinite sequence. Then 휔 is decidable if and only if + 퐶(휔; 푛) < 0. (3.2.1) Proof. If 휔 is decidable then (3.2.1) is obviously true. Suppose now that (3.2.1) holds: there is a constant 푐 such that for all 푛 we have 퐶(휔(1 : 푛); 푛) < 푐.

75 3. Information

Then there is a program 푝 of length ≤ 푐 with the property that for infinitely many 푛, decides 푥 on the optimal machine 푇0 up to 푛. Then for all 푖, we have 푇0 ( 푝, 푖) = 휔(푖), showing that 휔 is decidable. Let us use now the new tool to prove a somewhat more surprising result. Let 휔 be a 0-1 sequence that is the indicator function of a recursively enumerable set. In other words, the set {푖 : 휔(푖) = 1} is recursively enumerable. As we know such sequences are not always decidable, so 퐶(휔(1 : 푛); 푛) is not going to be bounded. How fast can it grow? The following theorem gives exact answer, showing that it grows only logarithmically. Theorem 3.2.2 (Barzdin, see [2]) a) Let 휔 be the indicator function of a recursively enumerable set 퐸. Then we have + 퐶(휔; 푛) < log 푛. b) The set 퐸 can be chosen such that for all 푛 we have 퐶(휔; 푛) ≥ log 푛. Proof. Let us prove (a) first. Let 푛0 be the first power of 2 larger than 푛. Let Í 푘(푛) = 푖≤푛0 휔(푖). For a constant 푑, let 푝 = 푝(푛, 푑) be a program that contains an initial segment 푞 of size 푑 followed by a the number 푘(푛) in binary notation, and padded to dlog 푛e + 푑 with 0’s. The program 푞 is self-delimiting, so it sees where 푘(푛) begins. The machine 푇0 ( 푝, 푖) works as follows, under the control of 푞. From the length of the whole 푝, it determines 푛0. It begins to enumerate 휔(1 : 푛0) until it found 푘(푛) 1’s. Then it knows the whole 휔(1 : 푛0), so it outputs 휔(푖). Let us prove now (b). Let us list all possible programs 푞 for the machine 푇0 as 푞 = 1, 2,.... Let 휔(푞) = 1 if and only if 푇0 (푞, 푞) = 0. The sequence 휔 is obviously the indicator function of a recursively enumerable set. To show that 휔 has the desired property, assume that for some 푛 there is a 푝 with 푇0 ( 푝, 푖) = 휔(푖) for all 푖 ≤ 푛. Then 푝 > 푛 since 휔( 푝) is defined to be different from 푇0 ( 푝, 푝). It follows that 푙( 푝) ≥ log 푛. Decision complexity has been especially convenient for the above theorem. Neither 퐶(휔(1 : 푛)) nor 퐶(휔(1 : 푛) | 푛) would have been suitable to formulate such a sharp result. To analyze the phenomenon further, we introduce some more concepts. Definition 3.2.3 Let us denote, for this section, by 퐸 the set of those binary strings 푝 on which the optimal prefix machine 푇 halts: 퐸푡 = { 푝 : 푇 ( 푝) halts in < 푡 steps}, (3.2.2) 퐸 = 퐸∞.

76 3.2. The complexity of decidable and enumerable sets

Let 휒 = 휒퐸 (3.2.3) be the infinite sequence that is the indicator function of the set 퐸, when the latter is viewed as a set of numbers. y It is easy to see that the set 퐸 is complete among recursively enumerable sets with respect to many-one reduction. The above theorems show that though it contains an infinite amount of information, this information is not stored in the sequence 휒 densely at all: there are at most log 푛 bits of it in the segment 휒(1 : 푛). There is an infinite sequence, though, in which the same information is stored much more densely: as we will see later, maximally densely. Definition 3.2.4 (Chaitin’s Omega) Let ∑︁ Ω푡 = 2−푙( 푝) , 푝∈퐸푡 (3.2.4) Ω = Ω∞. y Let Ω(1 : 푛) be the sequence of the first 푛 binary digits in the expansion of Ω, and let it also denote the binary number 0.Ω(1)··· Ω(푛). Then we have

Ω(1 : 푛) < Ω < Ω(1 : 푛) + 2−푛.

Theorem 3.2.3 The sequences Ω and 휒 are Turing-equivalent. Proof. Let us show first that given Ω as an oracle, a Turing machine can compute 휒. Suppose that we want to know for a given string 푝 of length 푘 whether 휒( 푝) = 1 that is whether 푇 ( 푝) halts. Let 푡(푛) be the first 푡 for which Ω푡 > Ω(1 : 푛). If a program 푝 of length 푛 is not in 퐸푡(푛) then it is not in 퐸 at all, since 2−푙( 푝) = 2푛 < Ω − Ω푡. It follows that 휒(1 : 2푛) can be computed from Ω(1 : 푛). To show that Ω can be computed from 퐸, let us define the recursively enumerable set 퐸0 = {푟 : 푟 rational, Ω > 푟}. The set 퐸0 is reducible to 퐸 since the latter is complete among recursively enumerable sets with respect to many-one reduction. On the other hand, Ω is obviously computable from 퐸0. The following theorem shows that Ω stores the information more densely. + Theorem 3.2.4 We have 퐾(Ω(1 : 푛)) > 푛.

77 3. Information

Proof. Let 푝1 be a self-delimiting program outputting Ω(1 : 푛). Recall the def- 푡 푡 inition of the sets 퐸 in (3.2.2) and the numbers Ω in (3.2.4). Let Ω1 denote the real number whose binary digits after 0 are given by Ω(1 : 푛), and let 푡1 be 푡 the ﬁrst 푡 with Ω > Ω1. Let 푥1 be the ﬁrst string 푥 such that 푇 ( 푝) ≠ 푥 for any 푝 ∈ 퐸푡1 . We have 퐾(푥1) ≥ 푛. On the other hand, we just computed 푥1 from 푝1, so + + + 퐾(푥1 | 푝1) < 0. We found 푛 ≤ 퐾(푥1) < 퐾( 푝1) + 퐾(푥1 | 푝1) < 퐾( 푝1).

3.3 The complexity of complexity

3.3.1 Complexity is sometimes complex This section is devoted to a quantitative estimation of the uncomputability of the complexity function 퐶(푥). Actually, we will work with the preﬁx complexity 퐾(푥), but the results would be very similar for 퐶(푥). The ﬁrst result shows that the value 퐾(푥) is not only not computable from 푥, but its conditional complexity 퐾(퐾(푥) | 푥) given 푥 is sometimes quite large. How large can it be expected to be? Certainly not much larger than log 퐾(푥) + 2 log log 퐾(푥), since we can describe any value 퐾(푥) using this many bits. But it can come close to this, as shown by Theorem 3.1.5. This theorem says that for all 푛, there exists 푥 of length 푛 with

+ 퐾(퐾(푥) | 푥) > log 푛 − log log 푛. (3.3.1)

Proof of Theorem 3.1.5. Let 푈( 푝, 푥) be the optimal self-delimiting machine for which 퐾( 푦 | 푥) = min푈 ( 푝,푥)=푦 푙( 푝). Let 푠 ≤ log 푛 be such that if 푙(푥) = 푛 then a 푝 of length 푥 can be found for which 푈( 푝, 푥) = 퐾(푥). We will show

+ 푠 > log 푛 − log log 푛. (3.3.2)

Let us say that 푝 ∈ {0, 1}푠 is suitable for 푥 ∈ {0, 1}푛 if there exists a 푘 = 푈( 푝, 푥) and a 푞 ∈ {0, 1}푘 with 푈( 푝, Λ) = 푥. Thus, 푝 is suitable for 푥 if it produces the length of some program of 푥, not necessarily a shortest program. Let 푀푖 denote the set of those 푥 of length 푛 for which there exist at least 푖 diﬀerent suitable 푝 of length 푠. We will examine the sequence

푛 {0, 1} = 푀0 ⊇ 푀1 ⊇ ... ⊇ 푀 푗 ⊇ 푀 푗+1 = ∅,

푠 where 푀 푗 ≠ ∅. It is clear that 2 ≥ 푗. To lowerbound 푗, we will show that the sets 푀푖 decrease only slowly:

+ log |푀푖 | < log |푀푖+1| + 4 log 푛. (3.3.3)

78 3.3. The complexity of complexity

We can assume log |푀푖 \ 푀푖+1| ≥ log |푀푖 | − 1, (3.3.4) otherwise (3.3.3) is trivial. We will write a program that ﬁnds an element 푥 of 푀푖 \ 푀푖+1 with the property 퐾(푥) ≥ log |푀푖 | − 1. The program works as follows. • It ﬁnds 푖, 푛 with the help of descriptions of length log 푛 + 2 log log 푛, and 푠 with the help of a description of length 2 log log 푛.

• It ﬁnds |푀푖+1| with the help of a description of length log |푀푖+1| + log 푛 + 2 log log 푛.

• From these data, it determines the set 푀푖+1, and then begins to enumerate the set 푀푖 \ 푀푖+1 as 푥1, 푥2,.... For each of these elements 푥푟, it knows that there are exactly 푖 programs suitable for 푥푟, ﬁnd all those, and ﬁnd the shortest program for 푥 produced by these. Therefore it can compute 퐾(푥푟).

• According to the assumption (3.3.4), there is an 푥푟 with 퐾(푥푟) ≥ log |푀푖 | − 1. The program outputs the ﬁrst such 푥푟. + The construction of the program shows that its length is < log |푀푖+1| + 4 log 푛, hence for the 푥 we found

+ log |푀푖 | − 1 ≤ 퐾(푥) < log |푀푖+1| + 4 log 푛, which proves (3.3.3). This implies 푗 ≥ 푛/(4 log 푛), and hence (3.3.2).

3.3.2 Complexity is rarely complex Let 푓 (푥) = 퐾(퐾(푥) | 푥). We have seen that 푓 (푥) is sometimes large. Here, we will show that the sequences 푥 for which this happens are rare. Recall that we deﬁned 휒 in (3.2.3) as the indicator sequence of the halting problem. Theorem 3.3.1 We have

+ 퐼(휒 : 푥) > 푓 (푥) − 2.4 log 푓 (푥).

In view of the inequality (3.1.18), this shows that such sequences are rare, even in terms of the universal probability, so they are certainly rare if we measure them by any computable distribution. So, we may even call such sequences “exotic”.

79 3. Information

In the proof, we start by showing that the sequences in question are rare. Then the theorem will follow when we make use of the fact that 푓 (푥) is computable from 휒. We need a lemma about the approximation of one measure by another from below. 푚 Lemma 3.3.1 For any semimeasure 휈 and measure 휇 ≤ 휈, let 푆푚 = {푥 : 2 휇(푥) ≤ −푚 Í −푛 −푛+1 휈(푥)}. Then 휇(푆푚) ≤ 2 . If 푥 (휈(푥) − 휇(푥)) < 2 then 휈(푆1) < 2 . The proof is straightforward. Let 푋 (푛) = {푥 : 푓 (푥) = 푛}. Lemma 3.3.2 We have ∗ m(푋 (푛)) < 푛1.22−푛. Proof. Using the deﬁnition of 퐸푡 in (3.2.2), let ∑︁ m푡 (푥) = { 푝 : 푇 ( 푝) = 푥 in < 푡 steps}, 퐾푡 (푥) = − log m푡 (푥).

Then according to the deﬁnition (1.6.14) we have m(푥) = m∞(푥), and 퐾(푥) =+ 퐾∞(푥). Let

푡(푘) = min{푡 : Ω(1 : 푘) < Ω푡 }, 휇 = m푡(푘) .

+ Clearly, 퐾푡(푘) (푥) > 퐾(푥). Let us show that 퐾푡(푘) (푥) is a good approximation for 퐾(푥) for most 푥. Let

푌 (푘) = {푥 : 퐾(푥) ≤ 퐾푡(푘) (푥) − 1}.

By deﬁnition, for 푥 ∉ 푌 (푘) we have

|퐾푡(푘) (푥) − 퐾(푥)| =+ 0.

On the other hand, applying Lemma 3.3.1 with 휇 = m푡(푘) , 휈 = m, we obtain

m(푌 (푘)) < 2−푘+1. (3.3.5)

Note that 퐾(퐾푡(푘) (푥) | 푥, Ω(1 : 푘)) =+ 0,

80 3.3. The complexity of complexity therefore for 푥 ∉ 푌 (푘) we have

퐾(퐾(푥) | 푥, Ω(1 : 푘)) =+ 0, + 퐾(퐾(푥) | 푥) < 푘 + 1.2 log 푘.

If 푛 = 푘 + 1.2 log 푘 then 푘 =+ 푛 − 1.2 log 푛, and hence, if 푥 ∉ 푌 (푛 − 1.2 log 푛) + then 퐾(퐾(푥) | 푥) < 푛. Thus, there is a constant 푐 such that

푋 (푛) ⊆ 푌 (푛 − 1.2 log 푛 − 푐).

Using (3.3.5) this gives the statement of the lemma. Proof of Theorem 3.3.1. Since 푓 (푥) is computable from Ω, the function

휈(푥) = m(푥)2 푓 (푥) ( 푓 (푥))−2.4 is computable from Ω. Let us show that it is a semimeasure (within a multiplicative constant). Indeed, using the above lemma:

∑︁ ∑︁ ∑︁ ∑︁ ∑︁ ∗ 휈(푥) = 휈(푥) = 2푘푘−2.4m(푋 (푘)) = 푘−1.2 < 1. 푥 푘 푥 ∈푋 (푘) 푘 푘

∗ Since m(· | Ω) is the universal semimeasure relative to Ω we ﬁnd m(푥 | Ω) > 휈(푥), hence

+ 퐾(푥 | Ω) < − log 휈(푥) = 퐾(푥) − 푓 (푥) + 2.4 log 푓 (푥), + 퐼(Ω : 푥) > 푓 (푥) − 2.4 log 푓 (푥).

Since Ω is equivalent to 휒, the proof is complete.

4 Generalizations

4.1 Continuous spaces, noncomputable measures

This section starts the consideration of randomness in continuous spaces and randomness with respect to noncomputable measures.

4.1.1 Introduction The algorithmic theory of randomness is well developed when the underlying space is the set of finite or infinite sequences and the underlying probability distribution is the uniform distribution or a computable distribution. These restrictions seem artificial. Some progress has been made to extend the theory to arbitrary Bernoulli distributions by Martin-Löf in [38], and to arbitrary distributions, by Levin in [30, 32, 33]. The paper [25] by Hertling and Weihrauch also works in general spaces, but it is restricted to computable measures. Similarly, Asarin’s thesis [1] defines randomness for sample paths of the Brownian motion: a fixed random process with computable distribution. The exposition here has been inspired mainly by Levin’s early paper [32] (and the much more elaborate [33] that uses different definitions): let us summarize part of the content of [32]. The notion of a constructive topological space X and the space of measures over X is introduced. Then the paper defines the notion of a uniform test. Each test is a lower semicomputable function ∫ (휇, 푥) ↦→ 푓휇 (푥), satisfying 푓휇 (푥)휇(푑푥) ≤ 1 for each measure 휇. There are also some additional conditions. The main claims are the following.

a) There is a universal test t휇 (푥), a test such that for each other test 푓 there is a constant 푐 > 0 with 푓휇 (푥) ≤ 푐 · t휇 (푥). The deﬁciency of randomness is deﬁned as d휇 (푥) = log t휇 (푥). b) The universal test has some strong properties of “randomness conservation”: these say, essentially, that a computable mapping or a computable random-

83 4. Generalizations

ized transition does not decrease randomness. c) There is a measure 푀 with the property that for every outcome 푥 we have t푀 (푥) ≤ 1. In the present work, we will call such measures neutral. d) Semimeasures (semi-additive measures) are introduced and it is shown that there is a lower semicomputable semi-measure that is neutral (let us assume that the 푀 introduced above is lower semicomputable). e) Mutual information 퐼(푥 : 푦) is defined with the help of (an appropriate version of) Kolmogorov complexity, between outcomes 푥 and 푦. It is shown that 퐼(푥 : 푦) is essentially equal to d푀×푀 (푥, 푦). This interprets mutual information as a kind of “deficiency of independence”. This impressive theory leaves a number of issues unresolved: 1. The space of outcomes is restricted to be a compact topological space, moreover, a particular compact space: the set of sequences over a finite alphabet (or, implicitly in [33], a compactified infinite alphabet). However, a good deal of modern probability theory happens over spaces that are not even locally compact: for example, in case of the Brownian motion, over the space of continuous functions. 2. The definition of a uniform randomness test includes some conditions (different ones in [32] and in [33]) that seem somewhat arbitrary. 3. No simple expression is known for the general universal test in terms of description complexity. Such expressions are nice to have if they are available. Here, we intend to carry out as much of Levin’s program as seems possible after removing the restrictions. A number of questions remain open, but we feel that they are worth to be at least formulated. A fairly large part of the exposition is devoted to the necessary conceptual machinery. This will also allow to carry further some other initiatives started in the works [38] and [30]: the study of tests that test nonrandomness with respect to a whole class of measures (like the Bernoulli measures). Constructive analysis has been developed by several authors, converging approximately on the same concepts, We will make use of a simplified version of the theory introduced in [57]. As we have not found a constructive measure theory in the literature fitting our purposes, we will develop this theory here, over (constructive) complete separable metric spaces. This generality is well supported by standard results in measure theoretical probability, and is sufficient for constructivizing a large part of current probability theory.

84 4.1. Continuous spaces, noncomputable measures

The appendix recalls some of the needed topology, measure theory, constructive analysis and constructive measure theory. We also make use of the notation introduced there.

4.1.2 Uniform tests We first define tests of randomness with respect to an arbitrary measure. Recall the definition of lower semicomputable real functions on a computable metric space X. Definition 4.1.1 Let us be given a computable complete metric space X = (푋, 푑, 퐷, 훼). For an arbitrary measure 휇 ∈ M(X), a 휇-test of randomness is a 휇-lower semicomputable function 푓 : 푋 → R+ with the property 휇 푓 ≤ 1. We call an element 푥 random with respect to 휇 if 푓 (푥) < ∞ for all 휇-tests 푓 . But even among random elements, the size of the tests quantifies (non-)randomness. A uniform test of randomness is a lower semicomputable function 푓 : X × M(X) → R+, written as (푥, 휇) ↦→ 푓휇 (푥) such that 푓휇 (·) is a 휇-test for each 휇. y The condition 휇 푓 ≤ 1 guarantees that the probability of those outcomes whose randomness is ≥ 푚 is at most 1/푚. The definition of tests is in the spirit of Martin-Löf’s tests. The important difference is in the semicomputability condition: instead of restricting the measure 휇 to be computable, we require the test to be lower semicomputable also in its argument 휇. The following result implies that every 휇-test can be extended to a uniform test.

Theorem 4.1.1 Let 휙푒, 푒 = 1, 2,... be an enumeration of all lower semicomputable functions X × Y × M(X) → R+, where Y is also a computable metric space, and 푠 : Y × M(X) → R a lower semicomputable function. There is a recursive function 푒 ↦→ 푒0 with the property that

a) For each 푒, the function 휙푒0 (푥, 푦, 휇) is everywhere deﬁned with 푥 휇 휙푒0 (푥, 푦, 휇) ≤ 푠( 푦, 휇). 푥 b) For each 푒, 푦, 휇, if 휇 휙푒 (푥, 푦휇) < 푠( 푦, 휇) then 휙푒0 (·, 푦, 휇) = 휙푒 (·, 푦, 휇). This theorem generalizes a theorem of Hoyrup and Rojas (in allowing a lower semicomputable upper bound 푠( 푦, 휇)).

Proof. By Proposition B.1.38, we can represent 휙푒 (푥, 푦, 휇) as a supremum 휙푒 = sup푖 ℎ푒,푖 where ℎ푒,푖 (푥, 푦, 휇) is a computable function of 푒, 푖 monotonically increasing in 푖. Similarly, we can represent 푠( 푦, 휇) as a supremum sup 푗 푠푗 ( 푦, 휇)

85 4. Generalizations

where 푠푗 ( 푦, 휇) is a computable function monotonically increasing in 푗. The inte- 푥 gral 휇 ℎ푒,푖 (푥, 푦, 휇) is computable as a function of ( 푦, 휇), in particular it is upper semicomputable. 0 푥 Define ℎ푒,푖, 푗 (푥, 푦, 휇) as ℎ푒,푖 (푥, 푦, 휇) for all 푗, 푦, 휇 with 휇 ℎ푒,푖 (푥, 푦, 휇) < 푠푗 ( 푦, 휇), and 0 otherwise. Since 푠푗 ( 푦, 휇) is computable this definition makes 0 00 the function ℎ푒,푖, 푗 (푥, 푦, 휇) lower semicomputable. The function ℎ푒,푖 (푥, 푦, 휇) = 0 00 sup 푗 ℎ푒,푖, 푗 (푥, 푦, 휇) is then also lower semicomputable, with ℎ푒,푖 (푥, 푦, 휇) ≤ 푥 0 00 ℎ푒,푖 (푥, 푦, 휇), and 휇 ℎ푒,푖 (푥, 푦, 휇) ≤ 푠( 푦, 휇). Also, ℎ푒,푖 (푥, 푦, 휇) is monotonically 0 00 increasing in 푖. The function 휙푒 (푥, 푦, 휇) = sup푖 ℎ푒,푖 (푥, 푦, 휇) is then also lower 푥 0 semicomputable, and by Fubini’s theorem we have 휇 휙푒 (푥, 푦, 휇) ≤ 푠( 푦, 휇). 0 Define 휙푒0 (푥, 푦, 휇) = 휙푒 (푥, 푦, 휇). Consider any 푒, 푦, 휇 such that 푥 푥 휇 휙푒 (푥, 푦, 휇) < 푠( 푦, 휇) holds. Then for every 푖 there is a 푗 with 휇 ℎ푒,푖 (푥, 푦, 휇) < 0 00 푠푗 ( 푦, 휇), and hence ℎ푒,푖, 푗 (푥, 푦, 휇) = ℎ푒,푖 (푥, 푦, 휇). It follows that ℎ푒,푖 (푥, 푦, 휇) = 0 ℎ푒,푖 (푥, 푦, 휇) for all 푖 and hence 휙푒 (푥, 푦, 휇) = 휙푒 (푥, 푦, 휇).

Corollary 4.1.2 (Uniform extension) There is an operation 퐻푒 (푥, 휇) ↦→ 퐻푒0 (푥, 휇) with the property that 퐻푒0 (푥, 휇) is a uniform test and if 2퐻푒 (·, 휇) is a 휇-test then 퐻푒0 (푥, 휇) = 퐻푒 (푥, 휇). ( ) 1 ( ) ( ) Proof. In Theorem 4.1.1 set 휙푒 푥, 푦, 휇 = 2 퐻푒 푥, 휇 with 푠 푦, 휇 = 1.

Corollary 4.1.3 (Universal generalized test) Let 푠 : Y × M(X) → R+ a lower semicomputable function. Let 퐸 be the set of lower semicomputable functions 휙(푥, 푦, 휇) ≥ 0 with 휇푥휙(푥, 푦, 휇) ≤ 푠( 푦, 휇). There is a function 휓 ∈ 퐸 that is optimal in the sense that for all 휙 ∈ 퐸 there is a constant 푐휙 with 휙 ≤ 2푐휙휓. 0 Proof. Apply the operation 푒 ↦→ 푒 of theorem 4.1.1 to the sequence 휙푒 (푥, 푦, 휇) (푒 = 1, 2,...) of all lower semicomputable functions of 푥, 푦, 휇. The elements of 0 0 the sequence 휙푒 (푥, 푦, 휇), 푒 = 1, 2,... are in 퐸 and and the sequence 2휙푒 (푥, 푦, 휇), Í∞ −푒 0 푒 = 1, 2,... contains all elements of 퐸. Hence the function 휓(푥, 푦, 휇) = 푒=1 2 휙푒 (푥, 푦, 휇) is in 퐸 and has the optimality property. Deﬁnition 4.1.4 A uniform test 푢 is called universal if for every other test 푡 there is a constant 푐푡 > 0 such that for all 푥, 휇 we have 푡휇 (푥) ≤ 푐푢휇 (푥). y Theorem 4.1.2 (Universal test,[26]) There is a universal uniform test. Proof. This is a special case of Corollary 4.1.3 with 푠( 푦, 휇) = 1.

86 4.1. Continuous spaces, noncomputable measures

Definition 4.1.5 Let us fix a universal uniform test, called t휇 (푥). An element 푥 ∈ 푋 is called random with respect to measure 휇 ∈ M(X) if t휇 (푥) < ∞. The deficiency of randomness is defined as d휇 (푥) = log t휇 (푥). y If the space is discrete then typically all elements are random with respect to 휇, but they will still be distinguished according to their different values of d휇 (푥).

4.1.3 Sequences Let our computable metric space X = (푋, 푑, 퐷, 훼) be the Cantor space of Exam- ple B.1.35.2: the set of sequences over a (finite or countable) alphabet ΣN. We may want to measure the non-randomness of finite sequences, viewing them as initial segments of infinite sequences. Take the universal test t휇 (푥). For this, it is helpful to apply the representation of Proposition B.1.23, taking into account that adding the extra parameter 휇 does not change the validity of the theorem: ∗ Proposition 4.1.6 There is a function 푔 : M(X) × Σ → R+ with t휇 (휉) = ≤푛 sup푛 푔휇 (휉 ), and with the following properties: a) 푔 is lower semicomputable.

b) 푣 v 푤 implies 푔휇 (푣) ≤ 푔휇 (푤). Í c) For all integer 푛 ≥ 0 we have 푤∈Σ푛 휇(푤)푔휇 (푤) ≤ 1. ≤푛 The properties of the function 푔휇 (푤) clearly imply that sup푛 푔휇 (휉 ) is a uniform test. The existence of a universal function among the functions 푔 can be proved by the usual methods:

Proposition 4.1.7 Among the functions 푔휇 (푤) satisfying the properties listed in Proposition 4.1.6, there is one that dominates to within a multiplicative constant. These facts motivate the following definition. Definition 4.1.8 (Extended test) Over the Cantor space, we extend the definition of a universal test t휇 (푥) to finite sequences as follows. We fix a function ∗ t휇 (푤) with 푤 ∈ Σ whose existence is assured by Proposition 4.1.7. For infinite ≤푛 sequences 휉 we define t휇 (휉) = sup푛 t휇 (휉 ). The test with values defined also on finite sequences will be called an extended test. y We could try to define extended tests also over arbitrary constructive metric spaces, extending them to the canonical balls, with the monotonicity property

87 4. Generalizations

that t휇 (푣) ≤ t휇 (푤) if ball 푤 is manifestly included in ball 푣. But there is nothing simple and obvious corresponding to the integral requirement (c). Over the space ΣN for a ﬁnite alphabet Σ, an extended test could also be extracted directly from a test, using the following formula (as observed by Vyugin and Shen). Deﬁnition 4.1.9

푢(푧) = inf{푢(휔) : 휔 is an inﬁnite extension of 푧}.

y This function is lower semicomputable, by Proposition B.1.31.

4.1.4 Conservation of randomness

For 푖 = 1, 0, let X푖 = (푋푖, 푑푖, 퐷푖, 훼푖) be computable metric spaces, and let M푖 = (M(X푖), 휎푖, 휈푖) be the effective topological space of probability measures over X푖. Let Λ be a computable probability kernel from X1 to X0 as defined in Subsection B.2.3. In the following theorem, the same notation d휇 (푥) will refer to the deficiency of randomness with respect to two different spaces, X1 and X0, but this should not cause confusion. Let us first spell out the conservation theorem before interpreting it.

Theorem 4.1.3 For a computable probability kernel Λ from X1 to X0, we have

푦 ∗ 휆푥 tΛ∗휇 ( 푦) < t휇 (푥). (4.1.1)

Proof. Let t휈 (푥) be the universal test over X0. The left-hand side of (4.1.1) can be written as 푢휇 = ΛtΛ∗휇 . ∗ According to (A.2.4), we have 휇푢휇 = (Λ 휇)tΛ∗휇 which is ≤ 1 since t is a test. If we show that (휇, 푥) ↦→ 푢휇 (푥) is lower semicomputable then the universality of ∗ t휇 will imply that 푢휇 < t휇. According to Proposition B.1.38, as a lower semicomputable function, t휈 ( 푦) can be written as sup푛 푔푛 (휈, 푦), where (푔푛 (휈, 푦)) is a computable sequence of computable functions. We pointed out in Subsection B.2.3 that the function ∗ ∗ 휇 ↦→ Λ 휇 is computable. Therefore the function (푛, 휇, 푥) ↦→ 푔푛 (Λ 휇, 푓 (푥)) is also a computable. So, 푢휇 (푥) is the supremum of a computable sequence of computable functions and as such, lower semicomputable.

88 4.1. Continuous spaces, noncomputable measures

It is easier to interpret the theorem ﬁrst in the special case when Λ = Λℎ for a computable function ℎ : 푋1 → 푋0, as in Example B.2.12. Then the theorem simpliﬁes to the following. + Corollary 4.1.10 For a computable function ℎ : 푋1 → 푋0, we have dℎ∗휇 (ℎ(푥)) < d휇 (푥).

Informally, this says that if 푥 is random with respect to 휇 in X1 then ℎ(푥) is ∗ essentially at least as random with respect to the output distribution ℎ 휇 in X0. Decrease in randomness can only be caused by complexity in the deﬁnition of the function ℎ. Let us specialize the theorem even more:

Corollary 4.1.11 For a probability distribution 휇 over the space 푋 × 푌 let 휇푋 be its marginal on the space 푋. Then we have

+ d휇푋 (푥) < d휇 ((푥, 푦)).

This says, informally, that if a pair is random then each of its elements is random (with respect to the corresponding marginal distribution). In the general case of the theorem, concerning random transitions, we cannot bound the randomness of each outcome uniformly. The theorem asserts that the average nonrandomness, as measured by the universal test with respect to the + 푦 dΛ∗휇 ( 푦) output distribution, does not increase. In logarithmic notation: 휆푥 2 < ∫ d ∗ ( 푦) + d휇 (푥), or equivalently, 2 Λ 휇 휆푥 (푑 푦) < d휇 (푥).

Corollary 4.1.12 Let Λ be a computable probability kernel from X1 to X0. There is a constant 푐 such that for every 푥 ∈ X1, and integer 푚 > 0 we have

−푚 휆푥 { 푦 : dΛ∗휇 ( 푦) > d휇 (푥) + 푚 + 푐} ≤ 2 .

Thus, in a computable random transition, the probability of an increase of randomness deﬁciency by 푚 units (plus a constant 푐) is less than 2−푚. The constant 푐 comes from the description complexity of the transition Λ. A randomness conservation result related to Corollary 4.1.10 was proved in [25]. There, the measure over the space X0 is not the output measure of the transformation, but is assumed to obey certain inequalities related to the transformation.

89 4. Generalizations

4.2 Test for a class of measures

4.2.1 From a uniform test A Bernoulli measure is what we get by tossing a (possibly biased) coin repeatedly. Definition 4.2.1 Let 푋 = BN be the set of infinite binary sequences, with the usual sequence topology. Let 퐵푝 be the measure on 푋 that corresponds to tossing a coin independently with probability 푝 of success: it is called the Bernoulli measure with parameter 푝. Let B denote the set of all Bernoulli measures. y Given a sequence 푥 ∈ 푋 we may ask the question whether 푥 is random with respect to at least some Bernoulli measure. (It can clearly not be random with respect to two different Bernoulli measures since if 푥 is random with respect to 퐵푝 then its relative frequencies converge to 푝.) This idea suggests two possible definitions for a test of the property of “Bernoulliness”:

1. We could define tB (푥) = inf휇 ∈B (푥). 2. We could define the notion of a Bernoulli test as a lower semicomputable function 푓 (푥) with the property 퐵푝 푓 ≤ 1 for all 푝. We will see that in case of this class of measures the two definitions lead to essentially the same test. Let us first extend the definition to more general sets of measures, still having a convenient property. Definition 4.2.2 (Class tests) Consider a class C of measures that is effectively compact in the sense of Definition B.1.27 or (equivalently for metric spaces) in the sense of Theorem B.1.1. A lower semicomputable function 푓 (푥) is called a C-test if for all 휇 ∈ C we have 휇 푓 ≤ 1. It is a universal C-test if it dominates all other C-tests to within a multiplicative constant. y Example 4.2.3 It is easy to show that the class B is effectively compact. One way is to appeal to the general theorem in Proposition B.1.32 saying that applying a computable function to an effectively compact set (in this case the interval [0, 1]), the image is also an effectively compact set. y For the case of infinite sequences, we can also define extended tests. Definition 4.2.4 (Extended class test) Let our space X be the Cantor space of infinite sequences ΣN. Consider a class C of measures that is effectively compact in the sense of Definition B.1.27 or (equivalently for metric spaces) in the sense ∗ of Theorem B.1.1. A lower semicomputable function 푓 : Σ → R+ is called an extended C-test if it is monotonic with respect to the prefix relation and for all

90 4.2. Test for a class of measures

휇 ∈ C and integer 푛 ≥ 0 we have ∑︁ 휇(푥) 푓 (푥) ≤ 1. 푥 ∈Σ푛

It is universal if it dominates all other extended C-tests to within a multiplicative constant. y The following observation is immediate. N Proposition 4.2.5 A function 푓 : Σ → R+ is a class test if and only if it can be ≤푛 represented as lim푛 푔(휉 ) where 푔(푥) is an extended class test. The following theorem deﬁnes a universal C-test.

Theorem 4.2.1 Let t휇 (푥) be a universal uniform test. Then 푢(푥) = inf휇 ∈C t휇 (푥) deﬁnes a universal C-test.

Proof. Let us show ﬁrst that 푢(푥) is a C-test. It is lower semicomputable according to Proposition B.1.31. Also, for each 휇 we have 휇푢 ≤ 휇t휇 ≤ 1, showing that 푢(푥) is a C-test. Let us now show that it is universal. Let 푓 (푥) be an arbitrary C-test. By Corollary 4.1.2 there is a uniform test 푔휇 (푥) such that for all 휇 ∈ C we have 푔휇 (푥) = 푓 (푥)/2. It follows from the universality of the uniform test t휇 (푥) that ∗ ∗ ∗ 푓 (푥) < 푔휇 (푥) < t휇 (푥) for all 휇 ∈ C. But then 푓 (푥) < inf휇 ∈C t휇 (푥) = 푢(푥). For the case of sequences, the same statement can be made for extended tests. (This is not completely automatic since a test is obtained from an extended test via a supremum, on the other hand a class test is obtained, according the theorem above, via an inﬁmum.)

4.2.2 Typicality and class tests The set of Bernoulli measures has an important property shared by many classes considered in practice: namely that random sequences determine the measure to which it belongs. Deﬁnition 4.2.6 Consider a class C of measures over a computable metric space X = (푋, 푑, 퐷, 훼). We will say that a lower semicomputable function

푠 : 푋 × C → R+ is a separating test for C if

• 푠휇 (·) is a test for each 휇 ∈ C.

91 4. Generalizations

• If 휇 ≠ 휈 then 푠휇 (푥) ∨ 푠휈 (푥) = ∞ for all 푥 ∈ 푋.

Given a separating test 푠휇 (푥) we call an element 푥 typical for 휇 ∈ C if 푠휇 (푥) < ∞. y A typical element determines uniquely the measure 휇 for which it is typical. Note that if a separating tests exists for a class then any two diﬀerent measures 휇1, 휇2 in the class are orthogonal to each other, that is there are disjoint measureable sets 퐴1, 퐴2 with 휇 푗 (퐴 푗) = 1. Indeed, let 퐴 푗 = {푥 : 푠휇 푗 (푥) < ∞}. Let us show a nontrivial example: the class of B of Bernoulli measures. Recall that by Chebyshev’s inequality we have

푛 ∑︁ 1/2 1/2 −2 퐵푝 ({푥 ∈ B : | 푥(푖) − 푛푝| ≥ 휆푛 ( 푝(1 − 푝)) }) ≤ 휆 . 푖 Since 푝(1 − 푝) ≤ 1/4, this implies

푛 ∑︁ 1/2 −2 퐵푝 ({푥 ∈ B : | 푥(푖) − 푛푝 > 휆푛 /2}) < 휆 . 푖

Setting 휆 = 푛0.1 and ignoring the factor 1/2 gives

푛 ∑︁ 0.6 −0.2 퐵푝 ({푥 ∈ B : | 푥(푖) − 푛푝| > 푛 }) < 푛 . 푖

Setting 푛 = 2푘:

2푘 ∑︁ 푘 0.6푘 −0.2푘 퐵푝 ({푥 ∈ B : | 푥(푖) − 2 푝| > 2 }) < 2 . (4.2.1) 푖 Now, for the example. Example 4.2.7 For a sequence 휉 in BN, and for 푝 ∈ [0, 1] let

2푘 ∑︁ 푘 0.6푘 푔푝 (푥) = 푔퐵푝 (푥) = sup{푘 : | 휉(푖) − 2 푝| > 2 }. 푖=1 Then we have

휉 ∑︁ −0.2푘 퐵푝푔푝 (휉) ≤ 푘 · 2 = 푐 < ∞ 푘 for some constant 푐, so 푠푝 (휉) = 푔푝 (푥)/푐 is a test for each 푝. The property 푠푝 (휉) < −푘 Í2푘 ∞ implies that 2 푖=1 휉(푖) converges to 푝. For a given 휉 this is impossible for both 푝 and 푞 for 푝 ≠ 푞, hence 푠푝 (휉) ∨ 푠푞 (휉) = ∞. y

92 4.2. Test for a class of measures

The following structure theorem gives considerable insight.

Theorem 4.2.2 Let C be an eﬀectively compact class of measures, let t휇 (푥) be the universal uniform test and let tC (푥) be a universal class test for C. Assume that a separating test 푠휇 (푥) exists for C. Then we have the representation

∗ t휇 (푥) = tC (푥) ∨ 푠휇 (푥) for all 휇 ∈ C, 푥 ∈ 푋.

∗ Proof. First, we have tC (푥) ∨ 푠휇 (푥) < t휇 (푥). Indeed as we know from the Uni- form Extension Corollary 4.1.2, we can extend 푠휇 (푥)/2 to a uniform test, hence ∗ 푠휇 (푥) < t휇 (푥). Also by deﬁnition tC (푥) ≤ t휇 (푥). On the other hand, let us show tC (푥) ∨ 푠휇 (푥) ≥ t휇 (푥). Suppose ﬁrst that 푥 is not random with respect to any 휈 ∈ C: then tC (푥) = ∞. Suppose now that 푥 is random with respect to some 휈 ∈ C, 휈 ≠ 휇. Then 푠휇 (푥) = ∞. Finally, suppose t휇 (푥) < ∞. Then t휈 (푥) = ∞ for all 휈 ∈ C, 휈 ≠ 휇, hence tC (푥) = inf휈∈C t휈 (푥) = t휇 (푥), so the inequality holds again. The above theorem separates the randomness test into two parts. One part tests randomness with respect to the class C, the other one tests typicality with respect to the measure 휇. In the Bernoulli example,

• Part tB (휉) checks “Bernoulliness”, that is independence. It encompasses all the irregularity criteria.

• Part 푠푝 (휉) checks (crudely) for the law of large numbers: whether relative frequency converges (fast) to 푝. If the independence of the sequence is taken for granted, we may assume that the class test is satisﬁed. What remains is typicality testing, which is similar to ordinary statistical parameter testing. Remarks 4.2.8

1. Separation is the only requirement of the test 푠휇 (푥), otherwise, for example in the Bernoulli test case, no matter how crude the convergence criterion expressed by 푠휇 (푥), the maximum tC (푥) ∨ 푠휇 (푥) is always (essentially) the same universal test. 2. Though the convergence criterion can be crude, but one still seems to need some kind of constructive convergence of the relative frequencies if the separation test is to be deﬁned in terms of relative frequency convergence. y

93 4. Generalizations

Example 4.2.9 For 휀 > 0 let 푃(휀) be the Markov chain 푋1, 푋2,... with set of states {0, 1}, with transition probabilities 푇 (0, 1) = 푇 (1, 0) = 휀 and 푇 (0, 0) = 푇 (1, 1) = 1 − 휀, and with 푃[ 푋1 = 0] = 푃[ 푋1 = 1] = 1/2. Let C(훿) be the class of all 푃(휀) with 휀 ≥ 훿. For each 훿 > 0, this is an eﬀectively compact class, and a separating test is easy to construct since an eﬀective law of large numbers holds for these Markov chains. We can generalize to the set of 푚-state stationary Markov chains whose eigenvalue gap is ≥ 휀. y This example is in contrast to V’yugin’s example [56] showing that, in the nonergodic case, in general no recursive speed of convergence can be guaranteed in the Ergodic Theorem (which is the appropriate generalization of the law of large numbers)1. We can show that if a computable Markov chain is ergodic then its law of large numbers does have a constructive speed of convergence. Hopefully, this observation can be extended to some interesting compact classes of ergodic pro- cesses.

4.2.3 Martin-Löf’s approach Martin-Löf also gave a definition of Bernoulli tests in [38]. For its definition let us introduce the following notation. Notation 4.2.10 The set of sequences with a given frequency of 1’s will be denoted as follows: ∑︁ B(푛, 푘) = {푥 ∈ B푛 : 푥(푖) = 푘}. 푖 y Martin-Löf’s definition translates to the integral constraint version as follows: Definition 4.2.11 Let 푋 = BN be the set of infinite binary sequences with the ∗ usual metrics. A combinatorial Bernoulli test is a function 푓 : B → R+ with the following constraints: a) It is lower semicomputable. b) It is monotonic with respect to the prefix relation.

1This paper of Vyugin also shows that though there is no recursive speed of convergence, a certain constructive proof of the pointwise ergodic theorem still gives rise to a test of randomness.

94 4.2. Test for a class of measures

c) For all 0 ≤ 푘 ≤ 푛 we have

∑︁ 푛 푓 (푥) ≤ . (4.2.2) 푘 푥 ∈B(푛,푘) y The following observation is useful. Proposition 4.2.12 If a combinatorial Bernoulli test 푓 (푥) is given on strings 푥 of length less than 푛, then extending it to longer strings using monotonicity we get a function that is still a combinatorial Bernoulli test.

Proof. It is sufficient to check the relation (4.2.2). We have (even for 푘 = 0, when B(푛 − 1, 푘 − 1) = ∅): ∑︁ ∑︁ ∑︁ 푓 (푥) ≤ 푓 ( 푦) + 푓 ( 푦) 푥 ∈B(푛,푘) 푦 ∈B(푛−1,푘−1) 푦 ∈B(푛−1,푘) 푛 − 1 푛 − 1 푛 ≤ + = . 푘 − 1 푘 푘 The following can be shown using standard methods: Proposition 4.2.13 (Universal combinatorial Bernoulli test) There is a universal combinatorial Bernoulli test 푓 (푥), that is a combinatorial Bernoulli test with the property that for every combinatorial Bernoulli test ℎ(푥) there is a constant 푐ℎ > 0 such that for all 푥 we have ℎ(푥) ≤ 푐ℎ푔(푥). Definition 4.2.14 Let us fix a universal combinatorial Bernoulli test 푏(푥) and extend it to infinite sequences 휉 by

푏(휉) = sup 푏(휉≤푛). 푛

N Let tB (휉) be a universal class test for Bernoulli measures, for 휉 ∈ B . y Let us show that the general class test for Bernoulli measures and Martin- Löf’s Bernoulli test yield the same random sequences. ∗ Theorem 4.2.3 With the above deﬁnitions, we have 푏(휉) = tB (휉). In words: a sequence is nonrandom with respect to all Bernoulli measures if and only if it is rejected by a universal combinatorial Bernoulli test.

95 4. Generalizations

Proof. We ﬁrst show 푏(휉) ≤ tB (휉). Moreover, we show that 푏(푥) is an extended class test for the class of Bernoulli measures. We only need to check the sum condition, namely that for all 0 ≤ 푝 ≤ 1, and all 푛 > 0 the inequality Í 푥 ∈B푛 퐵푝 (푥)푏(푥) ≤ 1 holds. Indeed, we have

푛 ∑︁ ∑︁ 푘 푛−푘 ∑︁ 퐵푝 (푥) 푓 (푥) = 푝 (1 − 푝) 푓 (푥) 푥 ∈B푛 푘=0 푥 ∈B(푛,푘) 푛 ∑︁ 푛 ≤ 푝푘 (1 − 푝)푛−푘 = 1. 푘 푘=0 ∗ On the other hand, let 푓 (푥) = tB (푥), 푥 ∈ B be the extended test for tB (휉). For all integers 푁 > 0 let 푛 = b√︁푁/2c. Then as 푁 runs through the integers, 푛 also runs through all integers. For 푥 ∈ B푁 let 퐹(푥) = 푓 (푥 ≤푛). Since 푓 (푥) is lower semicomputable and monotonic with respect to the preﬁx relation, this is also true of 퐹(푥). Í We need to estimate 푥 ∈B(푁,퐾) 퐹(푥). For this, note that for 푦 ∈ B(푛, 푘) we have 푁 − 푛 |{푥 ∈ B(푁, 퐾) : 푦 v 푥 }| = . (4.2.3) 퐾 − 푘

Now for 0 ≤ 퐾 ≤ 푁 we have ∑︁ ∑︁ 퐹(푥) = 푓 ( 푦)|{푥 ∈ B(푁, 퐾) : 푦 v 푥 }| 푥 ∈B(푁,퐾) 푦 ∈B푛 푛 (4.2.4) ∑︁ 푁 − 푛 ∑︁ = 푓 ( 푦). 퐾 − 푘 푘=0 푦 ∈B(푛,푘)

푁−푛 푁 Let us estimate 퐾−푘 퐾 . If 퐾 = 푘 = 0 then this is 1. If 푘 = 푛 then it is 퐾···(퐾−푛+1) / 푁···(푁−푛+1) . Otherwise, using 푝 = 퐾 푁:

푁−푛 (푁 − 푛)(푁 − 푛 − 1)···(푁 − 퐾 − (푛 − 푘) + 1)/(퐾 − 푘)! 퐾−푘 = 푁 푁 (푁 − 1)···(푁 − 퐾 + 1)/퐾! 퐾 퐾 ···(퐾 − 푘 + 1)·(푁 − 퐾)···(푁 − 퐾 − (푛 − 푘) + 1) = . 푁 ···(푁 − 푛 + 1) 퐾푘 (푁 − 퐾)푛−푘 푁 푛 ≤ = 푝푘 (1 − 푝)푛−푘 . (4.2.5) (푁 − 푛)푛 푁 − 푛

96 4.3. Neutral measure

Thus in all cases the estimate 푁 − 푛 푁 푁 푛 ≤ 푝푘 (1 − 푝)푛−푘 퐾 − 푛 퐾 푁 − 푛 holds. We have 푛 푁 푛 푛 푁 = 1 + ≤ 푒 2(푁−푛) ≤ 푒2, 푁 − 푛 푁 − 푛 since we assumed 2푛2 ≤ 푁. Substituting into (4.2.4) gives

∑︁ ∑︁푛 ∑︁ 퐹(푥) ≤ 푒2 푝푘 (1 − 푝)푛−푘 푓 ( 푦) ≤ 푒2, 푥 ∈B(푁,퐾) 푘=0 푦 ∈B(푛,푘) since 푓 (푥) is an extended class test for Bernoulli measures. It follows that −2 ∗ ∗ ≤푛 푒 퐹(푥) < 푏(푥), hence also 퐹(푥) < 푏(푥). But we have tC (휉) = sup푛 푓 (휉 ) = ≤푛 ∗ sup푛 퐹(휉 ), hence tC (휉) < 푏(휉).

4.3 Neutral measure

Let t휇 (푥) be our universal uniform randomness test. We call a measure 푀 neutral if t푀 (푥) ≤ 1 for all 푥. If 푀 is neutral then no experimental outcome 푥 could refute the theory (hypothesis, model) that 푀 is the underlying measure to our experiments. It can be used as “apriori probability”, in a Bayesian approach to statistics. Levin’s theorem says the following: Theorem 4.3.1 If the space X is compact then there is a neutral measure over X. The proof relies on a nontrivial combinatorial fact, Sperner’s Lemma, which also underlies the proof of the Brouwer ﬁxpoint theorem. Here is a version of Sperner’s Lemma, spelled out in continuous form: 푛 Proposition 4.3.1 Let 푝1, . . . , 푝푘 be points of some ﬁnite-dimensional space R . Suppose that there are closed sets 퐹1, . . . , 퐹푘 with the property that for every subset

1 ≤ 푖1 < ··· < 푖푗 ≤ 푘 of the indices, the simplex 푆( 푝푖1 , . . . , 푝푖 푗 ) spanned by Ñ 푝푖1 , . . . , 푝푖 푗 is covered by the union 퐹푖1 ∪ · · · ∪ 퐹푖 푗 . Then the intersection 푖 퐹푖 of all these sets is not empty. The following lemma will also be needed. Lemma 4.3.2 For every closed set 퐴 ⊂ X and measure 휇, if 휇(퐴) = 1 then there is a point 푥 ∈ 퐴 with t휇 (푥) ≤ 1.

97 4. Generalizations

푥 Proof. This follows easily from 휇 푡휇 = 휇 1퐴 (푥)푡휇 (푥) ≤ 1.

Proof of Theorem 4.3.1. For every point 푥 ∈ X, let 퐹푥 be the set of measures for which t휇 (푥) ≤ 1. If we show that for every ﬁnite set of points 푥1, . . . , 푥푘, we have

퐹푥1 ∩ · · · ∩ 퐹푥푘 ≠ ∅, (4.3.1) then we will be done. Indeed, according to Proposition A.2.50, the compactness of X implies the compactness of the space M(X) of measures. Therefore if every ﬁnite subset of the family {퐹푥 : 푥 ∈ X} of closed sets has a nonempty intersection, then the whole family has a nonempty intersection: this intersection consists of the neutral measures. To show (4.3.1), let 푆(푥1, . . . , 푥푘) be the set of probability measures concentrated on 푥1, . . . , 푥푘. Lemma 4.3.2 implies that each such measure belongs to one of the sets 퐹푥푖 . Hence 푆(푥1, . . . , 푥푘) ⊂ 퐹푥1 ∪ · · · ∪ 퐹푥푘 , and the same holds for every subset of the indices {1, . . . , 푘}. Sperner’s Lemma 4.3.1 implies

퐹푥1 ∩ · · · ∩ 퐹푥푘 ≠ ∅. When the space is not compact, there are generally no neutral probability measures, as shown by the following example. Proposition 4.3.3 Over the discrete space X = N of natural numbers, there is no neutral measure.

Proof. It is suﬃcient to construct a randomness test 푡휇 (푥) with the property that for every measure 휇, we have sup푥 푡휇 (푥) = ∞. Let

∑︁ −푘 푡휇 (푥) = sup{푘 ∈ N : 휇( 푦) > 1 − 2 }. (4.3.2) 푦<푥

By its construction, this is a lower semicomputable function with sup푥 푡휇 (푥) = ∞. Í It is a test if 푥 휇(푥)푡휇 (푥) ≤ 1. We have

∑︁ ∑︁ ∑︁ ∑︁ −푘 휇(푥)푡휇 (푥) = 휇(푥) < 2 ≤ 1.

푥 푘>0 푡휇 (푥) ≥푘 푘>0 Using a similar construction over the space NN of inﬁnite sequences of natural numbers, we could show that for every measure 휇 there is a sequence 푥 with t휇 (푥) = ∞. Proposition 4.3.3 is a little misleading, since N can be compactiﬁed into N = N ∪ {∞} (as in Part1 of Example A.1.22). Theorem 4.3.1 implies that there is a

98 4.3. Neutral measure neutral probability measure 푀 over the compactified space N. Its restriction to Í N is, of course, not a probability measure, since it satisfies only 푥<∞ 푀(푥) ≤ 1. We called these functions semimeasures. Remark 4.3.4 1. It is easy to see that Theorem 4.6.1 characterizing randomness in terms of complexity holds also for the space N. 2. The topological space of semimeasures over N is not compact, and there is no neutral one among them. Its topology is not the same as what we get when we restrict the topology of probability measures over N to N. The difference is that over N, for example the set of measures {휇 : 휇(N) > 1/2} is closed, since N (as the whole space) is a closed set. But over N, this set is not necessarily closed, since N is not a closed subset of N. y Neutral measures are not too simple, even over N, as the following theorem shows. Theorem 4.3.2 There is no neutral measure over N that is upper semicomputable over N or lower semicomputable over N. Proof. Let us assume that 휈 is a measure that is upper semicomputable over N. Then the set {(푥, 푟) : 푥 ∈ N, 푟 ∈ Q, 휈(푥) < 푟} is recursively enumerable: let (푥푖, 푟푖) be a particular enumeration. For each 푛, −푛 −푛 let 푖(푛) be the first 푖 with 푟푖 < 2 , and let 푦푛 = 푥푖(푛) . Then 휈( 푦푛) < 2 , and at + the same time 퐾( 푦푛) < 퐾(푛). As mentioned, in Remark 4.3.4, Theorem 4.6.1 characterizing randomness in terms of complexity holds also for the space N. Thus, + + d휈 ( 푦푛) = − log 휈( 푦푛) − 퐾( 푦푛 | 휈) > 푛 − 퐾(푛). Suppose now that 휈 is lower semicomputable over N. The proof for this case is longer. We know that 휈 is the monotonic limit of a recursive sequence 푖 ↦→ 휈푖 (푥) 푛 of recursive semimeasures with rational values 휈푖 (푥). For every 푘 = 0,..., 2 −2, let

−푛 푛 −푛 푉푛,푘 = {휇 ∈ M(N) : 푘 · 2 < 휇({0,..., 2 − 1}) < (푘 + 2)· 2 }, 퐽 = {(푛, 푘) : 푘 · 2−푛 < 휈({0,..., 2푛 − 1})}.

The set 퐽 is recursively enumerable. Let us deﬁne the functions 푗 : 퐽 → N and 푛 푛 푥 : 퐽 → {0,..., 2 − 1} as follows: 푗(푛, 푘) is the smallest 푖 with 휈푖 ({0,..., 2 −

99 4. Generalizations

1}) > 푘 · 2−푛, and

푛 −푛+1 푥푛,푘 = min{ 푦 < 2 : 휈 푗(푛,푘) ( 푦) < 2 }. 푛−2 Let us deﬁne the function 푓휇 (푥, 푛, 푘) as follows. We set 푓휇 (푥, 푛, 푘) = 2 if the following conditions hold:

a) 휇 ∈ 푉푛,푘; b) 휇(푥) < 2−푛+2;

c) (푛, 푘) ∈ 퐽 and 푥 = 푥푛,푘.

Otherwise, 푓휇 (푥, 푛, 푘) = 0. Clearly, the function (휇, 푥, 푛, 푘) ↦→ 푓휇 (푥, 푛, 푘) is lower semicomputable. Condition (b) implies

∑︁ −푛+2 푛−2 휇( 푦) 푓휇 ( 푦, 푛, 푘) ≤ 휇(푥푛,푘) 푓휇 (푥푛,푘, 푛, 푘) < 2 · 2 = 1. (4.3.3) 푦

Let us show that 휈 ∈ 푉푛,푘 implies 푛−2 푓휈 (푥푛,푘, 푛, 푘) = 2 . (4.3.4)

Consider 푥 = 푥푛,푘. Conditions (a) and (c) are satisfied by definition. Let us show that condition (b) is also satisfied. Let 푗 = 푗(푛, 푘). By definition, we have −푛+1 휈 푗 (푥) < 2 . Since by definition 휈 푗 ∈ 푉푛,푘 and 휈 푗 ≤ 휈 ∈ 푉푛,푘, we have −푛+1 −푛+1 −푛+1 −푛+2 휈(푥) ≤ 휈 푗 (푥) + 2 < 2 + 2 = 2 . Since all three conditions (a), (b) and (c) are satisfied, we have shown (4.3.4). Now we define ∑︁ 1 ∑︁ 푔휇 (푥) = 푓휇 (푥, 푛, 푘). 푛(푛 + 1) 푛≥2 푘

Let us prove that 푔휇 (푥) is a uniform test. It is lower semicomputable by deﬁni- Í tion, so we only need to prove 푥 휇(푥) 푓휇 (푥) ≤ 1. For this, let 퐼푛,휇 = {푘 : 휇 ∈ 푉푛,푘 }. Clearly by deﬁnition, |퐼푛,휇 | ≤ 2. We have, using this last fact and the test property (4.3.3): ∑︁ ∑︁ 1 ∑︁ ∑︁ ∑︁ 1 휇(푥)푔휇 (푥) = 휇(푥) 푓휇 (푥, 푛, 푘) ≤ · 2 ≤ 1. 푛(푛 + 1) 푛(푛 + 1) 푥 푛≥2 푘∈퐼푛,휇 푥 푛≥2

Thus, 푔휇 (푥) is a uniform test. If 휈 ∈ 푉푛,푘 then we have

푛−2 ∗ 1 2 t휈 (푥푛,푘) > 푔휈 (푥푛,푘) ≥ 푓휇 (푥푛,푘, 푛, 푘) ≥ . 푛(푛 + 1) 푛(푛 + 1) Hence 휈 is not neutral.

100 4.4. Monotonicity, quasi-convexity/concavity

Remark 4.3.5 In [32] and [33], Levin imposed extra conditions on tests which allow to find a lower semicomputable neutral semimeasure. y The universal lower semicomputable semimeasure m(푥) has a certain property similar to neutrality. According to Theorem 2.3.4 specialized to one-element + sequences, for every computable measure 휇 we have d휇 (푥) = − log 휇(푥) − 퐾(푥) (where the constant in =+ depends on 휇). So, for computable measures, the expression d휇 (푥) = − log 휇(푥) − 퐾(푥) (4.3.5) can serve as a reasonable deficiency of randomness. (We will also use the test d t = 2 .) If we substitute m for 휇 in d휇 (푥), we get 0. This substitution is not justified, of course. The fact that m is not a probability measure can be helped, using compactification as above, and extending the notion of randomness tests. But the test d휇 can replace d휇 only for computable 휇, while m is not computable. Anyway, this is the sense in which all outcomes might be considered random with respect to m, and the heuristic sense in which m may be considered “neutral”.

4.4 Monotonicity, quasi-convexity/concavity

Some people find that 휇-tests as defined in Definition 4.1.1 are too general, in case 휇 is a non-computable measure. In particular, randomness with respect to computable measures has a certain—intuitively meaningful—monotonicity property: roughly, if 휈 is greater than 휇 then if 푥 is random with respect to 휇, it should also be random with respect to 휈. Proposition 4.4.1 For computable measures 휇, 휈 we have for all rational 푐 > 0:

−푘 + 2 휇 ≤ 휈 ⇒ d휈 (푥) < d휇 (푥) + 푘 + 퐾(푘). (4.4.1)

+ Here the constant in < depends on 휇, 휈, but not on 푘.

−푘 −푘 Proof. We have 1 ≥ 휈t휈 ≥ 2 휇t휈, hence 2 t휈 is a 휇-test. Using the method of Theorem 4.1.1 in ﬁnding universal tests, one can show that the sum

∑︁ −푘−퐾 (푘) 2 t휈 −푘 푘:2 휇t휈<1

∗ is a 휇-test, and hence < t휇. Therefore this is true of each member of the sum, which is just what the theorem claims.

101 4. Generalizations

There are other properties true for tests on computable measures that we may want to require for all measures. For the following properties, let us define quasi-convexity, which is a weakening of the notion of convexity. Definition 4.4.2 A function 푓 : 푉 → R defined on a vector space 푉 is called quasi-convex if for every real number 푥 the set {푣 : 푓 (푣) ≤ 푥 } is convex. It is quasi-concave if − 푓 is quasi-convex. y It is easy to see that quasi-convexity is equivalent to the inequality

푓 (휆푢 + (1 − 휆)푣) ≤ 푓 (푢) ∨ 푓 (푣) for all 푢, 푣 and 0 < 휆 < 1, while quasi-concavity is equivalent to

푓 (휆푢 + (1 − 휆)푣) ≥ 푓 (푢) ∧ 푓 (푣)

The uniform test with respect to computable measures is approximately both quasi-convex and quasi-concave. Let 휈 = 휆휇1 + (1 − 휆)휇2. Quasi-convexity means, roughly, that if 푥 is random with respect to both 휇1 and 휇2 then it is also random with respect to 휈. This property strengthens monotonicity in the cases where it applies.

Proposition 4.4.3 Let 휇1, 휇2 be computable measures and 0 < 휆 < 1 computable, with 휈 = 휆휇1 + (1 − 휆)휇2. Then we have

+ d휈 (푥) < d휇1 (푥) ∨ d휇2 (푥) + 퐾(휆).

Proof. The relation 1 ≥ 휈t휈 = 휆휇1t휈 + (1 − 휆)휇2t휈 implies 1 ≥ 휇푖t휈 for some + 푖 ∈ {1, 2}. Then d휈 < d휇푖 + 퐾(휆) (since 휆 was used to deﬁne 휈 and thus t휈). Quasi-concavity means, roughly, that if 푥 is non-random with respect to both 휇1 and 휇2 then it is also nonrandom with respect to 휈:

Proposition 4.4.4 Let 휇1, 휇2 be computable measures and 0 < 휆 < 1 arbitrary (not even necessarily computable), with 휈 = 휆휇1 + (1 − 휆)휇2. Then we have

+ d휈 > d휇1 ∧ d휇2 .

Proof. The function t휇1 ∧ t휇2 is lower semicomputable, and is a 휇푖-test for each ∗ ∗ 푖. Therefore it is also a 휈-test, and as such is < t휈. Here, the constant in the < depends only on (the programs for) 휇1, 휇2, and not on 휆. These properties do not survive for arbitrary measures and arbitrary constants.

102 4.5. Algorithmic entropy

Example 4.4.5 Let measure 휇1 be uniform over the interval [0, 1/2], let 휇2 be uniform over [1/2, 1]. For 0 ≤ 푥 ≤ 1 let

휈푥 = (1 − 푥)휇1 + 푥휇2.

Then 휈1/2 is uniform over [0, 1]. Let 휙푥 be the uniform distribution over [푥, 푥 + 1/2], and 휓푥 the uniform distribution over [0, 푥] ∪ [푥 + 1/2, 1]. Let 푝 < 1/2 be random with respect to the uniform distribution 휈1/2. 1. The relations

−1 휈1/2 ≤ 푝 휈푝, d휈1/2 ( 푝) < ∞, d휈푝 ( 푝) = ∞ show that any statement analogous to the monotonicity property of Proposi- tion 4.4.1 fails when the measures involved are not required to be computable. 2. For rational 푟 with 0 < 푟 < 푝 the relations

휈 := (1 − 푝)휈푟 + 푝휈1−푟, d휈 ( 푝) = ∞, d휈푟 ( 푝) < ∞, d휈1−푟 ( 푝) < ∞ provide a similar counterexample to Proposition 4.4.3. 3. The relations

휈1/2 = (휈푝 + 휈1−푝)/2, d휈1/2 ( 푝) < ∞, d휈푝 ( 푝) = d휈1−푝 ( 푝) = ∞ provide a similar counterexample to Proposition 4.4.4. The following counterexample relies on a less subtle eﬀect:

휈1/2 = (휙푝 + 휓푝)/2, d휈1/2 ( 푝) < ∞, d휙푝 ( 푝) = d휓푝 ( 푝) = ∞,

since as a boundary point of the support, 푝 is computable from both 휙푝 and 휓푝 in a uniform way. For a complete proof, uniform tests must be provided for each of the cases: this is left as exercise for the reader. y The non-monotonicity example could be used to argue that the we allowed too many 휇-tests, that the test t휇 (푥) should not be allowed to depend on properties of 휇 that exploit the computational properties of 휇 so much stronger than its quantitative properties.

4.5 Algorithmic entropy

Some properties of description complexity make it a good expression of the idea of individual information content.

103 4. Generalizations

4.5.1 Entropy The entropy of a discrete probability distribution 휇 is deﬁned as ∑︁ H(휇) = − 휇(푥) log 휇(푥). 푥

To generalize entropy to continuous distributions the relative entropy is defined as follows. Let 휇, 휈 be two measures, where 휇 is taken (typically, but not always), to be a probability measure, and 휈 another measure, that can also be a probability measure but is most frequently not. We define the relative entropy H휈 (휇) as follows. If 휇 is not absolutely continuous with respect to 휈 then H휈 (휇) = −∞. Otherwise, writing 푑휇 휇(푑푥) = =: 푓 (푥) 푑휈 휈(푑푥) for the (Radon-Nikodym) derivative (density) of 휇 with respect to 휈, we define ∫ 푑휇 푥 휇(푑푥) 푥 H휈 (휇) = − log 푑휇 = −휇 log = −휈 푓 (푥) log 푓 (푥). 푑휈 휈(푑푥)

Thus, H(휇) = H# (휇) is a special case. Example 4.5.1 Let 푓 (푥) be a probability density function for the distribution 휇 over the real line, and let 휆 be the Lebesgue measure there. Then ∫ H휆 (휇) = − 푓 (푥) log 푓 (푥)푑푥.

y In information theory and statistics, when both 휇 and 휈 are probability measures, then −H휈 (휇) is also denoted 퐷(휇 k 휈), and called (after Kullback) the information divergence of the two measures. It is frequently used in the role of a distance between 휇 and 휈. It is not symmetric, but can be shown to obey the triangle inequality, and to be nonnegative. Let us prove the latter property: in our terms, it says that relative entropy is nonpositive when both 휇 and 휈 are probability measures. Proposition 4.5.2 Over a space 푋, we have

휇(푋) H휈 (휇) ≤ −휇(푋) log . (4.5.1) 휈(푋)

In particular, if 휇(푋) ≥ 휈(푋) then H휈 (휇) ≤ 0.

104 4.5. Algorithmic entropy

Proof. The inequality −푎 ln 푎 ≤ −푎 ln 푏 + 푏 − 푎 expresses the concavity of the logarithm function. Substituting 푎 = 푓 (푥) and 푏 = 휇(푋)/휈(푋) and integrating by 휈:

푥 휇(푋) 휇(푋) (ln 2)H휈 (휇) = −휈 푓 (푥) ln 푓 (푥) ≤ −휇(푋) ln + 휈(푋) − 휇(푋) 휈(푋) 휈(푋) 휇(푋) = −휇(푋) ln , 휈(푋) giving (4.5.1).

4.5.2 Algorithmic entropy Le us recall some facts on description complexity. Let us fix some (finite or infinite) alphabet Σ and consider the discrete space Σ∗. The universal lower semicomputable semimeasure m(푥) over Σ∗ was defined in Definition 1.6.9. It is possible to turn m(푥) into a measure, by compactifying the discrete space Σ∗ into Σ∗ = Σ∗ ∪ {∞} (as in part1 of Example A.1.22; this process makes sense also for a constructive Í discrete space), and setting m(∞) = 1 − 푥 ∈Σ∗ m(푥). The extended measure m is not quite lower semicomputable since the number 휇(Σ∗\{0}) is not necessarily lower semicomputable. Remark 4.5.3 A measure 휇 is computable over Σ∗ if and only if the function 푥 ↦→ 휇(푥) is computable for 푥 ∈ Σ∗. This property does not imply that the number ∑︁ 1 − 휇(∞) = 휇(Σ∗) = 휇(푥) 푥 ∈Σ∗ is computable. y Let us allow, for a moment, measures 휇 that are not probability measures: they may not even be finite. Metric and computability can be extended to this case, the universal test t휇 (푥) can also be generalized. The Coding Theorem 1.6.5 and other considerations suggest the introduction of the following notation, for an arbitrary measure 휇: Definition 4.5.4 We define the algorithmic entropy of a point 푥 with respect to measure 휇 as 퐻휇 (푥) = −d휇 (푥) = − log t휇 (푥). (4.5.2) y

105 4. Generalizations

Then, with # deﬁned as the counting measure over the discrete set Σ∗ (that is, #(푆) = |푆|), we have + 퐾(푥) = 퐻# (푥).

This allows viewing 퐻휇 (푥) as a generalization of description complexity. The following theorem generalizes an earlier known theorem stating that over a discrete space, for a computable measure, entropy is within an additive constant the same as “average complexity”: H(휇) =+ 휇푥 퐾(푥). Theorem 4.5.1 Let 휇 be a probability measure. Then we have

푥 H휈 (휇) ≤ 휇 퐻휈 (푥 | 휇). (4.5.3) If 푋 is a discrete space then the following estimate also holds:

+ 푥 H휈 (휇) > 휇 퐻휈 (푥 | 휇). (4.5.4)

Proof. Let 훿 be the measure with density t휈 (푥 | 휇) with respect to 휈: t휈 (푥 | 휇) = 훿(푑푥) ( ) ≤ ( | ) 휈(푑푥) . Then 훿 푋 1. It is easy to see from the maximality property of t휈 푥 휇 ( | ) 휈(푑푥) that t휈 푥 휇 > 0, therefore according to Proposition A.2.23, we have 훿(푑푥) = −1 훿(푑푥) 휈(푑푥) . Using Proposition A.2.23 and 4.5.2:

푥 휇(푑푥) H휈 (휇) = −휇 log , 휈(푑푥)

푥 푥 훿(푑푥) 푥 휈(푑푥) −휇 퐻휈 (푥 | 휇) = 휇 log = −휇 log , 휈(푑푥) 훿(푑푥)

푥 푥 휇(푑푥) 휇(푋) H휈 (휇) − 휇 퐻휈 (푥 | 휇) = −휇 log ≤ −휇(푋) log ≤ 0. 훿(푑푥) 훿(푋) This proves (4.5.3). ( ) ↦→ 휇 (푑푥) 휇 (푥) Over a discrete space 푋, the function 푥, 휇, 휈 휈(푑푥) = 휈(푥) is computable, ( | ) 휇 (푑푥) ∗ ( | ) therefore by the maximality property of 퐻휈 푥 휇 we have 휈(푑푥) < t휈 푥 휇 , ( ) − 푥 휇 (푑푥) + 푥 ( | ) hence H휈 휇 = 휇 log 휈(푑푥) > 휇 퐻휈 푥 휇 .

4.5.3 Addition theorem The Addition Theorem (3.1.5) can be generalized to the algorithmic entropy 퐻휇 (푥) introduced in (4.5.2) (a somewhat similar generalization appeared in [54]). The generalization, deﬁning 퐻휇,휈 = 퐻휇×휈, is

+ 퐻휇,휈 (푥, 푦) = 퐻휇 (푥 | 휈) + 퐻휈 ( 푦 | 푥, 퐻휇 (푥 | 휈), 휇). (4.5.5)

106 4.5. Algorithmic entropy

Before proving the general addition theorem, we establish a few useful facts. Proposition 4.5.5 We have

+ 푦 −퐻휇,휈 (푥,푦) 퐻휇 (푥 | 휈) < − log 휈 2 .

Proof. The function 푓 (푥, 휇, 휈) that is the right-hand side, is upper semicomputable by deﬁnition, and obeys 휇푥2− 푓 (푥,휇,휈) ≤ 1. Therefore the inequality follows from the minimum property of 퐻휇 (푥). Let 푧 ∈ N, then the inequality

+ 퐻휇 (푥) < 퐾(푧) + 퐻휇 (푥 | 푧) (4.5.6) will be a simple consequence of the general addition theorem. The following lemma, needed in the proof of the theorem, generalizes this inequality somewhat: Lemma 4.5.6 For a computable function ( 푦, 푧) ↦→ 푓 ( 푦, 푧) over N, we have

+ 퐻휇 (푥 | 푦) < 퐾(푧) + 퐻휇 (푥 | 푓 ( 푦, 푧)).

Proof. The function

∑︁ −퐻휇 (푥 | 푓 ( 푦,푧))−퐾 (푧) (푥, 푦, 휇) ↦→ 푔휇 (푥, 푦) = 2 푧

푥 Í −퐾 (푧) ∗ is lower semicomputable, and 휇 푔휇 (푥, 푦) ≤ 푧 2 ≤ 1. Hence 푔휇 (푥, 푦) < 2−퐻휇 (푥 | 푦) . The left-hand side is a sum, hence the inequality holds for each element of the sum: just what we had to prove. The following monotonicity property will be needed: Lemma 4.5.7 For 푖 < 푗 we have

+ 푖 + 퐻휇 (푥 | 푖) < 푗 + 퐻휇 (푥 | 푗).

Proof. From Lemma 4.5.6, with 푓 (푖, 푛) = 푖 + 푛 we have

+ + 퐻휇 (푥 | 푖) − 퐻휇 (푥 | 푗) < 퐾( 푗 − 푖) < 푗 − 푖.

Let us generalize the minimum property of 퐻휇 (푥).

107 4. Generalizations

Proposition 4.5.8 Let ( 푦, 휈) ↦→ 퐹휈 ( 푦) be an upper semicomputable function with values in Z = Z ∪ {−∞, ∞}. Then by Corollary 4.1.3 among the lower semicom- 푥 −퐹휈 ( 푦) putable functions (푥, 푦, 휈) ↦→ 푔휈 (푥, 푦) with 휈 푔휈 (푥, 푦) ≤ 2 there is one that is maximal to within a multiplicative constant. Choosing 푓휈 (푥, 푦) as such a function we have for all 푥 with 퐹휈 ( 푦) > −∞:

∗ −퐹휈 ( 푦) 푓휈 (푥, 푦) = 2 t휈 (푥 | 푦, 퐹휈 ( 푦)),

+ or in logarithmic notation − log 푓휈 (푥, 푦) = 퐹휈 ( 푦) + 퐻휈 (푥 | 푦, 퐹휈 ( 푦)). ∗ Proof. To prove the inequality >, deﬁne −푖 푔휈 (푥, 푦, 푚) = max 2 t휈 (푥 | 푦, 푖). 푖≥푚

Function 푔휈 (푥, 푦, 푚) is lower semicomputable and decreasing in 푚. Therefore

푔휈 (푥, 푦) = 푔휈 (푥, 푦, 퐹휈 ( 푦)) is also lower semicomputable since it is obtained by substituting an upper semicomputable function for 푚 in 푔휈 (푥, 푦, 푚). The multiplicative form of Lemma 4.5.7 implies ∗ −푚 푔휈 (푥, 푦, 푚) = 2 t휈 (푥 | 푦, 푚),

∗ −퐹휈 ( 푦) 푔휈 (푥, 푦) = 2 t휈 (푥 | 푦, 퐹휈 ( 푦)).

We have, since t휈 is a test: 푥 −푚 −푚 휈 2 t휈 (푥 | 푦, 푚) ≤ 2 , ∗ 푥 −퐹휈 ( 푦) 휈 푔휈 (푥, 푦) < 2 ,

∗ implying 푔휈 (푥, 푦) < 푓휈 (푥, 푦) by the optimality of 푓휈 (푥, 푦). To prove the upper bound, consider all lower semicomputable functions 휙푒 (푥, 푦, 푚, 휈) (푒 = 1, 2,... ). By Theorem 4.1.1, there is a recursive mapping 푒 ↦→ 푒0 with the 푥 −푚+1 푥 property that 휈 휙푒0 (푥, 푦, 푚, 휈) ≤ 2 , and for each 푦, 푚, 휈 if 휈 휙푒 (푥, 푦, 푚, 휈) < −푚+1 2 then 휙푒 = 휙푒0. Let us apply this transformation to the function 휙푒 (푥, 푦, 푚, 휈) = 0 푓휈 (푥, 푦). The result is a lower semicomputable function 푓휈 (푥, 푦, 푚) = 휙푒0 (푥, 푦, 푚, 휈) 푥 0 −푚+1 푥 −푚 with the property that 휈 푓휈 (푥, 푦, 푚) ≤ 2 , further 휈 푓휈 (푥, 푦) ≤ 2 implies 0 푚−1 0 푓휈 (푥, 푦, 푚) = 푓휈 (푥, 푦). Now (푥, 푦, 푚, 휈) ↦→ 2 푓휈 (푥, 푦, 푚) is a uniform test of ∗ 푥 conditional on 푦, 푚 and hence it is < t휈 (푥 | 푦, 푚). Substituting 퐹휈 ( 푦) for 푚 the 푥 −푚 relation 휈 푓휈 (푥, 푦) ≤ 2 is satisﬁed and hence we have ∗ 0 −퐹휈 ( 푦)+1 푓휈 (푥, 푦) = 푓휈 (푥, 푦, 퐹휈 ( 푦)) < 2 t휈 (푥 | 푦, 퐹휈 ( 푦)).

108 4.5. Algorithmic entropy

As mentioned above, the theory generalizes to measures that are not probability measures. Taking 푓휇 (푥, 푦) = 1 in Proposition 4.5.8 gives the inequality

+ 퐻휇 (푥 | blog 휇(푋)c) < log 휇(푋), with a physical meaning when 휇 is the phase space measure. Using (4.5.6), this implies + 퐻휇 (푥) < log 휇(푋) + 퐾(blog 휇(푋)c). (4.5.7) Theorem 4.5.2 (General addition) The following inequality holds:

+ 퐻휇,휈 (푥, 푦) = 퐻휇 (푥 | 휈) + 퐻휈 ( 푦 | 푥, 퐻휇 (푥 | 휈), 휇).

+ Proof. To prove the inequality < deﬁne 퐺휇,휈 (푥, 푦, 푚) = min푖≥푚 푖+퐻휈 ( 푦 | 푥, 푖, 휇). This function is upper semicomputable and increasing in 푚. Therefore function

퐺휇,휈 (푥, 푦) = 퐺휇,휈 (푥, 푦, 퐻휇 (푥 | 휈)) is also upper semicomputable since it is obtained by substituting an upper semicomputable function for 푚 in 퐺휇,휈 (푥, 푦, 푚). Lemma 4.5.7 implies

+ 퐺휇,휈 (푥, 푦, 푚) = 푚 + 퐻휈 ( 푦 | 푥, 푚, 휇), + 퐺휇,휈 (푥, 푦) = 퐻휇 (푥 | 휈) + 퐻휈 ( 푦 | 푥, 퐻휇 (푥 | 휈), 휇).

Now, we have

휈푦2−푚−퐻휈 ( 푦 | 푥,푚,휇) ≤ 2−푚, ∗ 휈푦2−퐺휇,휈 (푥,푦) < 2−퐻휇 (푥 | 휇) .

푥 푦 −퐺 ∗ + Integrating over 푥 by 휇 gives 휇 휈 2 < 1, implying 퐻휇,휈 (푥, 푦) < 퐺휇,휈 (푥, 푦) by + the minimality property of 퐻휇,휈 (푥, 푦). This proves the < half of the theorem. + −퐻휇,휈 (푥,푦) To prove the inequality > let 푓휈 (푥, 푦, 휇) = 2 . Proposition 4.5.5 푦 −퐻휇 (푥 | 휈)+푐 implies that there is a constant 푐 with 휈 푓휈 (푥, 푦, 휇) ≤ 2 . Let

퐹휈 (푥, 휇) = 퐻휇 (푥 | 휈).

Proposition 4.5.8 gives (substituting 푦 for 푥 and (푥, 휇) for 푦):

+ 퐻휇,휈 (푥, 푦) = − log 푓휈 (푥, 푦, 휇) > 퐹휈 (푥, 휇) + 퐻휈 ( 푦 | 푥, 퐹휈 (푥, 휇), 휇), which is what needed to be proved.

109 4. Generalizations

The function 퐻휇 (휔) behaves quite diﬀerently for diﬀerent kinds of measures 휇. Recall the following property of complexity: + + 퐾( 푓 (푥) | 푦) < 퐾(푥 | 푔( 푦)) < 퐾(푥). (4.5.8) for any computable functions 푓 , 푔. This implies + 퐾( 푦) < 퐾(푥, 푦). In contrast, if 휇 is a probability measure then + 퐻휈 ( 푦) > 퐻휇,휈 (휔, 푦). This comes from the fact that 2−퐻휈 ( 푦) is a test for 휇 × 휈.

4.5.4 Information Mutual information has been defined in Definition 3.1.9 as 퐼∗(푥 : 푦) = 퐾(푥) + 퐾( 푦)−퐾(푥, 푦). By the Addition theorem, we have 퐼∗(푥 : 푦) =+ 퐾( 푦)−퐾( 푦 | 푥, 퐾(푥)) =+ 퐾(푥) − 퐾(푥 | 푦, 퐾( 푦)). The two latter expressions show that in some sense, 퐼∗(푥 : 푦) is the information held in 푥 about 푦 as well as the information held in 푦 about 푥. (The terms 퐾(푥), 퐾( 푦) in the conditions are logarithmic-sized corrections to this idea.) Using (4.3.5), it is interesting to view mutual information 퐼∗(푥 : 푦) as a deficiency of randomness of the pair (푥, 푦) in terms of the expression d휇, with respect to m × m: ∗ 퐼 (푥 : 푦) = 퐾(푥) + 퐾( 푦) − 퐾(푥, 푦) = dm×m (푥, 푦). Taking m as a kind of “neutral” probability, even if it is not quite such, allows us to view 퐼∗(푥 : 푦) as a “deficiency of independence”. Is it also true that ∗ + 퐼 (푥 : 푦) = dm×m (푥)? This would allow us to deduce, as Levin did, “information conservation” laws from randomness conservation laws.2 Expression dm×m (푥) must be understood again in the sense of compactification, as in Section 4.3. There seem to be two reasonable ways to compactify the space N × N: we either compactify it directly, by adding a symbol ∞, or we form the product N × N. With either of them, preserving Theorem 4.6.1, we would have to check whether 퐾(푥, 푦 | m × m) =+ 퐾(푥, 푦). But, knowing the function m(푥) × m( 푦) we know the function 푥 ↦→ m(푥) =∗ m(푥) × m(0), hence also the function (푥, 푦) ↦→ m(푥, 푦) = m(h푥, 푦i). Using this knowledge, it is possible to develop an argument similar to the proof of Theorem 4.3.2, showing that 퐾(푥, 푦 | m × m) =+ 퐾(푥, 푦) does not hold. 2 We cannot use the test t휇 for this, since it can be shown easily that it does not to obey randomness conservation.

110 4.6. Randomness and complexity

Question 1 Is there a neutral measure 푀 with the property that 퐼∗(푥 : 푦) = d푀×푀 (푥, 푦)? Is this true maybe for all neutral measures 푀? If not, how far apart ∗ are the expressions d푀×푀 (푥, 푦) and 퐼 (푥 : 푦) from each other?

4.6 Randomness and complexity

We have seen in the discrete case that complexity and randomness are closely related. The connection is more delicate technically in the continuous case, but its exploration led to some nice results.

4.6.1 Discrete space

It is known that for computable 휇, the test d휇 (푥) can be expressed in terms of the description complexity of 푥 (we will prove these expressions below). Assume that X is the (discrete) space of all binary strings. Then we have

d휇 (푥) = − log 휇(푥) − 퐾(푥) + 푂(퐾(휇)). (4.6.1) The meaning of this equation is the following. The expression − log 휇(푥) is an upper bound (within 푂(퐾(휇))) of the complexity 퐾(푥), and nonrandomness of 푥 is measured by the difference between the complexity and this upper bound. Assume that X is the space of infinite binary sequences. Then equation (4.6.1) must be replaced with ≤푛 ≤푛 d휇 (푥) = sup − log 휇(푥 ) − 퐾(푥 ) + 푂(퐾(휇)) . (4.6.2) 푛 For noncomputable measures, we cannot replace 푂(퐾(휇)) in these relations with anything finite, as shown in the following example. Therefore however attractive and simple, exp(− log 휇(푥)−퐾(푥)) is not a universal uniform test of randomness. Proposition 4.6.1 There is a measure 휇 over the discrete space X of binary strings + such that for each 푛, there is an 푥 with d휇 (푥) = 푛− 퐾(푛) and − log 휇(푥) − 퐾(푥) < 0. Proof. Let us treat the domain of our measure 휇 as a set of pairs (푥, 푦). Let 푛 푥푛 = 0 , for 푛 = 1, 2,.... For each 푛, let 푦푛 be some binary string of length 푛 −푛 with the property 퐾(푥푛, 푦푛) > 푛. Let 휇(푥푛, 푦푛) = 2 , and 0 elsewhere. Then − log 휇(푥푛, 푦푛) − 퐾(푥푛, 푦푛) ≤ 푛 − 푛 = 0. On the other hand, let 푡휇 (푥, 푦) be the test nonzero only on pairs of strings (푥, 푦) of the form (푥푛, 푦): m(푛) 푡휇 (푥푛, 푦) = Í . 푧∈B푛 휇(푥푛, 푧)

111 4. Generalizations

The form of the deﬁnition ensures semicomputability and we also have ∑︁ ∑︁ 휇(푥, 푦)푡휇 (푥, 푦) ≤ m(푛) < 1, 푥,푦 푛

∗ therefore 푡휇 is indeed a test. Hence t휇 (푥, 푦) > 푡휇 (푥, 푦). Taking logarithms, + d휇 (푥푛, 푦푛) > 푛 − 퐾(푛). The same example shows that the test deﬁned as exp(− log 휇(푥) − 퐾(푥)) over discrete sets, does not satisfy the randomness conservation property.

Proposition 4.6.2 The test deﬁned as 푓휇 (푥) = exp(− log 휇(푥) − 퐾(푥)) over discrete spaces X does not obey the conservation of randomness.

Proof. Let us use the example of Proposition 4.6.1. Consider the function 휋 : (푥, 푦) ↦→ 푥. The image of the measure 휇 under the projection is Í −푛 (휋휇)(푥) = 푦 휇(푥, 푦). Thus, (휋휇)(푥푛) = 휇(푥푛, 푦푛) = 2 . Then we have seen that log 푓휇 (푥푛, 푦푛) ≤ 0. On the other hand,

+ log 푓휋휇 (휋(푥푛, 푦푛)) = − log(휋휇)(푥푛) − 퐾(푥푛) = 푛 − 퐾(푛).

Thus, the projection 휋 takes a random pair (푥푛, 푦푛) into an object 푥푛 that is very nonrandom (when randomness is measured using the test 푓휇). In the example, we have the abnormal situation that a pair is random but one of its elements is nonrandom. Therefore even if we would not insist on universality, the test exp(− log 휇(푥) − 퐾(푥)) is unsatisfactory. Looking into the reasons of the nonconservation in the example, we will notice that it could only have happened because the test 푓휇 is too special. The fact that − log(휋휇)(푥푛)−퐾(푥푛) is large should show that the pair (푥푛, 푦푛) can be enclosed into the “simple” set {푥푛} × Y of small probability; unfortunately, this observation does not reflect on − log 휇(푥, 푦) − 퐾(푥, 푦) (it does for computable 휇). It is a natural idea to modify equation (4.6.1) in such a way that the complexity 퐾(푥) is replaced with 퐾(푥 | 휇). However, this expression must be understood properly. We need to use the definition of 퐾(푥) as − log m(푥) directly, and not as prefix complexity. Let us mention the following easy fact: Proposition 4.6.3 If 휇 is a computable measure then 퐾(푥 | 휇) =+ 퐾(푥). The constant in =+ depends on the description complexity of 휇.

112 4.6. Randomness and complexity

Theorem 4.6.1 If X is the discrete space Σ∗ then we have

+ d휇 (푥) = − log 휇(푥) − 퐾(푥 | 휇). (4.6.3)

Note that in terms of the algorithmic entropy notation introduced in (4.5.2), this theorem can be expressed as

+ 퐻휇 (푥) = 퐾(푥 | 휇) + log 휇(푥).

∗ Proof. In exponential notation, equation (4.6.3) can be written as t휇 (푥) = m(푥 | 휇)/휇(푥). ∗ Let us prove > first. We will show that the right-hand side of this inequality is a ∗ test, and hence < t휇 (푥). However, the right-hand side is clearly lower semicomputable in (푥, 휇) and when we “integrate” it (multiply it by 휇(푥) and sum it), its sum is ≤ 1; thus, it is a test. ∗ Let us prove < now. The expression t휇 (푥)휇(푥) is clearly lower semicom- + putable in (푥, 휇), and its sum is ≤ 1. Hence, it is < m(푥 | 휇). Remark 4.6.4 It important not to consider relative computation with respect to 휇 as oracle computation in the ordinary sense. Theorem 4.3.1 below will show the existence of a measure with respect to which every element is random. If randomness is defined using 휇 as an oracle then we can always find elements nonrandom with respect to 휇. For similar reasons, the proof of the Coding Theorem does not transfer to the function 퐾(푥 | 휇) since an interpreter function should have the property of intensionality, depending only on 휇 and not on the sequence representing it. (It does transfer without problem to an oracle version of 퐾휇 (푥).) The Coding Theorem still may hold, at least in some cases: this is currently not known. Until we know this, we cannot interpret 퐾(푥 | 휇) as description complexity in terms of interpreters and codes. (Thanks to Alexander Shen for this observation: this remark corrects an error in the paper [21].) y

4.6.2 Non-discrete spaces For non-discrete spaces, unfortunately, we can only provide a less intuitive expression. Proposition 4.6.5 let X = (푋, 푑, 퐷, 훼) be a complete computable metric space, and let E be the enumerated set of bounded Lipschitz functions introduced in (A.1.3),

113 4. Generalizations

but for the space M(X) ×X. The uniform test of randomness t휇 (푥), can be expressed as ∗ ∑︁ m( 푓 | 휇) t휇 (푥) = 푓 (휇, 푥) . (4.6.4) 휇 푦 푓 (휇, 푦) 푓 ∈E

∗ Proof. For >, we will show that the right-hand side of the inequality is a test, ∗ and hence < t휇 (푥). For simplicity, we skip the notation about the enumeration of E and treat each element 푓 as its own name. Each term of the sum is clearly lower semicomputable in ( 푓 , 푥, 휇), hence the sum is lower semicomputable in (푥, 휇). It remains to show that the 휇-integral of the sum is ≤ 1. But, the 휇- integral of the generic term is ≤ m( 푓 | 휇), and the sum of these terms is ≤ 1 by the deﬁnition of the function m(· | ·). Thus, the sum is a test. ∗ For <, note that (휇, 푥) ↦→ t휇 (푥), as a lower semicomputable function, is the supremum of functions in E. Denoting their diﬀerences by 푓푖 (휇, 푥), we have Í Í 푥 t휇 (푥) = 푖 푓푖 (휇, 푥). The test property implies 푖 휇 푓푖 (휇, 푥) ≤ 1. Since the 푥 푥 ∗ function (휇, 푖) ↦→ 휇 푓푖 (휇, 푥) is lower semicomputable, this implies 휇 푓푖 (휇, 푥) < m(푖 | 휇), and hence ∗ m(푖 | 휇) 푓 (휇, 푥) < 푓 (휇, 푥) . 푖 푖 푥 휇 푓푖 (휇, 푥) It is easy to see that for each 푓 ∈ E we have ∑︁ m(푖 | 휇) ≤ 휇( 푓 | 휇),

푖: 푓푖= 푓 which leads to (4.6.4). ∗ Remark 4.6.6 If we only want the > part of the result, then E can be replaced with any enumerated computable sequence of bounded computable functions. y

4.6.3 Inﬁnite sequences In case of the space of inﬁnite sequences and a computable measure, Theo- rem 2.3.4 gives a characterization of randomness in terms of complexity. This theorem does not seem to transfer to a more general situation, but under some conditions, at least parts of it can be extended. For arbitrary measures and spaces, we can say a little less:

114 4.6. Randomness and complexity

Proposition 4.6.7 For all measures 휇 ∈ M푅 (푋), for the deﬁciency of randomness d휇 (푥), we have

+ ≤푛 ≤푛 d휇 (푥) > sup − log 휇(푥 ) − 퐾(푥 | 휇) . (4.6.5) 푛

Proof. Consider the function

∑︁ m(푠 | 휇) ∑︁ m(푥 ≤푛 | 휇) m(푥 ≤푛 | 휇) 푓휇 (푥) = 1Γ (푥) = ≥ sup . 푠 휇(Γ ) 휇(푥 ≤푛) 휇(푥 ≤푛) 푠 푠 푛 푛

The function (휇, 푥) ↦→ 푓휇 (푥) is clearly lower semicomputable and satisﬁes 푥 휇 푓휇 (푥) ≤ 1, and hence

+ + ≤푛 ≤푛 d휇 (푥) > log 푓 (푥) > sup − log 휇(푥 ) − 퐾(푥 | 휇) . 푛

Definition 4.6.8 Let M푅 (푋) be the set of measures 휇 with 휇(푋) = 푅. y + We will be able to prove the > part of the statement of Theorem 2.3.4 in a more general space, and without assuming computability. Assume that a separating sequence 푏1, 푏2,... is given as defined in Subsection 4.7, along with the 0 0 set 푋 . For each 푥 ∈ 푋 , the binary sequence 푥1, 푥2,... has been defined. Let

∑︁ 0 0 휇(Γ푠) = 푅 − {휇(Γ푠0) : 푙(푠) = 푙(푠 ), 푠 ≠ 푠}.

Then (푠, 휇) ↦→ 휇(Γ푠) is lower semicomputable, and (푠, 휇) ↦→ 휇(Γ푠) is upper semicomputable. And, every time that the functions 푏푖 (푥) form a partition with 휇-continuity, we have 휇(Γ푠) = 휇(Γ푠) for all 푠. Theorem 4.6.2 Suppose that the space 푋 is eﬀectively compact. Then for all com- 0 putable measures 휇 ∈ M푅 (푋), for the deﬁciency of randomness d휇 (푥), the characterization (2.3.3) holds.

+ Proof. The proof of part > of the inequality follows directly from Proposition 4.6.7, just as in the proof of Theorem 2.3.4. + The proof of < is also similar to the proof of that theorem. The only part that needs to be reproved is the statement that for every lower semicomputable ∗ function 푓 over 푋, there are computable sequences 푦푖 ∈ N and 푞푖 ∈ Q with

푓 (푥) = sup푖 푞푖1푦푖 (푥). This follows now, since according to Proposition 4.7.7, the cells Γ푦 form a basis of the space 푋.

115 4. Generalizations

4.6.4 Bernoulli tests In this part, we will give characterize Bernoulli sequences in terms of their complexity growth. Recall Definition 4.2.11: A function 푓 : B∗ → R is a combinatorial Bernoulli test if a) It is lower semicomputable. b) It is monotonic with respect to the prefix relation. ≤ ≤ Í ( ) ≤ 푛 c) For all 0 푘 푛 we have 푥 ∈B(푛,푘) 푓 푥 푘 . According to Theorem 4.2.3, a sequence 휉 is nonrandom with respect to all ≤푛 Bernoulli measures if and only if sup푛 푏(휉 ) = ∞, where 푏(푥) is a combinatorial Bernoulli test. We need some definitions. Í푛 Definition 4.6.9 For a finite or infinite sequence 푥 let 푆푛 (푥) = 푖=1 푥(푖). ≤ ≤ ≤ ≤ ( ) 푛 푘 ( − )푛−푘 For 0 푝 1 and integers 0 푘 푛, denote 퐵푝 푛, 푘 = 푘 푝 1 푝 . An upper semicomputable function 퐷 : N2 → N, defined for 푛 ≥ 1, 0 ≤ 푘 ≤ 푛 will be called a gap function if 푛 ∑︁ ∑︁ −퐷(푛,푘) 퐵푝 (푛, 푘)2 ≤ 1 (4.6.6) 푛≥1 푘=0 holds for all 0 ≤ 푝 ≤ 1. A gap function 퐷(푛, 푘) is optimal if for every other gap 0 0 function 퐷 (푛, 푘) there is a 푐퐷0 with 퐷(푛, 푘) ≤ 퐷 (푛, 푘) + 푐퐷0. y + Proposition 4.6.10 There is an optimal gap function 퐷(푛, 푘) < 퐾(푛). Proof. The existence is proved using the technique of Theorem 4.1.1. For the inequality it is sufficient to note that 퐾(푛) is a gap function. Indeed, we have 푛 ∑︁ ∑︁ −퐾 (푛) ∑︁ −퐾 (푛) 퐵푝 (푛, 푘)2 = 2 ≤ 1. 푛≥1 푘=0 푛≥1 Definition 4.6.11 Let us fix an optimal gap function and denote it by Δ(푛, 푘). y Now we can state the test characterization theorem for Bernoulli tests. Theorem 4.6.3 Denoting by 푏(휉) the universal class test for the Bernoulli sequences, we have 푏(휉) =∗ 푏(휉), where 푛 log 푏(휉) = sup log − 퐾(휉≤푛 | 푛, 푘, Δ(푛, 푘)) − Δ(푛, 푘), 푛 푘

116 4.6. Randomness and complexity

with 푘 = 푆푛 (휉). Proof. Let 훿(푛, 푘) = 2−Δ(푛,푘) . ∗ Claim 4.6.12 Consider lower semicomputable functions 훾 : B → R+ such that for all 푛, 푘 we have ∑︁ 훾( 푦) ≤ 2−Δ(푛,푘) . 푦 ∈B(푛,푘)

Among these functions there is one that is optimal (maximal to within a multiplicative constant). Calling it 훿( 푦) we have

훿( 푦) =∗ 훿(푛, 푘)· m( 푦 | 푛, 푘, Δ(푛, 푘)).

Thus, the right-hand side is equal, within a multiplicative constant, to a lower semicomputable function of 푦. Proof. This follows immediately from Proposition 4.5.8. with 휈 = # (the counting measure) over the sets B(푛, 푘). ∗ Let us show 푏(휉) < 푏(휉). We have with 푘 = 푆푛 (휉) in the ﬁrst line:

푛 푏(휉) = sup m(휉≤푛 | 푛, 푘, Δ(푛, 푘))훿(푛, 푘) 푛≥1 푘 푛 ∑︁ 푛 ∑︁ = sup 훿(푛, 푘) 1푦 (휉)m( 푦 | 푛, 푘, Δ(푛, 푘)) 푛≥1 푘 푘=0 푦 ∈B(푛,푘) 푛 ∑︁ ∑︁ 푛 ∑︁ ≤ 1푦 (휉)훿(푛, 푘)m( 푦 | 푛, 푘, Δ(푛, 푘)) 푘 푛≥1 푘=0 푦 ∈B(푛,푘) 푛 ∗ ∑︁ ∑︁ 푛 ∑︁ = 1푦 (휉)훿( 푦), 푘 푛≥1 푘=0 푦 ∈B(푛,푘) using the notation of Claim 4.6.12 above. Let 푡(휉) denote the right-hand side here, which is thus a lower semicomputable function. We have for all 푝:

푛 휉 ∗ ∑︁ ∑︁ ∑︁ 퐵푝푡(휉) = 퐵푝 (푛, 푘)훿(푛, 푘) m( 푦 | 푛, 푘, Δ(푛, 푘)) ≤ 1, 푛≥1 푘=0 푦 ∈B(푛,푘)

∗ ∗ so 푡(휉) > 푏(휉) is a Bernoulli test, showing 푏(휉) < 푏(휉).

117 4. Generalizations

∗ To show 푏(휉) < 푏(휉) we will follow the method of the proof of Theo- rem 2.3.4. Replace 푏(휉) with a rougher version: 1 푏0(휉) = max{2푛 : 2푛 < 푏(휉)}, 2 0 ∗ then we have 2푏 < 푏. There are computable sequences 푦푖 ∈ B and 푘푖 ∈ N with 0 푘푖 푏 (휉) = sup푖 2 1푦푖 (휉) with the property that if 푖 < 푗 and 1푦푖 (휉) = 1푦푗 (휉) = 1 0 Í 푘푖 then 푘푖 < 푘 푗. As in the imitated proof, we have 2푏 (휉) ≥ 푖 2 1푦푖 (휉). The ( ) = Í 푘푖 function 훾 푦 푦푖=푦 2 is lower semicomputable. We have 푛 0 ∑︁ 푘푖 ∑︁ ∑︁ ∑︁ ∑︁ 푏 ≥ 2푏 (휉) ≥ 2 1푦푖 (휉) = 1푦 (휉)훾( 푦) = 1푦 (휉)훾( 푦). 푖 푦 ∈N∗ 푛 푘=0 푦 ∈B(푛,푘) (4.6.7) Í ( ) ≤ 푛 By Theorem 4.2.3 we can assume 푦 ∈B(푛,푘) 훾 푦 푘 . Let − 푛 1 훿0( 푦) = 훾( 푦) ≤ 1, 푘 ∑︁ 훿0(푛, 푘) = 훿0( 푦). 푦 ∈B(푛,푘) 0 Since 1 ≥ 퐵푝푏 ≥ 퐵푝 (2푏 ) for all 푝, we have 푛 푛 ∑︁ ∑︁ ∑︁ 휉 ∑︁ ∑︁ 푘 푛−푘 ∑︁ 1 ≥ 훾( 푦)퐵푝1푦 (휉) = 푝 (1 − 푝) 훾( 푦) 푛 푘=0 푦 ∈B(푛,푘) 푛 푘=0 푦 ∈B(푛,푘) 푛 푛 ∑︁ ∑︁ ∑︁ 0 ∑︁ ∑︁ 0 = 퐵푝 (푛, 푘) 훿 ( 푦) = 퐵푝 (푛, 푘)훿 (푛, 푘). 푛 푘=0 푦 ∈B(푛,푘) 푛 푘=0 ∗ Thus 훿0(푛, 푘) is a gap function, hence 훿0(푛, 푘) < 2−Δ(푛,푘) , and by Claim 4.6.12 we have −1 푛 ∗ 훾( 푦) = 훿0( 푦) < 훿( 푦) =∗ 훿(푛, 푘)· m( 푦 | 푛, 푘, Δ(푛, 푘)). 푘 ∗ Substituting back into (4.6.7) ﬁnishes the proof of 푏(휉) < 푏(휉).

4.7 Cells

This section allows to transfer some of the results on sequence spaces to more general spaces, by encoding the elements into sequences. The reader who is only interested in sequences can skip this section.

118 4.7. Cells

4.7.1 Partitions The coordinates of the sequences into which we want to encode our elements will be obtained via certain partitions. Recall from Definition A.2.33 that a measurable set 퐴 is said to be a 휇- continuity set if 휇(휕퐴) = 0 where 휕퐴 is the boundary of 퐴. Definition 4.7.1 (Continuity partition) Let 푓 : 푋 → R be a bounded computable function, and let 훼1 < ··· < 훼푘 be rational numbers, and let 휇 be a −1 computable measure with the property that 휇 푓 (훼 푗) = 0 for all 푗. In this case, we will say that 훼 푗 are 휇-continuity points of 푓 . Let 훼0 = −∞, −1 훼푘+1 = ∞, and for 푗 = 0, . . . , 푘, let Let 푈 푗 = 푓 (( 푗, 푗 + 1)). The sequence of disjoint computably enumerable open sets (푈0, . . . , 푈푘) will be called the partition generated by 푓 , 훼1, . . . , 훼푘. If we have several partitions (푈푖0, . . . , 푈푖,푘), generated by different functions 푓푖 (푖 = 1, . . . , 푚) and cutoff sequences (훼푖 푗 : 푗 = 1, . . . , 푘푖) made up of 휇- continuity points of 푓푖 then we can form a new partition generated by all possible intersections

푉푗1,..., 푗푛 = 푈1, 푗1 ∩ · · · ∩ 푈푚, 푗푚 .

A partition of this kind will be called a continuity partition. The sets 푉푗1,..., 푗푛 will be called the cells of this partition. y The following is worth noting.

Proposition 4.7.2 In a partition as given above, the values 휇푉푗1,..., 푗푛 are computable from the names of the functions 푓푖 and the cutoff points 훼푖 푗. Proof. Straightforward. Let us proceed to defining cells. Definition 4.7.3 (Separating sequence) Assume that a computable sequence of functions 푏1 (푥), 푏2 (푥),... over 푋 is given, with the property that for every pair 푥1, 푥2 ∈ 푋 with 푥1 ≠ 푥2, there is a 푗 with 푏푗 (푥1)· 푏푗 (푥2) < 0. Such a sequence will be called a separating sequence. Let us give the correspondence between the set BN of infinite binary sequences and elements of the set

0 푋 = {푥 ∈ 푋 : 푏푗 (푥) ≠ 0, 푗 = 1, 2,...}.

∗ For a binary string 푠1 ··· 푠푛 = 푠 ∈ B , let

Γ푠

119 4. Generalizations

be the set of elements of 푋 with the property that for 푗 = 1, . . . , 푛, if 푠푗 = 0 then 푏푗 (휔) < 0, otherwise 푏푗 (휔) > 0. The separating sequence will be called 휇-continuity sequence if 휇(푋 0) = 0. y This correspondence has the following properties.

a) ΓΛ = 푋.

b) For each 푠 ∈ B, the sets Γ푠0 and Γ푠1 are disjoint and their union is contained in Γ푠. ∈ 0 { } = Ñ c) For 푥 푋 , we have 푥 푥 ∈Γ푠 Γ푠.

Deﬁnition 4.7.4 (Cells) If string 푠 has length 푛 then Γ푠 will be called a canonical 푛-cell, or simply canonical cell, or 푛-cell. From now on, whenever Γ denotes a subset of 푋, it means a canonical cell. We will also use the notation

푙(Γ푠) = 푙(푠).

y The three properties above say that if we restrict ourselves to the set 푋 0 then the canonical cells behave somewhat like binary subintervals: they divide 푋 0 in half, then each half again in half, etc. Moreover, around each point, these canonical cells become “arbitrarily small”, in some sense (though, they may not be a basis of neighborhoods). It is easy to see that if Γ푠1 , Γ푠2 are two canonical cells then they either are disjoint or one of them contains the other. If Γ푠1 ⊂ Γ푠2 0 0 then 푠2 is a preﬁx of 푠1. If, for a moment, we write Γ푠 = Γ푠 ∩ 푋 then we have 0 0 0 the disjoint union Γ푠 = Γ푠0 ∪ Γ푠1. Let us use the following notation.

Deﬁnition 4.7.5 For an 푛-element binary string 푠, for 푥 ∈ Γ푠, we will write

≤푛 푠 = 푥 = 푥1 ··· 푥푛, 휇(푠) = 휇(Γ푠).

푛 The 2 cells (some of them possibly empty) of the form Γ푠 for 푙(푠) = 푛 form a partition P푛 of 푋 0. y 0 Thus, for elements of 푋 , we can talk about the 푛-th bit 푥푛 of the description of 푥: it is uniquely determined. Examples 4.7.6

120 4.7. Cells

1. If X is the set of infinite binary sequences with its usual topology, the functions 0 푏푛 (푥) = 푥푛 − 1/2 generate the usual cells, and X = X. 푛 2. If X is the interval [0, 1], let 푏푛 (푥) = − sin(2 휋푥). Then cells are open intervals of the form (푘 · 2−푛, (푘 + 1)· 2푛), the correspondence between infinite binary strings and elements of 푋 0 is just the usual representation of 푥 as the binary decimal string 0.푥1푥2 .... y When we fix canonical cells, we will generally assume that the partition chosen is also “natural”. The bits 푥1, 푥2,... could contain information about the point 푥 in decreasing order of importance from a macroscopic point of view. For example, for a container of gas, the first few bits may describe, to a reasonable degree of precision, the amount of gas in the left half of the container, the next few bits may describe the amounts in each quarter, the next few bits may describe the temperature in each half, the next few bits may describe again the amount of gas in each half, but now to more precision, etc. From now on, whenever Γ denotes a subset of 푋, it means a canonical cell. From now on, for 0 elements of 푋 , we can talk about the 푛-th bit 푥푛 of the description of 푥: it is uniquely determined. The following observation will prove useful. Proposition 4.7.7 Suppose that the space X is effectively compact3 and we have a separating sequence 푏푖 (푥) as given above. Then the cells Γ푠 form a basis of the space X.

Proof. We need to prove that for every ball 퐵(푥, 푟), the there is a cell 푥 ∈ Γ푠 ⊂ 퐵(푥, 푟). Let 퐶 be the complement of 퐵(푥, 푟). For each point 푦 of 퐶, there is an 0 1 푖 such that 푏푖 (푥)· 푏푖 ( 푦) < 0. In this case, let 퐽 = {푧 : 푏푖 (푧) < 0}, 퐽 = {푧 : 푝 푝 Ð 푏푖 (푧) > 0}. Let 퐽( 푦) = 퐽 such that 푦 ∈ 퐽 . Then 퐶 ⊂ 푦 퐽( 푦), and com- Ð푘 pactness implies that there is a ﬁnite sequence 푦1, . . . , 푦푘 with 퐶 ⊂ 푗=1 퐽( 푦푗). Clearly, there is a cell

Ø푘 푥 ∈ Γ푠 ⊂ 퐵(푥, 푟)\ 퐽( 푦푗). 푗=1

3It was noted by Hoyrup and Rojas that the qualiﬁcation “eﬀectively” is necessary here.

121 4. Generalizations

4.7.2 Computable probability spaces If a separating sequence is given in advance, we may restrict attention to the class of measures that make our sequence a 휇-continuity sequence: Deﬁnition 4.7.8 Let M0 (푋) be the set of those probability measures 휇 in M(푋) for which 휇(푋 \ 푋 0) = 0. y On the other hand, for each computable measure 휇, a computable separating sequence can be constructed that is a 휇-continuity sequence. Recall that 퐵(푥, 푟) is the ball of center 푥 and radius 푟. Let 퐷 = {푠1, 푠2,... } = {훼(1), 훼(2),... } be the canonical enumeration of the canonical dense set 퐷. Theorem 4.7.1 (Hoyrup-Rojas) There is a sequence of uniformly computable reals (푟푛)푛∈N such that (퐵(푠푖, 푟푛))푖,푛 is a basis of balls that are 휇-continuity sets. This basis is constructively equivalent to the original one, consisting of all balls 퐵(푠푖, 푟), 푟 ∈ Q. Corollary 4.7.9 There is a computable separating sequence with the 휇-continuity property.

Proof. Let us list all balls 퐵(푠푖, 푟푛) into a single sequence 퐵(푠푖푘 , 푟푛푘 ). The functions

푏푘 (푥) = 푑(푠푖푘 , 푥) − 푟푛푘 give rise to the desired sequence. For the proof of the theorem, we use some preparation. Recall from Defini- tion A.2.5 that an atom is a point with positive measure. Lemma 4.7.10 Let 푋 be R or R+ or [0, 1]. Let 휇 be a computable probability measure on 푋. Then there is a sequence of uniformly computable reals (푥푛)푛 which is dense in 푋 and contains no atoms of 휇. Proof. Let 퐼 be a closed rational interval. We construct 푥 ∈ 퐼 with 휇({푥}) = 0. To do this, we construct inductively a nested sequence of closed intervals 퐽푘 of −푘+1 measure < 2 , with 퐽0 = 퐼. Suppose 퐽푘 = [푎, 푏] has been constructed, with −푘+1 휇(퐽푘) < 2 . Let 푚 = (푏 − 푎)/3: one of the intervals [푎, 푎 + 푚] and [푏 − 푚, 푏] −푘 must have measure < 2 , and we can find it effectively—let it be 퐽푘+1. From a constructive enumeration (퐼푛)푛 of all the dyadic intervals, we can construct 푥푛 ∈ 퐼푛 uniformly.

122 4.7. Cells

Corollary 4.7.11 Let (X, 휇) be a computable metric space with a computable measure and let ( 푓푖)푖 be a sequence of uniformly computable real valued functions on 푋. Then there is a sequence of uniformly computable reals (푥푛)푛 that is dense in R and such that each 푥푛 is a 휇-continuity point of each 푓푖. −1 Proof. Consider the uniformly computable measures 휇푖 = 휇 ◦ 푓푖 and deﬁne Í −푖 휈 = 푖 2 휇푖. It is easy to see that 휈 is a computable measure and then, by Lemma 4.7.10, there is a sequence of uniformly computable reals (푥푛)푛 which is dense in R and contains no atoms of 휈. Since 휈(퐴) = 0 iﬀ 휇푖 (퐴) = 0 for all 푖, −1 we get 휇({ 푓푖 (푥푛)}) = 0 for all 푖, 푛.

Proof of Theorem 4.7.1. Apply Corollary 4.7.11 to 푓푖 (푥) = 푑(푠푖, 푥). Since every ball 퐵(푠푖, 푟) can be expressed as a computably enumerable union of the balls of the type 퐵(푠푖, 푟푛) just constructed, the two bases are constructively equivalent.

123

5 Exercises and problems

푟 Exercise 1 Deﬁne, for any two natural numbers 푟, 푠, a standard encoding cnv푠 of base 푟 strings 푥 into base 푠 strings with the property log 푟 |cnv푟 (푥)| ≤ |푥| + 1. (5.0.1) 푠 log 푠

푁 푁 Solution. We will use X = 푁 , X푟 = 푍푟 for the sets of inﬁnite strings of natural numbers and 푟-ary digits respectively. For a sequence 푝 ∈ X푟, let [푝]푟 denote the real number in the interval [0, 1] which 0.푝 denotes in the base 푟 number system. For 푝 in S푟, let [푝]푟 = {[푝푞]푟 : 푞 ∈ X푟 }. For the 푟-ary string 푥, let 푣 be the size of the largest 푠-ary intervals [푦]푠 contained in the 푟-ary interval [푥]푟. If [푧]푠 is the leftmost among these intervals, 푟 −|푥 | then let cnv푠 (푥) = 푧. This is a one-to-one encoding. We have 푟 < 2푠푣, since any 2푠 consecutive 푠-ary intervals of length 푣 contain an 푠-ary interval of length 푠푣. Therefore log 푟 log 2 |푧| = − log 푣 < |푥| + 1 + 푠 log 푠 log 푠 hence, since 2 ≤ 푠 and |푥| is integer, we have the inequality (5.0.1).

Exercise 2 A function 퐴 from S푟 × S to S is called an 푟-ary interpreter. Prove the following generalization of the Invariance Theorem. For any 푠, there is a p.r. 푠-ary interpreter 푈 such that for any p.r. interpreter 퐴 there is a constant 푐 < ∞ such that for all 푥, 푦 we have

퐶푈 (푥 | 푦) ≤ 퐶퐴 (푥 | 푦) + 푐. (5.0.2) ∗ ∗ Solution. Let 푉 : Z푠 × Z푠 × S → S be a partial recursive function which is universal: such that for any p.r. 푠-ary interpreter 퐴, there is a string 푎 such that for all 푝, 푥, we have 퐴( 푝, 푥) = 푉 (푎, 푝, 푥). The machine computing 푈( 푝, 푥) tries to decompose 푝 into 푢표푣 and outputs 푉 (푢, 푣, 푥). Let us verify that 푈 is optimal. Let 퐴 be a p.r. 푟-ary interpreter, 퐵 an

125 5. Exercises and problems

푟 푠-ary p.r. interpreter such that 퐵(cnv푠 ( 푝), 푥) = 퐴( 푝, 푥) for all 푝, 푥, 푎 a binary string such that 퐵( 푝, 푥) = 푈(푎, 푝, 푥) for all 푝, 푥. Let 푥, 푦 be two strings. If 퐶퐴 (푥 | 푦) = ∞, then the inequality (5.0.2) holds trivially. Otherwise, let 푝 be a binary string of length 퐶퐴 (푥 | 푦)/log 푟 with 퐴( 푝, 푦) = 푥. Then

표 푟 푟 푟 푈(푎 cnv푠 ( 푝), 푦) = 푉 (푎, cnv푠 ( 푝), 푦) = 퐵(cnv푠 ( 푝), 푦) = 퐴( 푝, 푦) = 푥.

퐶푈 (푥 | 푦) ≤ (2|푎| + 퐶퐴 (푥 | 푦)/log 푠 + 1) log 푠 = 퐶퐴 (푥 | 푦) + (2|푎|) + 1) log 푠.

푚 Exercise 3 (푆푛 -theorem) Prove that there is a binary string 푏 such that 푈( 푝, 푞, 푥) = 푈(푏표 푝표푞, 푥) holds for all binary strings 푝, 푞 and arbitrary strings x. Exercise 4 (Schnorr) Notice that, apart from the conversion between representations, what our optimal interpreter does is the following. It treats the program 푝 as a pair ( 푝(1), 푝(2)) whose ﬁrst member is the Gödel number of some interpreter for the universal p.r. function 푉 ( 푝1, 푝2, 푥), and the second argument as a program for this interpreter. Prove that indeed, for any recursive pairing function 푤( 푝, 푞), if there is a function 푓 such that |푤( 푝, 푞)| ≤ 푓 ( 푝) + |푞| then 푤 can be used to deﬁne an optimal interpreter. + Exercise 5 Refute the inequality 퐶(푥, 푦) < 퐶(푥) + 퐶( 푦). Exercise 6 Prove the following sharpenings of Theorem 1.4.2.

+ 퐶(푥, 푦) < 퐽(퐶(푥)) + 퐶( 푦 | 푥, 퐶(푥)), + 퐶(푥, 푦) < 퐶(푥) + 퐽(퐶( 푦 | 푥, 퐶(푥))).

Prove Theorem 1.4.2 from here. Exercise 7 (Schnorr) Prove 퐶(푥 +퐶(푥)) =+ 퐶(푥). Can you generalize this result? + Exercise 8 (Schnorr) Prove that if 푚 < 푛 then 푚 + 퐶(푚) < 푛 + 퐶(푛). Exercise 9 Prove

푛 푛 푛 1 푛 log =+ 푘 log + (푛 − 푘) log + log . 푘 푘 푛 − 푘 2 푘(푛 − 푘)

126 Exercise 10 (Kamae) Prove that for any natural number 푘 there is a string 푥 such that for for all but ﬁnitely many strings 푦, we have

퐶(푥) − 퐶(푥 | 푦) ≥ 푘.

In other words, there are some strings 푥 such that almost any string 푦 contains a large amount of information about them. Solution. Strings 푥 with this property are for example the ones which contain information obtainable with the help of any sufficiently large number. Let 퐸 be a r.e. set of integers. Let 푒0푒1푒2 ... be the infinite string which is the characteristic function of 퐸, and let 푥(푘) = 푒0 . . . 푒2푘 . We can suppose that 퐶(푥(푘)) ≥ 푘. Let 푛1, 푛2,... be a recursive enumeration of 퐸 without repetition, and let 훼(푘) = 푘 + max{푖 : 푛푖 ≤ 2 }. Then for any number 푡 ≥ 훼(푘) we have 퐶(푥(푘) | 푡) < log 푘. Indeed, a binary string of length 푘 describes the number 푘. Knowing 푡 we can enumerate 푛1, . . . , 푛푡 and thus learn 푥(푘). Therefore with any string 푦 of length + ≥ 훼(푘) we have 퐶(푥) − 퐶(푥 | 푦) > 푘 − log 푘. Exercise 11 a) Prove that a real function 푓 is computable iff there is a recursive function 푔(푥, 푛) with rational values, and | 푓 (푥) − 푔(푥, 푛)| < 1/푛. b) Prove that a function 푓 is semicomputable iff there exists a recursive function with rational values, (or, equivalently, a computable real function) 푔(푥, 푛) nondecreasing in 푛, with 푓 (푥) = lim푛→∞ 푔(푥, 푛). Exercise 12 Prove that in Theorem 1.5.2 one can write “semicomputable” for “partial recursive”. Exercise 13 (Levin) Show that there is an upper semicomputable function 퐺( 푝, 푥, 푦) which for different finite binary strings 푝 enumerates all upper semicomputable functions 퐹(푥, 푦) satisfying the inequality (1.5.1). Prove

퐶(푥 | 푦) =+ inf 퐺( 푝, 푥, 푦) + 퐽(| 푝|). 푝

Exercise 14 Prove ∑︁ ∗ m( 푝) = m(푛). 푝∈B푛 Exercise 15 (Solovay) Show that we cannot find effectively infinitely many places where some recursive upper bound of 퐾(푛) is sharp. Moreover, suppose that 퐹(푛) is a recursive upper bound of 퐾(푛). Then there is no recursive function D(푛) ordering to each 푛 a finite set of natural numbers (represented for example as a string) larger than 푛 such that for each 푛 there is an 푥 in D(푛) with

127 5. Exercises and problems

퐹(푥) ≤ 퐾(푥). Notice that the function log 푛 (or one almost equal to it) has this property for 퐶(푛). Solution. Suppose that such a function exists. Then we can select a sequence 푛1 < 푛2 < . . . of integers with the property that the sets D(푛푖) are all disjoint. Let ∑︁ 푎(푛) = 2−퐹 (푥) . 푥 ∈D(푛) Í Then the sequence 푎(푛푘) is computable and 푘 푎(푛푘) < 1. It is now easy to construct a computable subsequence 푚푖 = 푛푘푖 of 푛푘 and a computable sequence Í 푏푖 such that 푏푖/푎(푚푖) → ∞ and 푖 푏푖 < 1. Let us define the semimeasure 휇 setting −퐹 (푥) 휇(푥) = 2 푏푖/푎(푚푖) + for any 푥 in D(푚푖) and 0 otherwise. Then for any 푥 in D(푚푖) we have 퐾(푥) < − log 휇(푥) = 퐹(푥) − log 푐푖 where 푐푖 = 푏푖/푎(푚푖) → ∞, so we arrived a contradic- tion with the assumption. Exercise 16 Prove that there is a recursive upper bound 퐹(푛) of 퐾(푛) and a constant 푐 with the property that there are infinitely many natural numbers 푛 such that for all 푘 > 0, the quantity of numbers 푥 ≤ 푛 with 퐾(푥) < 퐹(푥) − 푘 is less than 푐푛2−푘. Solution. Use the upper bound 퐺 found in Theorem 1.7.4 and define 퐹(푛) = log 푛 + 퐺(blog 푛c). The property follows from the facts that log 푛 + 퐾(blog 푛c) is a sharp upper bound for 퐾(푥) for “most” 푥 less than 푛 and that 퐺(푘) is infinitely often close to 퐾(푘).

Exercise 17 Give an example of a computable sequence 푎푛 > 0 of with the Í property that 푛 푎푛 < ∞ but for any other computable sequence 푏푛 > 0, if Í 푏푛/푎푛 → 푖푛 푓 푡푦 then 푛 푏푛 = ∞.

Hint: Let 푟푛 be a recursive, increasing sequence of rational numbers with lim푛 푟푛 = Í 푥 m(푥) and let 푎푛 = 푟푛+1 − 푟푛.

Exercise 18 (Universal coding, Elias) Let 푓 (푛) = log2 푛 + 2 log2 log2 푛. Show that when 푃 runs over all nonincreasing probability distributions over 푁 then ∑︁ lim 퐾(푃)−1 푃(푛) 푓 (푛) = 1. 퐾 (푃)→∞ 푛

128 ∗ Exercise 19 (T. Cover) Let log2 푛 = log2 푛+log2 log2 푛+... (all positive terms). Prove that ∑︁ − log∗ 푛 2 2 < ∞, 푛 + ∗ + ∗ hence 퐾(푛) < log2 푛. For which logarithm bases does 퐾(푛) < log2 푛 hold? Exercise 20 Prove that Kamae’s result in Exercise 10 does not hold for 퐾(푥 | 푦). Exercise 21 Prove that for each 휀 there is an 푚 such that if H(푃) > 푚 then Í | 푥 푃(푥)퐾(푥)/H(푃) − 1| < 휀. Exercise 22 If a ﬁnite sequence 푥 of length 푛 has large complexity then its bits are certainly not predictable. Let ℎ(푘, 푛) = −(푘/푛)푙표푔(푘/푛)−(1−(푘/푛))푙표푔(1− (푘/푛)). A quantitative relation of this sort is the following. If, for some 푘 > 푛/2, a program of length 푚 can predict 푥푖 from 푥1, . . . , 푥푖−1 for at least 푛 − 푘 values of 푖 then 퐶(푥) < 푚 + 푛ℎ(푘, 푛) + 표(푛). Exercise 23 This is a more open-ended problem, having to do with the converse of the previous exercise. Show (if true) that for each 푐 there is a 푑 such that predictability of 푥 at signiﬁcantly more than 푛/2 + 푐 places is equivalent to the possibility to enclose 푥 into a set of complexity 표(푛) and size 푛 − 푑푛.

129

A Background from mathematics

A.1 Topology

In this section, we summarize the notions and results of topology that are needed in the main text.

A.1.1 Topological spaces A topological space is a set of points with some kind of—not quantitatively expressed—closeness relation among them. Definition A.1.1 A topology on a set 푋 is defined by a class 휏 of its subsets called open sets. It is required that the empty set and 푋 are open, and that arbitrary union and finite intersection of open sets is open. The pair (푋, 휏) is called a topological space. A set is called closed if its complement is open. y Having a set of open and closed sets allows us to speak about closure operations. Definition A.1.2 A set 퐵 is called the neighborhood of a set 퐴 if 퐵 contains an open set that contains 퐴. We denote by 퐴, 퐴o the closure (the intersection of all closed sets containing 퐴) and the interior of 퐴 (the union of all open sets in 퐴) respectively. Let 휕퐴 = 퐴 \ 퐴o denote the boundary of set 퐴. y An alternative way of defining a topological space is via a basis. Definition A.1.3 A basis of a topological space is a subset 훽 of 휏 such that every open set is the union of some elements of 훽.A neighborhood of a point is a basis element containing it. A basis of neighborhoods of a point 푥 is a set 푁 of neighborhoods of 푥 with the property that each neighborhood of 푥 contains an

131 A. Background from mathematics element of 푁.A subbasis is a subset 휎 of 휏 such that every open set is the union of ﬁnite intersections from 휎. y Examples A.1.4 1. Let 푋 be a set, and let 훽 be the set of all points of 푋. The topology with basis 훽 is the discrete topology of the set 푋. In this topology, every subset of 푋 is open (and closed).

2. Let 푋 be the real line R, and let 훽R be the set of all open intervals (푎, 푏). The topology 휏R obtained from this basis is the usual topology of the real line. When we refer to R as a topological space without qualiﬁcation, this is the topology we will always have in mind.

3. Let 푋 = R = R ∪ {−∞, ∞}, and let 훽R consist of all open intervals (푎, 푏) and in addition of all intervals of the forms [−∞, 푎) and (푎, ∞]. It is clear how the space R+ is defined. > 4. Let 푋 be the real line R. Let 훽R be the set of all open intervals (−∞, 푏). The > topology with basis 훽R is also a topology of the real line, different from the < usual one. Similarly, let 훽R be the set of all open intervals (푏, ∞). 5. Let Σ be a finite or countable alphabet. On the space ΣN of infinite sequences N ∗ with elements in Σ, let 휏퐶 = { 퐴Σ : 퐴 ⊆ Σ } be called the topology of the Cantor space (over Σ). Note that if a set 퐸 has the form 퐸 = 퐴ΣN where 퐴 is finite then 퐸 is both open and closed. y Starting from open sets, we can define some other kinds of set that are still relatively simple:

Definition A.1.5 A set is called a 퐺훿 set if it is a countable intersection of open sets, and it is an 퐹휎 set if it is a countable union of closed sets. y Different topologies over the same space have a natural partial order relation among them: Definition A.1.6 A topology 휏0 on 푋 is called larger, or finer than 휏 if 휏0 ⊇ 휏. For two topologies 휏1, 휏2 over the same set 푋, we define the topology 휏1 ∨ 휏2 = 휏1 ∩ 휏2, and 휏1 ∧ 휏2 as the smallest topology containing 휏1 ∪ 휏2. In the example < > topologies of the real numbers above, we have 휏R = 휏R ∧ 휏R. y Most topologies used in practice have some separion property.

Deﬁnition A.1.7 A topology is said to have the 푇0 property if every point is determined by the class of open sets containing it. This is the weakest one of a number of other possible separation y

132 A.1. Topology

Both our above example topologies of the real line have this property. All topologies considered in this survey will have the 푇0 property. A stronger such property is the following:

Definition A.1.8 A space is called a Hausdorff space, or 푇2 space, if for every pair of different points 푥, 푦 there is a pair of disjoint open sets 퐴, 퐵 with 푥 ∈ 퐴, 푦 ∈ 퐵. y > The real line with topology 휏R in Example A.1.4.4 above is not a Hausdorff space. A space is Hausdorff if and only if every open set is the union of closures of basis elements.

A.1.2 Continuity We introduced topology in order to be able to speak about continuity:

Definition A.1.9 Given two topological spaces (푋푖, 휏푖) (푖 = 1, 2), a function 푓 : 푋1 → 푋2 is called continuous if for every open set 퐺 ⊆ 푋2 its inverse image −1 푓 (퐺) is also open. If the topologies 휏1, 휏2 are not clear from the context then we will call the function (휏1, 휏2)-continuous. Clearly, the property remains the same if we require it only for all elements 퐺 of a subbasis of 푋2. If there are two continuous functions between 푋 and 푌 that are inverses of each other then the two spaces are called homeomorphic. We say that function 푓 is continuous at point 푥 if for every neighborhood 푉 of 푓 (푥) there is a neighborhood 푈 of 푥 with 푓 (푈) ⊆ 푉. y Clearly, function 푓 is continuous if and only if it is continuous in each point. There is a natural sense in which every subset of a topological space is also a topological space: Definition A.1.10 A subspace of a topological space (푋, 휏) is defined by a subset 푌 ⊆ 푋, and the topology 휏푌 = {퐺 ∩ 푌 : 퐺 ∈ 휏}, called the induced topology on 푌. This is the smallest topology on 푌 making the identity mapping 푥 ↦→ 푥 continuous. A partial function 푓 : 푋 → 푍 with dom( 푓 ) = 푌 is continuous iff 푓 : 푌 → 푍 is continuous. y Given some topological spaces, we can also form larger ones using for example the product operation:

Definition A.1.11 For two topological spaces (푋푖, 휏푖) (푖 = 1, 2), we define the product topology on their product 푋 × 푌: this is the topology defined by the subbasis consisting of all sets 퐺1×푋2 and all sets 푋1×퐺2 with 퐺푖 ∈ 휏푖. The product topology is the smallest topology making the projection functions (푥, 푦) ↦→ 푥,

133 A. Background from mathematics

(푥, 푦) ↦→ 푦 continuous. Given topological spaces 푋, 푌, 푍 we call a two-argument function 푓 : 푋 × 푌 ↦→ 푍 continuous if it is continuous as a function from 푋 × 푌 Î to 푍. The product topology is defined similarly for the product 푖∈퐼 푋푖 of an arbitrary number of spaces, indexed by some index set 퐼. We say that a function is (휏1, . . . , 휏푛, 휇)-continuous if it is (휏1 × · · · × 휏푛, 휇)-continuous. y Examples A.1.12 1. The space R × R with the product topology has the usual topology of the Euclidean plane. 2. Let 푋 be a set with the discrete topology, and 푋N the set of infinite sequences with elements from 푋, with the product topology. A basis of this topology is provided by all sets of the form 푢푋N where 푢 ∈ 푋∗. The elements of this basis are closed as well as open. When 푋 = {0, 1} then this topology is the usual topology of infinite binary sequences. y In some special cases, one of the topologies in the definition of continuity is fixed and known in advance:

Deﬁnition A.1.13 A real function 푓 : 푋1 → R is called continuous if it is (휏1, 휏R)-continuous. y

A.1.3 Semicontinuity For real functions, a restricted notion of continuity is often useful. Definition A.1.14 A function 푓 : 푋 → R is lower semicontinuous if the set {(푥, 푟) ∈ 푋 × R : 푓 (푥) > 푟} is open. The definition of upper semicontinuity is similar. Lower semicontinuity on subspaces is defined similarly to continuity on subspaces. y Clearly, a real function 푓 is continuous if and only if it is both lower and upper semicontinuous. The requirement of lower semicontinuity of 푓 is equivalent to saying that for each 푟 ∈ R, the set {푥 : 푓 (푥) > 푟} is open. This can be seen to be equivalent to the following characterization. Proposition A.1.15 A real function 푓 over X = (푋, 휏) is lower semicontinuous if < and only if it is (휏, 휏R)-continuous.

Example A.1.16 The indicator function 1퐺 (푥) of an arbitrary open set 퐺 is lower semicontinuous. y The following is easy to see.

134 A.1. Topology

Proposition A.1.17 The supremum of any set of lower semicontinuous functions is lower semicontinuous. The following representation is then also easy to see. Proposition A.1.18 Let 푋 be a topological space with basis 훽. The function 푓 : 푋 → R+ is lower semicontinuous if and only if there is a function 푔 : 훽 → R+ with 푓 (푥) = sup푥 ∈훽 푔(훽).

Corollary A.1.19 Let 푋 be a topological space with basis 훽 and 푓 : 푋 → R+ a lower semicontinuous function deﬁned on a subset 퐷 of 푋. Then 푓 can be extended in a lower semicontinuous way to the whole space 푋.

Proof. Indeed by the above proposition there is a function 푔 : 훽 → R+ with 푓 (푥) = sup푥 ∈훽 푔(훽) for all 푥 ∈ 퐷. Let us define 푓 by this same formula for all 푥 ∈ 푋. In the important special case of Cantor spaces, the basis is given by the set of finite sequences. In this case we can also require the function 푔(푤) to be monotonic in the words 푤: Proposition A.1.20 Let 푋 = ΣN be a Cantor space as defined in Example A.1.4.5. Then 푓 : 푋 → R+ is lower semicontinuous if and only if there is a function 푔 : ∗ Σ → R+ monotonic with respect to the relation 푢 v 푣, with 푓 (휉) = sup푢v휉 푔(푢).

A.1.4 Compactness There is an important property of topological spaces that, when satisfied, has many useful implications. Definition A.1.21 Let (푋, 휏) be a topological space, and 퐵 a subset of 푋. An open cover of 퐵 is a family of open sets whose union contains 퐵. A subset 퐾 of 푋 is said to be compact if every open cover of 퐾 has a finite subcover. y Compact sets have many important properties: for example, a continuous function over a compact set is bounded. Example A.1.22 1. Every finite discrete space is compact. An infinite discrete space X = (푋, 휏) is not compact, but it can be turned into a compact space X by adding a new element called ∞: let 푋 = 푋 ∪ {∞}, and 휏 = 휏 ∪ { 푋 \ 퐴 : 퐴 ⊂ 푋 closed }. More generally, this simple operation can be performed with every space that is locally compact, that each of its points has a compact neighborhood.

135 A. Background from mathematics

2. In a ﬁnite-dimensional Euclidean space, every bounded closed set is compact.

3. It is known that if (X푖)푖∈퐼 is a family of compact spaces then their direct product is also compact. y There are some properties that are equivalent to compactness in simple cases, but not always: Deﬁnition A.1.23 A subset 퐾 of 푋 is said to be sequentially compact if every sequence in 퐾 has a convergent subsequence with limit in 퐾. The space is locally compact if every point has a compact neighborhood. y

A.1.5 Metric spaces Metric spaces are topological spaces with more structure: in them, the closeness concept is quantiﬁable. In our examples for metric spaces, and later in our treatment of the space of probability measures, we refer to [5]. Deﬁnition A.1.24 A metric space is given by a set 푋 and a distance function 푑 : 푋 × 푋 → R+ satisfying the triangle inequality 푑(푥, 푧) ≤ 푑(푥, 푦) + 푑( 푦, 푧) and also property that 푑(푥, 푦) = 0 implies 푥 = 푦. For 푟 ∈ R+, the sets

퐵(푥, 푟) = { 푦 : 푑(푥, 푦) < 푟}, 퐵(푥, 푟) = { 푦 : 푑(푥, 푦) ≤ 푟} are called the open and closed balls with radius 푟 and center 푥. A metric space is bounded when 푑(푥, 푦) has an upper bound on 푋. y A metric space is also a topological space, with the basis that is the set of all open balls. Over this space, the distance function 푑(푥, 푦) is obviously continuous. Each metric space is a Hausdorﬀ space; moreover, it has the following stronger property.

Deﬁnition A.1.25 A topological space is said to have the 푇3 property if for every pair of diﬀerent points 푥, 푦 there is a continuous function 푓 : 푋 → R with 푓 (푥) ≠ 푓 ( 푦). y

To see that metric spaces are 푇3, take 푓 (푧) = 푑(푥, 푧). Deﬁnition A.1.26 A topological space is called metrizable if its topology can be derived from some metric space. y

It is known that a topological space is metrizable if and only if it has the 푇3 property.

136 A.1. Topology

Notation A.1.27 For an arbitrary set 퐴 and point 푥 let

푑(푥, 퐴) = inf 푑(푥, 푦), 푦 ∈퐴 퐴휀 = {푥 : 푑(푥, 퐴) < 휀}. (A.1.1)

y Examples A.1.28 1. A discrete topological space 푋 can be turned into a metric space as follows: 푑(푥, 푦) = 0 if 푥 = 푦 and 1 otherwise. 2. The real line with the distance 푑(푥, 푦) = |푥 − 푦| is a metric space. The topology of this space is the usual topology 휏R of the real line. 3. The space R × R with the Euclidean distance gives the same topology as the product topology of R × R. 4. An arbitrary set 푋 with the distance 푑(푥, 푦) = 1 for all pairs 푥, 푦 of different elements, is a metric space that induces the discrete topology on 푋. 5. Let 푋 be a bounded metric space, and let 푌 = 푋N be the set of infinite se- N Í −푖 quences 푥 = (푥1, 푥2,... ) with distance function 푑 (푥, 푦) = 푖 2 푑(푥푖, 푦푖). The topology of this space is the same as the product topology defined in Example A.1.12.2. 6. Specializing the above example, if Σ is the discrete space defined in Example1 above then we obtain a metrization of the Cantor space of Example A.1.45. For every finite sequence 푥 ∈ Σ∗ and every infinite sequence 휉 w 푥 the ball 퐵(휉, 2−푙(푥) ) is equal to a basis element that is the open-closed cylinder set 푥ΣN. 7. Let 푋 be a metric space, and let 푌 = 푋N be the set of infinite bounded se-

quences 푥 = (푥1, 푥2,... ) with distance function 푑(푥, 푦) = sup푖 푑(푥푖, 푦푖). 8. Let 푋 be a topological space, and let 퐶(푋) be the set of bounded continuous 0 functions over 푋 with distance function 푑 ( 푓 , 푔) = sup푥 푑( 푓 (푥), 푔(푥)).A special case is 퐶[0, 1] where the interval [0, 1] of real numbers has the usual topology.

9. Let 푙2 be the set of inﬁnite sequences 푥 = (푥1, 푥2,... ) of real numbers with Í 2 the property that 푖 푥푖 < ∞. The metric is given by the distance 푑(푥, 푦) = Í 2 1/2 ( 푖 |푥푖 − 푦푖 | ) . y

137 A. Background from mathematics

In metric spaces, certain previously deﬁned topological objects have richer properties. Examples A.1.29 Each of the following facts holds in metric spaces and is relatively easy to prove.

1. Every closed set is a 퐺훿 set (and every open set is an 퐹휎 set). 2. A set is compact if and only if it is sequentially compact. 3. A set is compact if and only if it is closed and for every 휀, there is a finite set of 휀-balls (balls of radius 휀) covering it. y In metric spaces, the notion of continuity can be strengthened. Definition A.1.30 A function 푓 : 푋 → 푌 between metric spaces 푋, 푌 is uniformly continuous if for each 휀 > 0 there is a 훿 > 0 such that 푑푋 (푎, 푏) < 훿 implies 푑푌 ( 푓 (푎), 푓 (푏)) < 휀. y It is known that over a compact metric space, every continuous function is uniformly continuous. Definition A.1.31 (Lipschitz) Given a function 푓 : 푋 → 푌 between metric spaces and 훽 > 0, let Lip훽 (푋, 푌) denote the set of functions (called the Lipschitz(훽) functions, or simply Lipschitz functions) satisfying

푑푌 ( 푓 (푥), 푓 ( 푦)) ≤ 훽푑푋 (푥, 푦). (A.1.2) Ð Let Lip(푋) = Lip(푋, R) = 훽 Lip훽 be the set of real Lipschitz functions over 푋. y As an example, every differentiable real function 푓 (푥) with | 푓 0(푥)| ≤ 1 everywhere is a Lipschitz(1) function. All these functions are uniformly continuous. We introduce a certain fixed, enumerated sequence of Lipschitz functions that will be used frequently as “building blocks” of other functions. Definition A.1.32 (Hat functions) Let

+ + 푔푢,푟,휀 (푥) = |1 − |푑(푥, 푢) − 푟| /휀| .

This is a continuous function that is 1 in the ball 퐵(푢, 푟), it is 0 outside the ball 퐵(푢, 푟+휀), and takes intermediate values in between. It is clearly a Lipschitz(1/휀) function.

138 A.1. Topology

If a dense set 퐷 is ﬁxed, let F0 = F0 (퐷) be the set of functions of the form 푔푢,푟,1/푛 where 푢 ∈ 퐷, 푟 is rational, 푛 = 1, 2,.... Let F1 = F1 (퐷) be the maximum of a ﬁnite number of elements of F0 (퐷). Each element 푓 of F1 is bounded between 0 and 1. Let

E = E(퐷) = {푔1, 푔2,... } ⊃ F1 (A.1.3) be the smallest set of functions containing F0 and the constant 1, and closed under ∨, ∧ and rational linear combinations. For each element of E, from its definition we can compute a bound 훽 such that 푓 ∈ Lip훽. y For the effective representation of points in a topological space the following properties are important. Definition A.1.33 A topological space has the first countability property if each point has a countable basis of neighborhoods. y Every metric space has the first countability property since we can restrict ourselves to balls with rational radius. Definition A.1.34 Given a topological space (푋, 휏) and a sequence 푥 = (푥1, 푥2,... ) of elements of 푋, we say that 푥 converges to a point 푦 if for every neighborhood 퐺 of 푦 there is a 푘 such that for all 푚 > 푘 we have 푥푚 ∈ 퐺. We will write 푦 = lim푛→∞ 푥푛. y

It is easy to show that if spaces (푋푖, 휏푖)(푖 = 1, 2) have the first countability property then a function 푓 : 푋 → 푌 is continuous if and only if for every convergent sequence (푥푛) we have 푓 (lim푛 푥푛) = lim푛 푓 (푥푛). Definition A.1.35 A topological space has the second countability property if the whole space has a countable basis. y For example, the space R has the second countability property for all three < > topologies 휏R, 휏R, 휏R. Indeed, we also get a basis if instead of taking all intervals, we only take intervals with rational endpoints. On the other hand, the metric space of Example A.1.28.7 does not have the second countability property. Definition A.1.36 In a topological space (푋, 휏), a set 퐵 of points is called dense at a point 푥 if it intersects every neighborhood of 푥. It is called everywhere dense, or dense, if it is dense at every point. A metric space is called separable if it has a countable everywhere dense subset. y It is easy to see that a metric space is separable if and only if as a topological space it has the second countability property.

139 A. Background from mathematics

Example A.1.37 In Example A.1.28.8, for 푋 = [0, 1], we can choose as our everywhere dense set the set of all polynomials with rational coefficients, or alternatively, the set of all piecewise linear functions whose graph has finitely many nodes at rational points. More generally, let 푋 be a compact separable metric space with a dense set 퐷. Then it can be shown that in the metric space 퐶(푋), the set of functions E(퐷) introduced in Definition A.1.32 is dense, and turns it into a complete (not necessarily compact!) separable metric space. y

Definition A.1.38 In a metric space, let us call a sequence 푥1, 푥2,... a Cauchy −푖 sequence if for all 푖 < 푗 we have 푑(푥푖, 푥 푗) < 2 . y It is easy to see that if an everywhere dense set 퐷 is given then every element of the space can be represented as the limit of a Cauchy sequence of elements of 퐷. But not every Cauchy sequence needs to have a limit. Definition A.1.39 A metric space is called complete if every Cauchy sequence in it has a limit. y For example, if 푋 is the real line with the point 0 removed then 푋 is not complete, since there are Cauchy sequences converging to 0, but 0 is not in 푋. It is well-known that every metric space can be embedded (as an everywhere dense subspace) into a complete space. Example A.1.40 Consider the set 퐷[0, 1] of functions over [0, 1] that are right continuous and have left limits everywhere. The book [5] introduces two different metrics for them: the Skorohod metric 푑 and the 푑0 metric. In both metrics, two functions are close if a slight monotonic continuous deformation of the coordinates makes them uniformly close. But in the 푑0 metric, the slope of the deformation must be close to 1. It is shown that the two metrics give rise to the same topology; however, the space with metric 푑 is not complete, and the space with metric 푑0 is. y We will develop the theory of randomness over separable complete metric spaces. This is a wide class of spaces encompassing most spaces of practical interest. The theory would be simpler if we restricted it to compact or locally compact spaces; however, some important spaces like 퐶[0, 1] (the set of contin- uouos functions over the interval [0, 1], with the maximum difference as their distance) are not locally compact.

140 A.2. Measures

A.2 Measures

For a survey of measure theory, see for example [40].

A.2.1 Set algebras Event in a probability space are members of a class of sets that is required to be a so-called 휎-algebra (sigma-algebra). Definition A.2.1 A (Boolean set-) algebra is a set of subsets of some set 푋 closed under intersection and complement (and then, of course, under union). It is a 휎-algebra (sigma-algebra) if it is also closed under countable intersection (and then, of course, under countable union). A semialgebra is a set L of subsets of some set 푋 closed under intersection, with the property that the complement of every element of L is the disjoint union of a finite number of elements of L. y If L is a semialgebra then the set of finite unions of elements of L is an algebra. Examples A.2.2

1. The set L1 of left-closed intervals of the line (including intervals of the form (−∞, 푎)) is a semialgebra.

2. The set L2 of all intervals of the line (which can be open, closed, left-closed or right-closed), is a semialgebra. N 3. In the set {0, 1} of inﬁnite 0-1-sequences, the set L3 of all subsets of the form 푢{0, 1}N with 푢 ∈ {0, 1}∗, is a semialgebra.

4. The 휎-algebra B generated by L1, is the same as the one generated by L2, and is also the same as the one generated by the set of all open sets: it is called the family of Borel sets of the line. The Borel sets of the extended real line R are deﬁned similarly. 5. More generally, the class of Borel sets in an arbitrary topological space is the smallest 휎-algebra containing all open sets. 6. Given 휎-algebras A, B in sets 푋, 푌, the product 휎-algebra A × B in the space 푋 × 푌 is the one generated by all elements 퐴 × 푌 and 푋 × 퐵 for 퐴 ∈ A and 퐵 ∈ B. y

A.2.2 Measures Probability is an example of the more general notion of a measure.

141 A. Background from mathematics

Definition A.2.3 A measurable space is a pair (푋, S) where S is a 휎-algebra of sets of 푋.A measure on a measurable space (푋, S) is a function 휇 : 퐵 → R+ that is 휎-additive: this means that for every countable family 퐴1, 퐴2,... of disjoint Ð Í elements of S we have 휇( 푖 퐴푖) = 푖 휇(퐴푖). A measure 휇 is 휎-finite if the whole space is the union of a countable set of subsets whose measure is finite. It is finite if 휇(푋) < ∞. It is a probability measure if 휇(푋) = 1. y

Example A.2.4 (Delta function) For any point 푥, the measure 훿푥 is defined as follows: ( 1 if 푥 ∈ 퐴, 훿푥 (퐴) = 0 otherwise. y Definition A.2.5 If 휇 is a measure, a point 푥 is called an atom if 휇(푥) > 0. y Generally, we will consider measures over either a countable set (a discrete measure for which the union of atoms has total measure) or an uncountable one, with no atoms. But mixed cases are possible. It is important to understand how a measure can be defined in practice. Alge- bras are generally simpler to grasp constructively than 휎-algebras; semialgebras are yet simpler. Suppose that 휇 is defined over a semialgebra L and is additive. Then it can always be uniquely extended to an additive function over the algebra generated by L. The following is an important theorem of measure theory. Proposition A.2.6 (Caratheodory’s extension theorem) Suppose that a nonnegative set function defined over a semialgebra L is 휎-additive. Then it can be extended uniquely to the 휎-algebra generated by L. Examples A.2.7 1. Let 푥 be point and let 휇(퐴) = 1 if 푥 ∈ 퐴 and 0 otherwise. In this case, we say that 휇 is concentrated on the point 푥.

2. Consider the the line R, and the algebra L1 defined in Example A.2.2.1. Let 푓 : R → R be a monotonic real function. We define a set function over L1 as follows. Let [푎푖, 푏푖),(푖 = 1, . . . , 푛) be a set of disjoint left-closed intervals. Ð Í Then 휇( 푖 [푎푖, 푏푖)) = 푖 푓 (푏푖) − 푓 (푎푖). It is easy to see that 휇 is additive. It is 휎-additive if and only if 푓 is left-continuous. 3. Let 퐵 = {0, 1}, and consider the set 퐵N of infinite 0-1-sequences, and the ∗ semialgebra L3 of Example A.2.2.3. Let 휇 : 퐵 → R+ be a function. Let us write 휇(푢퐵N) = 휇(푢) for all 푢 ∈ 퐵∗. Then it can be shown that the following

142 A.2. Measures

conditions are equivalent: 휇 is 휎-additive over L3; it is additive over L3; the equation 휇(푢) = 휇(푢0) + 휇(푢1) holds for all 푢 ∈ 퐵∗. 4. The nonnegative linear combination of any finite number of measures is also a measure. In this way, it is easy to construct arbitrary measures concentrated on a finite number of points. 5. Given two measure spaces (푋, A, 휇) and (푌, B, 휈) it is possible to define the product measure 휇 × 휈 over the measureable space (푋 × 푌, A × B). The definition is required to satisfy 휇×휈(퐴×퐵) = 휇(퐴)×휈(퐵), and is determined uniquely by this condition. If 휈 is a probability measure then, of course, 휇(퐴) = 휇 × 휈(퐴 × 푌). y Let us finally define measureable functions. Definition A.2.8 (Measureable functions) Let (푋, A), (푌, B) be two measureable spaces. A function 푓 : 푋 → 푌 is called measureable if and only if 푓 −1 (퐸) ∈ A for all 퐸 ∈ B. y The following is easy to check. Proposition A.2.9 Let (푋, A) be a measureable space and (R, B) be the measureable space of the real numbers, with the Borel sets. Then 푓 : 푋 → R is measureable if and only if all sets of the form 푓 −1 ((푟, ∞)) = {푥 : 푓 (푥) > 푟} are measureable, where 푟 is a rational number. Remark A.2.10 Example A.2.7.3 shows a particularly attractive way to define measures. Keep splitting the values 휇(푢) in an arbitrary way into 휇(푢0) and 휇(푢1), and the resulting values on the semialgebra define a measure. Exam- ple A.2.7.2 is less attractive, since in the process of defining 휇 on all intervals and only keeping track of finite additivity, we may end up with a monotonic function that is not left continuous, and thus with a measure that is not 휎-additive. In the subsection on probability measures over a metric space, we will find that even on the real line, there is a way to define measures in a step-by-step manner, and only checking for consistency along the way. y A probability space, according to the axioms introduced by Kolmogorov, is just a measureable space with a normed measure. Definition A.2.11 A probability space is a triple (푋, S, 푃) where (푋, S) is a measurable space and 푃 is a probability measure over it. Let (푋푖, S푖) (푖 = 1, 2) be measurable spaces, and let 푓 : 푋 → 푌 be a mapping. Then 푓 is measurable if and only if for each element 퐵 of S2, its inverse image

143 A. Background from mathematics

−1 푓 (퐵) is in S1. If 휇1 is a measure over (푋1, S1) then 휇2 deﬁned by 휇2 (퐴) = −1 휇1 ( 푓 (퐴)) is a measure over 푋2 called the measure induced by 푓 . y

A.2.3 Integral The notion of integral also generalizes to arbitrary measures, and is sometimes also used to define measures. First we define integral on very simple functions. Definition A.2.12 A measurable function 푓 : 푋 → R is called a step function if its range is finite. The set of step functions is closed with respect to linear combinations and also with respect to the operations ∧, ∨. Any such set of functions is called a Riesz space. y

Definition A.2.13 Given a step function 푓 which takes values 푥푖 on sets 퐴푖, and a finite measure 휇, we define ∫ ∫ ∑︁ 휇( 푓 ) = 휇 푓 = 푓 푑휇 = 푓 (푥)휇(푑푥) = 푥푖휇(퐴푖). 푖 y This is a linear positive functional on the set of step functions. Moreover, it can be shown that it is continuous on monotonic sequences: if 푓푖 & 0 then 휇 푓푖 & 0. The converse can also be shown: Let 휇 be a linear positive functional on step functions that is continuous on monotonic sequences. Then the set function 휇(퐴) = 휇(1퐴) is a finite measure. Proposition A.2.14 Let E be any Riesz space of functions with the property that 1 ∈ E. Let 휇 be a positive linear functional on E continuous on monotonic sequences, with 휇1 = 1. The functional 휇 can be extended to the set E+ of monotonic limits of nonnegative elements of E, by continuity. In case when E is the set of all step functions, the set E+ is the set of all nonnegative measurable functions. Now we extend the notion of integral to a wider class of functions. Definition A.2.15 Let us fix a finite measure 휇 over a measurable space (푋, S). A measurable function 푓 is called integrable with respect to 휇 if 휇| 푓 |+ < ∞ and 휇| 푓 |− < ∞. In this case, we define 휇 푓 = 휇| 푓 |+ − 휇| 푓 |−. y The set of integrable functions is a Riesz space, and the positive linear functional 휇 on it is continuous with respect to monotonic sequences. The continuity over monotonic sequences also implies the following theorem.

144 A.2. Measures

Proposition A.2.16 (Bounded convergence theorem) Suppose that functions 푓푛 are integrable and | 푓푛| < 푔 for some integrable function 푔. Then 푓 = lim푛 푓푛 is integrable and 휇 푓 = lim푛 휇 푓푛. Deﬁnition A.2.17 Two measurable functions 푓 , 푔 are called equivalent with respect to measure 휇 if 휇( 푓 − 푔) = 0. y For two-dimensional integration, the following theorem holds. Proposition A.2.18 (Fubini theorem) Suppose that function 푓 (·, ·) is integrable over the space (푋 × 푌, A × B, 휇 × 휈). Then for 휇-almost all 푥, the function 푓 (푥, ·) is integrable over (푌, B, 휈), and the function 푥 ↦→ 휈푦 푓 (푥, 푦) is integrable over (푋, A, 휇) with (휇 × 휈) 푓 = 휇푥 휇 푦 푓 . To express a continuity property of measures, we can say the following (recall the deﬁnition of 퐶(푋) in Example A.1.28.8). Proposition A.2.19 Let 푋 be a metric space and 휇 a measure. Then 휇 is a bounded (and thus continuous) linear functional over the space 퐶(푋).

A.2.4 Density When does one measure have a density function with respect to another? Definition A.2.20 Let 휇, 휈 be two measures over the same measurable space. We say that 휈 is absolutely continuous with respect to 휇, or that 휇 dominates 휈, if for each set 퐴, 휇(퐴) = 0 implies 휈(퐴) = 0. y Every nonnegative integrable function 푓 defines a new measure 휈 via the formula 휈(퐴) = 휇( 푓 · 1퐴). This measure 휈 is absolutely continuous with respect to 휇. The Radon-Nikodym theorem says that the converse is also true. Proposition A.2.21 (Radon-Nikodym theorem) If 휈 is dominated by 휇 then there is a nonnegative integrable function 푓 such that 휈(퐴) = 휇( 푓 · 1퐴) for all measurable sets 퐴. The function 푓 is defined uniquely to within equivalence with respect to 휇. Definition A.2.22 The function 푓 of the Radom-Nikodym Theorem above is called the density of 휈 with respect to 휇. We will denote it by

휇(푑푥) 푑휇 푓 (푥) = = . 휈(푑푥) 푑휈 y The following theorem is also standard.

145 A. Background from mathematics

Proposition A.2.23 (Chain rule and inverse function) 1. Let 휇, 휈, 휂 be measures such that 휂 is absolutely continuous with respect to 휇 and 휇 is absolutely continuous with respect to 휈. Then the “chain rule” holds:

푑휂 푑휂 푑휇 = . (A.2.1) 푑휈 푑휇 푑휈

휈(푑푥) 2. If 휇 (푑푥) > 0 for all 푥 then 휇 is also absolutely continuous with respect to 휈 and −1 휇 (푑푥) 휈(푑푥) 휈(푑푥) = 휇 (푑푥) . There is a natural distance to be used between measures, though later we will see that it is not the preferred one in metric spaces. Deﬁnition A.2.24 (Total variation distance) Let 휇, 휈 be two measures, then both are dominated by some measure 휂 (for example by 휂 = 휇 + 휈). Let their densities with respect to 휂 be 푓 and 푔. Then we deﬁne the total variation distance of the two measures as 퐷(휇, 휈) = 휂(| 푓 − 푔|). It is independent of the dominating measure 휂. y Example A.2.25 Suppose that the space 푋 can be partitioned into disjoint sets 퐴, 퐵 such that 휈(퐴) = 휇(퐵) = 0. Then 퐷(휇, 휈) = 휇(퐴)+휈(퐵) = 휇(푋)+휈(푋). y

A.2.5 Random transitions What is just a transition matrix in case of a Markov chain also needs to be defined more carefully in the non-discrete cases. We follow the definition given in [40]. Definition A.2.26 Let (푋, A), (푌, B) be measureable spaces (defined in Sub- section A.2.2). Suppose that a family of probability measures Λ = {휆푥 : 푥 ∈ 푋 } on B is given. We call it a probability kernel, (or Markov kernel, or conditional distribution) if the map 푥 ↦→ 휆푥 퐵 is measurable for each 퐵 ∈ B. y When 푋, 푌 are finite sets then 휆 is a Markov transition matrix. The following theorem shows that 휆 assigns a joint distribution over the space (푋 × 푌, A × B) to each input distribution 휇. Proposition A.2.27 For each nonnegative A×B-measureable function 푓 over 푋×푌, 1. The function 푦 ↦→ 푓 (푥, 푦) is B-measurable for each fixed 푥. 푦 2. The function 푥 ↦→ 휆푥 푓 (푥, 푦) is A-measurable. 푥 푦 3. The integral 푓 ↦→ 휇 휆푥 푓 (푥, 푦) defines a measure on A × B.

146 A.2. Measures

The above proposition allows to define a mapping over measures. Definition A.2.28 According to the above proposition, given a probability kernel Λ, to each measure 휇 over A corresponds a measure over A × B. We will denote its marginal over B as Λ∗휇. (A.2.2) For every measurable function 푔( 푦) over 푌, we can define the measurable funct- 푦 ion 푓 (푥) = 휆푥 푔 = 휆푥 푔( 푦): we write 푓 = Λ푔. (A.2.3) y The operator Λ is linear, and monotone with Λ1 = 1. By these definitions, we have 휇(Λ푔) = (Λ∗휇)푔. (A.2.4) An example is the simple case of a deterministic mapping:

Example A.2.29 Let ℎ : 푋 → 푌 be a measureable function, and let 휆푥 be the measure 훿ℎ(푥) concentrated on the point ℎ(푥). This operator, denoted Λℎ is, in fact, a deterministic transition, and we have Λℎ푔 = 푔 ◦ ℎ. In this case, we will simplify the notation as follows: ∗ ∗ ℎ 휇 = Λℎ. y

A.2.6 Probability measures over a metric space We follow the exposition of [5]. Whenever we deal with probability measures on a metric space, we will assume that our metric space is complete and separable (Polish space). Let X = (푋, 푑) be a complete separable metric space. Then X gives rise to a measurable space, where the measurable sets are its Borel sets. It can be shown that, if 퐴 is a Borel set and 휇 is a ﬁnite measure then there are sets 퐹 ⊆ 퐴 ⊆ 퐺 where 퐹 is an 퐹휎 set, 퐺 is a 퐺훿 set, and 휇(퐹) = 휇(퐺). It can be shown that a measure is determined by its values on the elements of any a basis of open sets closed under intersections. The following proposition follows then essentially from Proposition A.2.6. Proposition A.2.30 Let B be a basis of open sets closed under intersections. Let B∗ be the set algebra generated by this basis and let 휇 be any 휎-additive set function on B∗ with 휇(푋) = 1. Then 휇 can be extended uniquely to a probability measure.

147 A. Background from mathematics

Weak topology One can introduce the notion of convergence of measures in a number of ways. We have already introduced the total variation distance in Deﬁnition A.2.24 above. But in some cases, the requirement of being close in this distance is too strong. Let M(X) be the set of probability measures on the metric space X. Let 푥푛 be a sequence of points converging to point 푥 but with 푥푛 ≠ 푥. We would like to say that the delta measure 훿푥푛 (concentrated on point 푥푛, see Example A.2.4) converges to

훿푥. But the total variation distance 퐷(훿푥푛 , 훿푥) is 2 for all 푛. Deﬁnition A.2.31 (Weak convergence) We say that a sequence of probability measures 휇푛 over a metric space (푋, 푑) weakly converges to measure 휇 if for all bounded continuous real functions 푓 over 푋 we have 휇푛 푓 → 휇 푓 . For a bounded continuous function 푓 and real numbers 푐 let

퐴 푓 ,푐 = {휇 : 휇 푓 < 푐} y

A topology of weak convergence (M, 휏푤) can be defined using a number of different subbases. The one used in the original definition is the subbasis consisting of all sets of the form 퐴 푓 ,푐 above. We also get a subbasis (see for example [40]) if we restrict ourselves to the set Lip(푋) of Lipschitz functions defined in (A.1.2). Another possible subbasis giving rise to the same topology consists of all sets of the form

퐵퐺,푐 = {휇 : 휇(퐺) > 푐} (A.2.5) for open sets 퐺 and real numbers 푐. Let us find some countable subbases. Since the space X is separable, there is a sequence 푈1, 푈2,... of open sets that forms a basis of X. Then we can restrict the subbasis of the space of measures to those sets 퐵퐺,푐 where 퐺 is the union of a finite number of basis elements 푈푖 of X and 푐 is rational. This way, the space (M, 휏푤) itself has the second countability property. It is more convenient to define a countable subbasis using bounded continuous functions 푓 , since the function 휇 ↦→ 휇 푓 is continuous on such functions, while 휇 ↦→ 휇푈 is typically not continuous when 푈 is an open set. Example A.2.32 If X = R and 푈 is the open interval (0, 1), the sequence of probability measures 훿1/푛 (concentrated on 1/푛) converges to 훿0, but 훿1/푛 (푈) = 1, and 훿0 (푈) = 0. y

148 A.2. Measures

For some fixed dense set 퐷, let F1 = F1 (퐷) be the set of functions introduced in Definition A.1.32. Definition A.2.33 We say that a set 퐴 is a continuity set of measure 휇 if 휇(휕퐴) = 0: the boundary of 퐴 has measure 0. y Proposition A.2.34 The following conditions are equivalent:

1. 휇푛 weakly converges to 휇.

2. 휇푛 푓 → 휇 푓 for all 푓 ∈ F1.

3. For every Borel set 퐴, that is a continuity set of 휇, we have 휇푛 (퐴) → 휇(퐴).

4. For every closed set 퐹, lim inf푛 휇푛 (퐹) ≥ 휇(퐹).

5. For every open set 퐺, lim sup푛 휇푛 (퐺) ≤ 휇(퐺). Deﬁnition A.2.35 To deﬁne the topological space M(푋) of the set of measures over the metric space 푋, we choose as subbasis

휎M (A.2.6) the sets {휇 : 휇 푓 < 푟} and {휇 : 휇 푓 > 푟} for all 푓 ∈ F1 and 푟 ∈ Q. y The simple functions we introduced can also be used to define measure and integral in themselves. Recall the definition of the set E in (A.1.3).T This set is a Riesz space as defined in Subsection A.2.3. A reasoning combining Proposi- tions A.2.6 and A.2.14 gives the following. Proposition A.2.36 Suppose that a positive linear functional 휇 with 휇1 = 1 is defined on E that is continuous with respect to monotone convergence. Then 휇 can ∫ be extended uniquely to a probability measure over X with 휇 푓 = 푓 (푥)휇(푑푥) for all 푓 ∈ E. Having a topology over the set of measures we can also extend Proposi- tion A.2.19: Proposition A.2.37 Let 푋 be a complete separable metric space and M(푋) the space of bounded measures over 푋 with the weak topology. The function (휇, 푓 ) ↦→ 휇 푓 is a continuous function M(푋) × 퐶(푋) → R. As mentioned above, for an open set 퐺 the value 휇(퐺) is not a continuous function of the measure 휇. We can only say the following: Proposition A.2.38 Let 푋 be a complete separable metric space, and M(푋) the space of bounded measures over 푋 with the weak topology, and 퐺 ⊆ 푋 an open set. The function 휇 ↦→ 휇(퐺) is lower semicontinuous.

149 A. Background from mathematics

Distances for the weak topology The definition of measures in the style of Proposition A.2.36 is not sufficiently constructive. Consider a gradual definition of the measure 휇, extending it to more and more elements of E, while keeping the positivity and linearity property. It can happen that the function 휇 we end up with in the limit, is not continuous with respect to monotone convergence. Let us therefore metrize the space of measures: then an arbitrary measure can be defined as the limit of a Cauchy sequence of simple meaures. One metric that generates the topology of weak convergence is the following. Definition A.2.39 (Prokhorov distance) The Prokhorov distance 휌(휇, 휈) of two measures is the infimum of all those 휀 for which, for all Borel sets 퐴 we have (using the notation (A.1.1))

휇(퐴) ≤ 휈(퐴휀) + 휀. y It can be shown that this is a metric and it generates the weak topology. In computer science, it has been reinvented by the name of “earth mover distance”. The following important result helps visualize it: Proposition A.2.40 (Coupling Theorem, see [50]) Let 휇, 휈 be two probability measures over a complete separable metric space X with 휌(휇, 휈) ≤ 휀. Then there is a probability measure P on the space X × X with marginals 휇 and 휈 such that for a pair of random variables 휉, 휂 having joint distribution P we have

P{푑(휉, 휂) > 휀} ≤ 휀.

Since weak topology has the second countability property, the metric space deﬁned by the distance 휌(·, ·) is separable. This can also be seen directly: let us deﬁne a dense set in weak topology.

Definition A.2.41 For each point 푥, let us define by 훿푥 the measure which con- centrates its total weight 1 in point 푥. Let 퐷 be a countable everywhere dense set of points in 푋. Let 퐷M be the set of finite convex rational combinations of measures of the form 훿푥푖 where 푥푖 ∈ 퐷, that is those probability measures that are concentrated on finitely many points of 퐷 and assign rational values to them. y

It can be shown that 퐷M is everywhere dense in the metric space (M(푋), 휌); so, this space is separable. It can also be shown that (M(푋), 휌) is complete. Thus, a measure can be given as the limit of a Cauchy sequence of elements 휇1, 휇2,... of 퐷M.

150 A.2. Measures

The definition of the Prokhorov distance uses quantification over all Borel sets. However, in an important simple case, it can be handled efficiently. Proposition A.2.42 Assume that measure 휈 is concentrated on a finite set of points 푆 ⊂ 푋. Then the condition 휌(휈, 휇) < 휀 is equivalent to the finite set of conditions

휇(퐴휀) > 휈(퐴) − 휀 (A.2.7) for all 퐴 ⊂ 푆. Another useful distance for measures over a bounded space is the Wasserstein distance. Deﬁnition A.2.43 Over a bounded metric space, we deﬁne the Wasserstein distance by

푊(휇, 휈) = sup |휇 푓 − 휈 푓 |. 푓 ∈Lip1 (푋) y The Wasserstein distance also has a coupling representation: Proposition A.2.44 (See [51]) Let 휇, 휈 be two probability measures over a complete separable metric space X with 휌(휇, 휈) ≤ 휀. Then there is a probability measure P on the space X × X with marginals 휇 and 휈 such that ∫ 푑(푥, 푦) P(푑푥푑 푦) = 푊(휇, 휈).

Proposition A.2.45 The Prokhorov and Wasserstein metrics are equivalent: the identity function creates a uniformly continuous homeomorphism between the two metric spaces.

Proof. Let 푀 = sup푥,푦 ∈푋 푑(푥, 푦). The proof actually shows for 휀 < 1:

휌(휇, 휈) ≤ 휀 ⇒ 푊(휇, 휈) ≤ (푀 + 1)휀, 푊(휇, 휈) ≤ 휀2 ⇒ 휌(휇, 휈) ≤ 휀.

Suppose 휌(휇, 휈) ≤ 휀 < 1. By the Coupling Theorem (Proposition A.2.40), there are random variables 휉, 휂 over 푋 with a joint distribution, and having respectively the distribution 휇 and 휈, such that P{푑(휉, 휂) > 휀} ≤ 휀. Then we have

|휇 푓 − 휈 푓 | = | E 푓 (휉) − E 푓 (휂)| ≤ E | 푓 (휉) − 푓 (휂)| ≤ 휀 P{ 휌(휉, 휂) ≤ 휀} + 푀 P{ 휌(휉, 휂) > 휀} ≤ (푀 + 1)휀.

151 A. Background from mathematics

2 퐴 Now suppose 푊(휇, 휈) ≤ 휀 < 1. For a Borel set 퐴 deﬁne 푔휀 (푥) = |1 − + 퐴 퐴 휀 퐴 휌(푥, 퐴)/휀| . Then we have 휇(퐴) ≤ 휇(푔휀 ) and 휈푔휀 ≤ 휈(퐴 ). Further 휀푔휀 ∈ Lip1, 2 퐴 퐴 2 and hence 푊(휇, 휈) ≤ 휀 implies 휀휇(푔휀 ) ≤ 휀휈(휀푔휀 ) + 휀 . This concludes the proof by

퐴 퐴 휀 휇(퐴) ≤ 휇(푔휀 ) ≤ 휈(휀푔휀 ) + 휀 ≤ 휈(퐴 ) + 휀.

Relative compactness Convergence over the set of measures, even over a noncompact space, is sometimes obtained via compactness properties. Deﬁnition A.2.46 A set Π of measures in (M(푋), 휌) is called sequentially compact if every sequence of elements of Π contains a convergent subsequence. A set of Π of measures is called tight if for every 휀 there is a compact set 퐾 such that 휇(퐾) > 1 − 휀 for all 휇 in Π. y The following important theorem is using our assumptions of separability and completeness of the underlying metric space (푋, 휌). Proposition A.2.47 (Prokhorov) A set of measures is sequentially compact if and only if it is tight and if and only if its closure is compact in the metric space (M(푋), 휌). The following, simplest example is interesting in its own right. Example A.2.48 The one-element set {휇} is compact and therefore by Prokhorov’s theorem tight. Here, tightness says that for each 휀 a mass of size 1 − 휀 of 휇 is concentrated on some compact set. y The following theorem strengthens the observation of this example. Proposition A.2.49 A ﬁnite measure 휇 over a separable complete metric space has the property

휇(퐵) = sup{휇(퐾) : compact 퐾 ⊆ 퐵} for all Borel sets 퐵. In case of compact metric spaces, the following known theorem helps. Proposition A.2.50 The metric space (M(X), 휌) of measures is compact if and only if the underlying metric space (푋, 푑) is compact.

152 A.2. Measures

So, if (푋, 푑) is not compact then the set of measures is not compact. But still, by Proposition A.2.49, each ﬁnite measure is “almost” concentrated on a compact set.

Semimeasures Let us generalize then notion of semimeasure for the case of general Polish spaces. We use Proposition A.2.36 as a starting point. Definition A.2.51 A function 푓 ↦→ 휇 푓 defined over the set of all bounded continuous functions is called a semimeasure if it has the following properties: a) Nonnegative: 푓 ≥ 0 implies 휇 푓 ≥ 0. b) Positive homogenous: we have 휇(푎 푓 ) = 푎휇 푓 for all nonnegative real 푎 > 0. c) Superadditive: 휇( 푓 + 푔) ≥ 휇 푓 + 휇푔. d) Normed: 휇1 = 1. y Remark A.2.52 The weaker requirement 휇1 ≤ 1 suffices, but if we have 휇1 ≤ 1 we can always set simply 휇1 = 1 without violating the other requirements. y

Functionals of measures For a deeper study of randomness tests, we will need to characterize certain functionals of ﬁnite measures over a Polish space. Proposition A.2.53 Let X = (푋, 푑) be a complete separable metric space with M˜ (X) the set of ﬁnite measures over it. A weakly continuous linear function 퐹 : M˜ (X) → R can always be written as 퐹(휇) = 휇 푓 for some bounded continuous function 푓 over X.

Proof. Deﬁne 푓 (푥) = 퐹(훿푥). The weak continuity of 퐹 implies that 푓 (푥) is continuous. Let us show that it is bounded. Suppose it is not. Then there is a −푛 sequence of distinct points 푥푛 ∈ 푋, 푛 = 1, 2,... with 푓 (푥푛) > 2 . Let 휇 be the Í −푛 Í −푛 Í measure 푛 2 훿푥푛 , then by linearity we have 퐿(휇) = 푛 2 푓 (푥푛) > 푛 1 = ∞, which is not allowed. The function 휇 ↦→ 휇 푓 is continuous and coincides with 퐹(휇) on the dense set of points 퐷M, so they are equal.

153

B Constructivity

B.1 Computable topology

B.1.1 Representations There are several equivalent ways that notions of computability can be extended to spaces like real numbers, metric spaces, measures, and so on. We use the the concepts of numbering (notation) and representation, as defined in [57]. Notation B.1.1 We will denote by N the set of natural numbers and by B the set {0, 1}. Given a set (an alphabet) Σ we denote by ΣN = ΣN the set of infinite sequences with elements in Σ. If for some finite or infinite sequences 푥, 푦, 푧, 푤, we have 푧 = 푤푥 푦 then we write 푤 v 푧 (푤 is a prefix of 푧) and 푥 ⊳ 푧. After [57], let us define the wrapping function 휄 : Σ∗ → Σ∗ by

휄(푎1푎2 ··· 푎푛) = 110푎10푎20 ··· 푎푛011. (B.1.1)

Note that 푙(휄(푥)) = (2푙(푥) + 5) ∨ 6. (B.1.2) ∗ N For strings 푥, 푥푖 ∈ Σ , 푝, 푝푖 ∈ Σ , 푘 ≥ 1, appropriate tupling functions h푥1, . . . , 푥푘i, h푥, 푝i, h푝, 푥i, and so on can be defined with the help of h·, ·i and 휄(·). y Definition B.1.2 Given a countable set 퐶, a numbering (or notation) of 퐶 is a surjective partial mapping 훿 : N → 퐶. Given some finite alphabet Σ ⊇ {0, 1} and an arbitrary set 푆, a representation of 푆 is a surjective partial mapping 휒 : ΣN → 푆.A naming system is a notation or a representation. y Here are some standard naming systems: 1. id, the identity over Σ∗ or ΣN.

2. 휈N, 휈Z, 휈Q for the set of natural numbers, integers and rational numbers.

155 B. Constructivity

3. Cf : ΣN → 2N, the characteristic function representation of sets of natural numbers, is defined by Cf( 푝) = {푖 : 푝(푖) = 1}. 4. En : ΣN → 2N, the enumeration representation of sets of natural numbers, is defined by En( 푝) = {푛 ∈ N : 110푛+111 ⊳ 푝}. N Δ∗ ∗ 5. For Δ ⊆ Σ, EnΔ : Σ → 2 , the enumeration representation of subsets of Δ , ∗ is defined by EnΔ ( 푝) = {푤 ∈ Σ : 휄(푤) ⊳ 푝}. Using Turing machines with infinite input tapes, work tapes and output tapes, one can define names for all computable functions between spaces that are Cartesian products of terms of the kind Σ∗ and ΣN. (One wonders whether Σ∗ ∪ ΣN is not needed, since a Turing machine with an infinite input tape may still produce only a finite output. But in this case one can also encode the result into an infinite sequence.) Then, the notion of computability can be transferred to other spaces as follows.

Deﬁnition B.1.3 Let 훿푖 : 푌푖 → 푋푖, 푖 = 1, 0 be naming systems of the spaces 푋푖. Let 푓 :⊆ 푋1 → 푋0, 푔 :⊆ 푌1 → 푌0. We say that function 푔 realizes function 푓 if

푓 (훿1 ( 푦)) = 훿0 (푔( 푦)) (B.1.3) holds for all 푦 for which the left-hand side is defined. Realization of multi-argument functions is defined similarly. We say that a function 푓 : 푋1 × 푋2 → 푋0 is (훿1, 훿2, 훿0)-computable if there is a computable function 푔 :⊆ 푌1 × 푌2 → 푌0 realizing it. In this case, a name for 푓 is naturally derived from a name of 푔.1 y Definition B.1.4 For representations 휉, 휂, we write 휉 ≤ 휂 if there is a computable partial function 푓 : ΣN → ΣN with 휉(푥) = 휂( 푓 (푥)). In words, we say that 휉 is reducible to 휂, or that 푓 reduces (translates) 휉 to 휂. There is a similar definition of reduction for notations. We write 휉 ≡ 휂 if 휉 ≤ 휂 and 휂 ≤ 휉. y

B.1.2 Constructive topological space Let us start with the definition of topology with the help of a subbasis of (possibly empty) open sets. Definition B.1.5 A constructive topological space X = (푋, 휎, 휈) is a topological space over a set 푋 with a subbasis 휎 effectively enumerated (not necessarily without repetitions) as a list 휎 = {휈(1), 휈(2),...}, and having the 푇0 property (thus, every point is determined uniquely by the subset of elements of 휎 containing it).

1Any function 푔 realizing 푓 via (B.1.3) automatically has a certain extensionality property: if 0 0 훿1 ( 푦) = 훿1 ( 푦 ) then 푔( 푦) = 푔( 푦 ).

156 B.1. Computable topology

By definition, a constructive topological space satisfies the second countability axiom. We obtain a basis 휎∩ of the space X by taking all possible finite intersections of elements of 휎. It is easy to produce an effective enumeration for 휎∩ from 휈. We will denote this enumeration by 휈∩. This basis will be called the canonical basis. For every nonempty subset 푌 of the space 푋, the subspace of 푋 will naturally get the same structure Y = (푌, 휎, 휈) defined by {푉 ∩ 푌 : 푉 ∈ 휎}. The product operation is defined over constructive topological spaces in the natural way. y Remark B.1.6 The definition of a subspace shows that a constructive topological space is not a “constructive object” by itself, since the set 푋 itself is not necessarily given effectively. For example, any nonempty subset of the real line is a constructive topological space, as a subspace of the real line. y Examples B.1.7 The following examples will be used later. 1. A discrete topological space, where the underlying set is finite or countably infinite, with a fixed enumeration. 2. The real line, choosing the basis to be the open intervals with rational endpoints with their natural enumeration. Product spaces can be formed to give the Euclidean plane a constructive topology. > 3. The real line R, with the subbasis 휎R defined as the set of all open intervals < (−∞, 푏) with rational endpoints 푏. The subbasis 휎R, defined similarly, leads to another topology. These two topologies differ from each other and from the usual one on the real line, and they are not Hausdorff spaces. 4. This is the constructive version of Example A.1.4.5. Let 푋 be a set with a constructive discrete topology, and 푋N the set of infinite sequences with elements from 푋, with the product topology: a natural enumerated basis is also easy to define. y

Definition B.1.8 Due to the 푇0 property, every point in our space is determined uniquely by the set of open sets containing it. Thus, there is a representation 훾X of X defined as follows. We say that 훾X ( 푝) = 푥 if EnΣ ( 푝) = {푤 : 푥 ∈ 휈(푤)}. If 훾X ( 푝) = 푥 then we say that the infinite sequence 푝 is a complete name of 푥: it encodes all names of all subbasis elements containing 푥. From now on, we will call 훾X the complete standard representation of the space X. y

157 B. Constructivity

Remark B.1.9 Here it becomes important that representations are allowed to be partial functions. y Deﬁnition B.1.10 (Constructive open sets) In a constructive topological space X = (푋, 휎, 휈), a set 퐺 ⊆ 푋 is called constructive open, or lower semicomputable Ð ∩ open in set 퐵 if there is a computably enumerable set 퐸 with 퐺 = 푤∈퐸 휈 (푤)∩퐵. It is constructife open if it is constructive open in 푋. y In the special kind of spaces in which randomness has been developed until now, constructive open sets have a nice characterization:

Proposition B.1.11 Assume that the space X = (푋, 휎, 휈) has the form 푌1 ×· · ·×푌푛 ∗ N where each 푌푖 is either Σ or Σ . Then a set 퐺 is constructive open iﬀ it is open and Ñ the set {(푤1, . . . , 푤푛) : 푖 휈(푤푖) ⊂ 퐺 } is recursively enumerable. Proof. The proof is not diﬃcult, but it relies on the discrete nature of the space Σ∗ and on the fact that the space ΣN is compact and its basis consists of sets that are open and closed at the same time. It is easy to see that if two sets are constructive open then so is their union. The above remark implies that a space having the form 푌1 ×· · ·×푌푛 where each 푌푖 is either Σ∗ or ΣN, also the intersection of two recursively open sets is recursively open. We will see that this statement holds, more generally, in all computable metric spaces.

B.1.3 Computable functions

Definition B.1.12 Let X푖 = (푋푖, 휎푖, 휈푖) be constructive topological spaces, and let 푓 : 푋1 → 푋0 be a function. As we know, 푓 is continuous iff the inverse image 푓 −1 (퐺) of each open set 퐺 is open in its domain. Computability is an effective version of continuity: it requires that the inverse image of basis elements is uniformly constructively open. More precisely, 푓 : 푋1 → 푋0 is computable if the set Ø 푓 −1 (푉) × {푉} ∈ ∩ 푉 휎0 ∩ ∩ is a constructive open subset of 푋1 × 휎0 . Here the basis 휎0 of X0 is treated as a discrete constructive topological space, with its natural enumeration. A partial function is computable if its restriction to the subspace that is its domain, is computable. y

158 B.1. Computable topology

The above deﬁnition depends on the enumerations 휈1, 휈0. The following theorem shows that this computability coincides with the one obtained by transfer via the representations 훾X푖 .

Proposition B.1.13 (Hertling) For 푖 = 0, 1, let X푖 = (푋푖, 휎푖, 휈푖) be constructive topological spaces. Then a function 푓 : 푋1 → 푋0 is computable iff it is (훾X1 , 훾X0 )- computable for the representations 훾X푖 defined above. The notion of computable functions helps define morphisms between constructive topological spaces.

Definition B.1.14 Let us call two spaces 푋1 and 푋0 effectively homeomorphic if there are computable maps between them that are inverses of each other. In the special case when 푋0 = 푋1, we say that the enumerations of subbases 휈0, 휈1 are equivalent if the identity mapping is a effective homeomorphism. This means that there are recursively enumerable sets 퐹, 퐺 such that

Ø ∩ Ø ∩ 휈1 (푣) = 휈0 (푤) for all 푣, 휈0 (푤) = 휈1 (푣) for all 푤. (푣,푤) ∈퐹 (푤,푣) ∈퐺 y

B.1.4 Computable elements and sequences Let us deﬁne computable elements.

Deﬁnition B.1.15 Let U = ({0}, 휎0, 휈0) be the one-element space turned into a trivial constructive topological space, and let X = (푋, 휎, 휈) be another constructive topological space. We say that an element 푥 ∈ 푋 is computable if the function 0 ↦→ 푥 is computable. It is easy to see that this is equivalent to the requirement that the set {푢 : 푥 ∈ 휈(푢)} is recursively enumerable. Let X 푗 = (푋 푗, 휎푗, 휈 푗), for 푖 = 0, 1 be constructive topological spaces. A sequence 푓푖, 푖 = 1, 2,... of functions with 푓푖 : 푋1 → 푋0 is a computable sequence of computable functions if (푖, 푥) ↦→ 푓푖 (푥) is a computable function. y It is easy to see that this statement is equivalent to the statement that there is a recursive function computing from each 푖 a name for the computable function 푓푖. To do this formally, one sometimes using the s-m-n theorem of recursion theory, or calls the method “curriﬁcation”. The notion of computability can be relativized.

Deﬁnition B.1.16 Let 푋 푗, 푗 = 1, 2 be two constructive topological spaces (or more general, representations) and let 푥 푗 ∈ 푋 푗 be given. We say that 푥2 is 푥1-

159 B. Constructivity

computable if there is a computable partial function 푓 : 푋1 → 푋2 with 푓 (푥1) = 푥2. y Remark B.1.17 The requirement that 푓 computes 푦 from every representation of 푥 makes 푥-computability a very diﬀerent requirement of mere Turing reducibility of 푦 to 푥. We will point out an important implication of this diﬀerence later, in the notion of 휇-randomness. It is therefore preferable not to use the term “oracle computation” when referring to computations on representations . y

B.1.5 Semicomputability Lower semicomputability is a constructive version of lower semicontinuity, as given in Definition A.1.14, but the sets that are required to be open there are required to be constructive open here. The analogue of Proposition A.1.17 and Corollary A.1.19 holds also: a lower semicomputable function is the supremum of simple constant functions defined on basis elements, and a lower semicomputable function defined on a subset can always be extended to one over the whole space. Definition B.1.18 Let X = (푋, 휎, 휈) be a constructive topological space. A partial function 푓 : 푋 → R+ with domain 퐷 is called lower semicomputable if the set {(푥, 푟) : 푓 (푥) > 푟} is a constructive open subset of 퐷 × R+. We define the notion of 푥-lower semicomputable similarly to the notion of 푥-computability. y < < Let Y = (R+, 휎R, 휈R) be the effective topological space introduced in Ex- > ample B.1.7.2, in which 휈R is an enumeration of all open intervals of the form (푟, ∞] with rational 푟. The following characterization is analogous to Proposi- tion A.1.15. Proposition B.1.19 A function 푓 : 푋 → R is lower semicomputable if and only if > it is (휈, 휈R)-computable. As a name of a computable function, we can use the name of the enumeration algorithm derived from the definition of computability, or the name derivable using this representation theorem. The following example is analogous to Example A.1.16.

Example B.1.20 The indicator function 1퐺 (푥) of an arbitrary constructive open set 퐺 is lower semicomputable. y The following fact is analogous to Proposition A.1.17:

160 B.1. Computable topology

Proposition B.1.21 Let 푓1, 푓2,... be a computable sequence of lower semicomputable functions (via their names) over a constructive topological space X. Then sup푖 푓푖 is also lower semicomputable. The following fact is analogous to Proposition A.1.18: Proposition B.1.22 Let X = (푋, 휎, 휈) be a constructive topological space with ∩ enumerated basis 훽 = 휎 and 푓 : 푋 → R+ a lower semicomputable function. Then there is a lower semicomputable function 푔 : 훽 → R+ (where 훽 is taken with the discrete topology) with 푓 (푥) = sup푥 ∈훽 푔(훽). In the important special case of Cantor spaces, the basis is given by the set of finite sequences. In this case we can also require the function 푔(푤) to be monotonic in the words 푤: Proposition B.1.23 Let 푋 = ΣN be a Cantor space as defined in Example B.1.7.4. Then 푓 : 푋 → R+ is lower semicomputable if and only if there is a lower semicom- ∗ ∗ putable function 푔 : Σ → R+ (where Σ is taken as a discrete space) monotonic with respect to the relation 푢 v 푣, with 푓 (휉) = sup푢v휉 푔(푢). Remark B.1.24 In the above representation, we could require the function 푔 to be computable. Indeed, we can just replace 푔(푤) with 푔0(푤) where 푔0(푤) = max푣v푤 푔(푣, 푙(푤)), and 푔(푣, 푛) is as much of 푔(푣) as can be computed in 푛 steps. y Upper semicomputability is defined analogously: Definition B.1.25 We will call a closed set upper semicomputable (co-recursively enumerable) if its complement is a lower semicomputable (r.e.) open set. y The proof of the following statement is not difficult.

Proposition B.1.26 Let X푖 = (푋푖, 휎푖, 휈푖) for 푖 = 1, 2, 0 be constructive topological spaces, and let 푓 : 푋1 × 푋2 → 푋0, and assume that 푥1 ∈ 푋1 is a computable element.

1. If 푓 is computable then 푥2 ↦→ 푓 (푥1, 푥2) is also computable.

2. If X0 = R, and 푓 is lower semicomputable then 푥2 ↦→ 푓 (푥1, 푥2) is also lower semicomputable.

B.1.6 Eﬀective compactness Compactness is an important property of topological spaces. In the constructive case, however, a constructive version of compactness is frequently needed.

161 B. Constructivity

Definition B.1.27 A constructive topological space 푋 with canonical basis 훽 has recognizeable covers if the set Ø {푆 ⊆ 훽 : 푆 is finite and 푈 = 푋 } 푈 ∈푆 is recursively enumerable. A compact space with recognizeable covers will be called effectively compact. y Example B.1.28 Let 훼 ∈ [0, 1] be a real number such that the set of rationals less than 훼 is recursively enumerable but the set of rationals larger than 훼 are Í −퐾 (푥) not. (It is known that there are such numbers, for example 푥 ∈N 2 .) Let 푋 be the subspace of the real line on the interval [0, 훼], with the induced topology. A basis of this topology is the set 훽 of all nonempty intervals of the form 퐼∩[0, 훼], where 퐼 is an open interval with rational endpoints. This space is compact, but not effectively so. y The following is immediate. Proposition B.1.29 In an effectively compact space, in every recursively enumerable set of basis elements covering the space one can effectively find a finite covering. It is known that every closed subset of a compact space is compact. This statement has a constructive counterpart: Proposition B.1.30 Every upper semicomputable closed subset 퐸 of an effectively compact space 푋 is also effectively compact.

Proof. 푈1, . . . , 푈푘 be a finite set covering 퐸. Let us enumerate a sequence 푉푖 of canonical basis elements whose union is the complement of 퐸. If 푈1 ∪ · · · ∪ 푈푘 covers 퐸 then along with the sequence 푉푖 it will cover 푋, and this will be recognized in a finite number of steps. On an effectively compact space, computable functions have computable ex- trema: Proposition B.1.31 Let 푋 be an effectively compact space, let 푓 (푥) be lower semicomputable function on 푋. Then the infimum (which is always a minimum) of 푓 (푥) is lower semicomputable uniformly from the definition of 푋 and 푓 . Proof. For any rational number 푟 we must be able to recognize that the minimum is greater than 푟. For each 푟 the set {푥 : 푓 (푥) > 푟} is a lower semicomputable open set. It is the union of an enumerated sequence of canonical basis elements. If it covers 푋 this will be recognized in a finite number of steps.

162 B.1. Computable topology

It is known that a continuous function map compact sets into compact ones. This statement also has a constructive counterpart. Proposition B.1.32 Let 푋 be an eﬀectively compact space, and 푓 a computable function from 푋 into another constructive topological space 푌. Then 푓 (푋) is eﬀec- tively compact.

Proof. Let 훽푋 and 훽푌 be the enumerated bases of the space 푋 and 푌 respectively. Since 푓 is computable, there is a recursively enumerable set E ⊆ 훽푋 × 훽푌 such −1 Ð that 푓 (푉) = (푈,푉) ∈퐸 푈 holds for all 푉 ∈ 훽푌 . Let E푉 = {푈 : (푈, 푉) ∈ E}, then −1 Ð 푓 (푉) = E푉 . Consider some finite cover 푓 (푋) ⊆ 푉1 ∪ · · · ∪ 푉푘 with some 푉푖 ∈ 훽푌 . We Ð Ð will show that it will be recognized. Let U = 푖 E푉푖 , then 푋 ⊆ U, and the whole set U ⊆ 훽푋 is enumerable. By compactness, there is a finite number of Ð elements 푈1, . . . , 푈푛 ∈ U with 푋 ⊆ 푖 푈푖. By effective compactness, every one of these finite covers will be recognized. And any one such recognition will serve to recognize that 푉1, . . . , 푉푘 is a cover of 푓 (푋).

B.1.7 Computable metric space Following [7], we deﬁne a computable metric space as follows. Deﬁnition B.1.33 A constructive metric space is a tuple X = (푋, 푑, 퐷, 훼) where (푋, 푑) is a metric space, with a countable dense subset 퐷 and an enumeration 훼 of 퐷. It is assumed that the real function 푑(훼(푣), 훼(푤)) is computable. y As 푥 runs through elements of 퐷 and 푟 through positive rational numbers, we obtain the enumeration of a countable basis { 퐵(푥, 푟) : 푥 ∈ 퐷, 푟 ∈ Q} (of balls or radius 푟 and center 푥) of X, giving rise to a constructive topological space X˜.

Definition B.1.34 Let us call a sequence 푥1, 푥2,... a Cauchy sequence if for all −푖 푖 < 푗 we have 푑(푥푖, 푥 푗) ≤ 2 . To connect to the type-2 theory of computability developed above, the Cauchy-representation 훿X of the space is defined in a natural way. y ˜ It can be shown that as a representation of X, it is equivalent to 훾X˜ : 훿X ≡ 훾X˜ . Examples B.1.35 1. Example A.1.37 is a constructive metric space, with either of the two (equivalent) choices for an enumerated dense set. 2. Consider the metric space of Example A.1.28.6: the Cantor space (푋, 푑). Let ∗ 푠0 ∈ Σ be a distinguished element. For each finite sequence 푥 ∈ Σ let us

163 B. Constructivity

deﬁne the inﬁnite sequence 휉푥 = 푥푠0푠0 .... The elements 휉푥 form a (naturally enumerated) dense set in the space 푋, turning it into a constructive metric space. y Let us point out a property of metric spaces that we use frequently.

Definition B.1.36 For balls 퐵 푗 = 퐵(푥 푗, 푟푗), 푗 = 1, 2 we will say 퐵1 < 퐵2 if 푑(푥1, 푥2) < 푟2 − 푟1. In words, we will say that 퐵1 is manifestly included in 퐵2. This relation is useful since it is an easy way to see 퐵1 ⊂ 퐵2. y The property can be generalized to some constructive topological spaces. Definition B.1.37 A constructive topological space (푋, 훽, 휈) with basis 훽 has the manifest inclusion property if there is a relation 푏 < 푏0 among its basis elements with the following properties. a) 푏 < 푏0 implies 푏 ⊆ 푏0. b) The set of pairs 푏 < 푏0 is recursively enumerable. c) For every point 푥, and pair of basis elements 푏, 푏0 containing 푥 there is a basis element 푏00 containing 푥 with 푏00 < 푏, 푏0. We express this relation by saying that 푏 is manifestly included in 푏0. In such a Ñ space, a sequence 푏1 > 푏2 > . . . with 푖 푏푖 = {푥} is called a manifest representation of 푥. y Note that if the space has the manifest inclusion property then for every pair 푥 ∈ 푏 there is a manifest representation of 푥 beginning with 푏. A constructive metric space has the manifest inclusion property as a topological space, and Cauchy representations are manifest. Similarly to the definition of a computable sequence of computable functions in Subsection B.1.4, we can define the notion of a computable sequence of bounded computable functions, or the computable sequence 푓푖 of computable Lipschitz functions: the bound and the Lipschitz constant of 푓푖 are required to be computable from 푖. The following statement shows, in an effective form, that a function is lower semicomputable if and only if it is the supremum of a computable sequence of computable functions. Proposition B.1.38 Let X be a computable metric space. There is a computable mapping that to each name of a nonnegative lower semicomputable function 푓 assigns a name of a computable sequence of computable bounded Lipschitz functions 푓푖 whose supremum is 푓 .

164 B.1. Computable topology

Proof sketch. Show that 푓 is the supremum of a computable sequence of functions 푐푖1퐵(푢푖,푟푖) where 푢푖 ∈ 퐷 and 푐푖, 푟푖 > 0 are rational. Clearly, each indicator function 1퐵(푢푖,푟푖) is the supremum of a computable sequence of computable functions 푔푖, 푗. We have 푓 = sup푛 푓푛 where 푓푛 = max푖≤푛 푐푖푔푖,푛. It is easy to see that the bounds on the functions 푓푛 are computable from 푛 and that they all are in Lip for a 훽 that is computable from 푛. 훽푛 푛 The following is also worth noting. Proposition B.1.39 In a computable metric space, the intersection of two constructive open sets is constructive open. Proof. Let 훽 = { 퐵(푥, 푟) : 푥 ∈ 퐷, 푟 ∈ Q} be a basis of our space. For a pair (푥, 푟) with 푥 ∈ 퐷, 푟 ∈ Q, let Γ(푥, 푟) = {( 푦, 푠) : 푦 ∈ 퐷, 푠 ∈ Q, 푑(푥, 푦) + 푠 < 푟}.

If 푈 is a constructive open set, then there is a computably enumerable set 푆푈 ⊂ Ð 0 Ð 퐷 × Q with 푈 = (푥,푟) ∈푆 퐵(푥, 푟). Let 푆 = {Γ(푥, 푟) : (푥, 푟) ∈ 푆푈 }, then we Ð 푈 푈 have 푈 = ( ) ∈ 0 퐵(푥, 푟). Now, it is easy to see 푥,푟 푆푈 Ø 푈 ∩ 푉 = 퐵(푥, 푟). ( ) ∈ 0 ∩ 0 푥,푟 푆푈 푆푉 The following theorem is very useful. Theorem B.1.1 A computable metric space 푋 is eﬀectively compact if and only if from each (rational) 휀 one can compute a ﬁnite set of 휀-balls covering 푋.

Proof. Suppose first that the space is effectively compact. For each 휀, let 퐵1, 퐵2,... be a list of all canonical balls with radius 휀. This sequence covers the space, so already some 퐵1, . . . , 퐵푛 covers the space, and this will be recognized. Suppose now that for every rational 휀 one can find a finite set of 휀-balls covering the space. Let S ⊆ 훽 be a finite set of basis elements (balls) covering the space. For each element 퐺 = 퐵(푢, 푟) ∈ S, let 퐺휀 = 퐵(푢, 푟 − 휀) be its 휀- Ð Ð Ð interior, and S휀 = {퐺휀 : 퐺 ∈ S}. Then 퐺 = 휀 > 0퐺휀, and 푋 = 휀>0 S휀. Compactness implies that there is an 휀 > 0 such that already S푒푝푠 covers the space. Let 퐵1, . . . , 퐵푛 be a finite set of 휀/2-balls 퐵푖 = 퐵(푐푖, 푟푖) covering the space that can be computed from 휀. Each of these balls 퐵푖 intersects one of the the sets 퐺휀 = 퐵(푢, 푟 − 휀), 푑(푢, 푐푖) ≤ 푟 − 휀/2. But then 퐵푖 ⊆ 퐵(푢, 푟) in a recognizeable way. Once all the relations 푑(푢, 푐푖) ≤ 푟 − 휀/2 will be recognized we will also recognize that S covers the space.

165 B. Constructivity

We can strengthen now Example A.1.37: Example B.1.40 (Prove!) Let 푋 be an effectively compact computable metric space. Then the metric space 퐶(푋) with the dense set of functions E(퐷) introduced in Definition A.1.32 is a computable metric space. y The structure of a constructive metric space will be inherited on certain subsets: Definition B.1.41 Let X = (푋, d, 퐷, 훼) be a constructive metric space, and 퐺 ⊂ 푋 a constructive open subset. Then there is an enumeration 훼퐺 of the set 퐷 ∩ 퐺 that creates a constructive metric space G = (퐺, d, 퐷, 훼퐺). y Remark B.1.42 An arbitrary subset of a constructive metric space will inherit the constructive topology. It will also inherit the metric, but not necessarily the structure of a constructive metric space. Indeed, first of all it is not necessarily a complete metric space. It also does not inherit an enumerated dense subset. y

B.2 Constructive measure theory

The basic concepts and results of measure theory are recalled in Section A.2. For the theory of measures over metric spaces, see Subsection A.2.6. We introduce a certain ﬁxed, enumerated sequence of Lipschitz functions that will be used frequently. Let E be the set of functions introduced in Deﬁnition A.1.32. The following construction will prove useful later. Proposition B.2.1 All bounded continuous functions can be obtained as the limit of an increasing sequence of functions from the enumerated countable set E of bounded computable Lipschitz functions introduced in (A.1.3). The proof is routine.

B.2.1 Space of measures Let X = (푋, 푑, 퐷, 훼) be a computable metric space. In Subsection A.2.6, the space M(X) of measures over X is defined, along with a natural enumeration 휈 = 휈M for a subbasis 휎 = 휎M of the weak topology. This is a constructive topological space M which can be metrized by introducing, as in A.2.6, the Prokhorov distance 휌(휇, 휈). Recall that we defined 퐷M as the set of those probability measures that are concentrated on finitely many points of 퐷 and assign rational values to them. Let 훼M be a natural enumeration of 퐷M. Then

(M, 휌, 퐷M, 훼M) (B.2.1)

166 B.2. Constructive measure theory is a computable metric space whose constructive topology is equivalent to M. Let 푈 = 퐵(푥, 푟) be one of the balls in X, where 푥 ∈ 퐷X, 푟 ∈ Q. The function 휇 ↦→ 휇(푈) is typically not computable, since as mentioned in (A.2.6), it is not even continuous. The situation is better with 휇 ↦→ 휇 푓 . The following theorem is the computable strengthening of part of Proposition A.2.37: Proposition B.2.2 Let X = (푋, 푑, 퐷, 훼) be a computable metric space, and let M = (M(X), 휎, 휈) be the constructive topological space of probability measures over X. If the function 푓 : X → R is bounded and computable then 휇 ↦→ 휇 푓 is computable.

Proof sketch. To prove the theorem for bounded Lipschitz functions, we can in- voke the Strassen coupling theorem A.2.40. The function 푓 can be obtained as a limit of a computable monotone in- > creasing sequence of computable Lipschitz functions 푓푛 , and also as a limit of a computable monotone decreasing sequence of computable Lipschitz functions < > 푓푛 . In step 푛 of our computation of 휇 푓 , we can approximate 휇 푓푛 from above to < > < within 1/푛, and 휇 푓푛 from below to within 1/푛. Let these bounds be 푎푛 and 푎푛 . > < To approximate 휇 푓 to within 휀, find a stage 푛 with 푎푛 − 푎푛 + 2/푛 < 휀. Using Example B.1.40, we can extend this as follows: Proposition B.2.3 (Prove!) Let X be an effectively compact metric space (and thus 퐶(X) is a computable metric space). Then the mapping (휇, 푓 ) ↦→ 휇 푓 over M(X) × 퐶(X) is computable. Talking about open sets, only a weaker statement can be made. Proposition B.2.4 Let 퐺 ⊆ 푋 be a constructive open set. The function 휇 ↦→ 휇(퐺) is lower semicomputable. Remark B.2.5 Using the notion of an enumerative lattice defined by Hoyrup and Rojas, one can make this a statement about the two-argument function (휇, 퐺) → 휇(퐺). y It is known that if our metric space X is compact then so is the space M(X) of measures. This can be strengthened: Proposition B.2.6 (Prove!) If a complete computable metric space X is effectively compact then M(X) is also effectively compact.

167 B. Constructivity

B.2.2 Computable and semicomputable measures A measure 휇 is called computable if it is a computable element of the space of measures. Let {푔푖} be the set of bounded Lipschitz functions over X introduced in Definition A.1.32. Proposition B.2.7 Measure 휇 is computable if and only if so is the function 푖 ↦→ 휇푔푖. Proof. The “only if” part follows from Proposition B.2.2. For the “if” part, note that in order to trap 휇 within some Prokhorov neighborhood of size 휀, it is sufficient to compute 휇푔푖 within a small enough 훿, for all 푖 ≤ 푛 for a large enough 푛. A non-computable density function can lead to a computable measure: Example B.2.8 Let our probability space be the set R of real numbers with its standard topology. Let 푎 < 푏 be two computable real numbers. Let 휇 be the ( ) 1 ( ) probability distribution with density function 푓 푥 = 푏−푎 1[푎,푏] 푥 (the uniform distribution over the interval [푎, 푏]). Function 푓 (푥) is not computable, since it is not even continuous. However, the measure 휇 is computable: indeed, 휇푔푖 = ∫ 푏 1 푔 (푥)푑푥 is a computable sequence, hence Proposition B.2.7 implies that 푏−푎 푎 푖 휇 is computable. y The following theorem compensates somewhat for the fact mentioned earlier, that the function 휇 ↦→ 휇(푈) is generally not computable. Proposition B.2.9 Let 휇 be a finite computable measure. Then there is a computable map ℎ with the property that for every bounded computable function 푓 with | 푓 | ≤ 1 with the property 휇( 푓 −1 (0)) = 0, if 푤 is the name of 푓 then ℎ(푤) is the name of a program computing the value 휇{푥 : 푓 (푥) < 0}. Proof. Straightforward. Can we construct a measure just using the pattern of Proposition B.2.7? Sup- pose that there is a computable function (푖, 푗) ↦→ 푚푖 ( 푗) with the following properties: − 푗1 a) 푖 < 푗1 < 푗2 implies |푚푖 ( 푗1) − 푚푖 ( 푗2)| < 2 .

b) For all 푛, there is a probability measure 휇푛 with 푚푖 (푛) = 휇푛푔푖 for all 푖 < 푛.

Thus, the sequences converge, and for each 푛, all values 푚푖 (푛) for 푖 ≤ 푛 are consistent with coming from a probability measure 휈푛 assigning this value to 푔푖. Is there a probability measure 휇 with the property that for each 푖 we have lim 푗 푚푖 ( 푗) = 휇푔푖? Not necessarily, if the space is not compact.

168 B.2. Constructive measure theory

Example B.2.10 Let 푋 = {1, 2, 3,... } with the discrete topology. Define a probability measure 휇푛 with 휇푛푔푖 = 0 for 푖 < 푛 and otherwise arbitrarily. Since we only posed 푛 − 1 linear conditions on a finite number of variables, it is easy to see that such a 휇푛 exists. Then define 푚푖 (푛) = 휇푛 (푖) for all 푖. Now all the numbers 푚푖 (푛) converge with 푛 to 0, but 휇 = 0 is not a probability measure. y

To guarantee that the sequences 푚푖 ( 푗) indeed deﬁne a probability measure, progress must be made, for example, in terms of the narrowing of Prokhorov neighborhoods.

B.2.3 Random transitions Consider random transitions now. Definition B.2.11 (Computable kernel) Let X, Y be computable metric spaces, giving rise to measurable spaces with 휎-algebras A, B respectively. Let Λ = {휆푥 : 푥 ∈ 푋 } be a probability kernel from 푋 to 푌 (as defined in Subsection A.2.5). Let {푔푖} be the set of bounded Lipschitz functions over 푌 introduced in Defini- tion A.1.32. To each 푔푖, the kernel assigns a (bounded) measurable function

푦 푓푖 (푥) = (Λ푔푖)(푥) = 휆푥 푔푖 ( 푦).

We will call the kernel Λ computable if so is the assignment (푖, 푥) ↦→ 푓푖 (푥). y

When Λ is computable, each function 푓푖 (푥) is of course continuous. The ∗ ∗ measure Λ 휇 is determined by the values Λ 푔푖 = 휇(Λ푔푖), which are computable from (푖, 휇) and so the mapping 휇 ↦→ Λ∗휇 is computable. The following example is the simplest case, when the transition is actually deterministic. Example B.2.12 (Computable deterministic kernel) A computable function ℎ : 푋 → 푌 deﬁnes an operator Λℎ with Λℎ푔 = 푔 ◦ ℎ (as in Example A.2.29). This is a deterministic computable transition, in which 푓푖 (푥) = (Λℎ푔푖)(푥) = 푔푖 (ℎ(푥)) ( ) ∗ ∗ is, of course, computable from 푖, 푥 . We deﬁne ℎ 휇 = Λℎ휇. y

169

Bibliography

[1] Yevgeniy A. Asarin. Individual Random Signals: a Complexity Approach. PhD thesis, Moscow State University, Moscow, Russia, 1988. In Russian. 4.1.1 [2] Ya. M. Barzdin’. The complexity of programs to determine whether natural numbers not greater than 푛 belong to a recursively enumerable set. Soviet Math. Doklady, 9:1251–54, 1968. 3.2.2 [3] Ya. M. Barzdin’ and R. Freivald. On the prediction of general recursive functions. Soviet Math. Doklady, 206:1224–28, 1972. 1.1.2 [4] Charles H. Bennett and Martin Gardner. The random number Omega bids fair to hold mysteries of the universe. Scientiﬁc American, 242(5):20–34, Nov 1979. 1.1.3 [5] Patrick Billingsley. Convergence of Probability Measures. Wiley, 1968. First edition. Second edition 1999. A.1.5, A.1.40, A.2.6 [6] Manuel Blum and Silvio Micali. How to generate cryptographically strong sequences of pseudo-random bits. SIAM Journal on Computing, 13:850– 864, 1984. 1.1.2 [7] Vasco Brattka and Gero Presser. Computability on subsets of metric spaces. Theoretical Computer Science, 305:43–76, 2003. B.1.7 [8] Gregory J. Chaitin. Information-theoretical limitations of formal systems. Journal of the ACM, 21:403–424, 1974. 1.1.2, 1.1.3 [9] Gregory J. Chaitin. Scientiﬁc American, 232(5):47–52, May 1975. 1.1.2, 1.1.3 [10] Gregory J. Chaitin. A theory of program size formally identical to information theory. J. Assoc. Comput. Mach., 22:329–340, 1975. 1.1.3, 1.6.5

171 Bibliography

[11] Gregory J. Chaitin. Algorithmic entropy of sets. Computers and Mathem. with Applications, 2:233–245, 1976. 1.1.3 [12] Gregory J. Chaitin. Algorithmic information theory. IBM J. Res.& Dev., 21:350–359, 1977. 1.1.3 [13] Imre Csiszár and János Körner. Information Theory. Academic Press, New York, 1981. 1.6.13, 1.6.15, 3.1.1 [14] Robert Daley. Math. Syst. Th., 9(1):83–94, 1975. 1.1.2 [15] Pierre Simon de Laplace. A Philosophical Essay on Probabilities. Dover, 1951. 1.1, 1.1.3 [16] Terrence Fine. Theories of Probability. Academic Press, New York, 1973. 1.1.3 [17] Peter Gács. On the symmetry of algorithmic information. Soviet Math. Dokl., 15:1477–1780, 1974. Translation of the Russian version. 1.1.3, 1.6.5, 3.1.5, 3.1.1 [18] Peter Gács. Exact expressions for some randomness tests. Z. Math. Log. Grdl. M., 26:385–394, 1980. There is an earlier, shorter conference version. 1.1.3, 2.3.6 [19] Peter Gács. On the relation between descriptional complexity and algorithmic probability. Theoretical Computer Science, 22:71–93, 1983. Short version: Proc. 22nd IEEE FOCS (1981) 296-303. 1.1.3 [20] Peter Gács. Randomness and probability–complexity of description. In Samuel Kotz and Norman L. Johnson, editors, Encyclopedia of Statistical Sciences, Vol. 7, pages 551–556. Wiley, New York, 1986. 1.1.3 [21] Peter Gács. Uniform test of algorithmic randomness over a general space. Theoretical Computer Science, 341(1-3):91–137, 2005. 4.6.4 [22] Peter Gács and János Körner. Common information is far less than mutual information. Problems of Control and Inf. Th., 2:149–162, 1973. 3.1.1 [23] Peter Gács, John Tromp, and Paul M. B. Vitányi. Algorithmic statistics. IEEE Transactions on Information Theory, 47:2443–2463, 2001. arXiv:math/0006233 [math.PR]. Short version with similar title in Algo- rithmic Learning Theory, LNCS 1968/2000. 3.1, 3.1.2

172 Bibliography

[24] Oded Goldreich, Shaﬁ Goldwasser, and Silvio Micali. How to construct random functions. Journal of the Association for Computing Machinery, 33(4):792–807, 1986. 1.1.2 [25] Peter Hertling and Klaus Weihrauch. Randomness spaces. In Proc. of ICALP’98, volume 1443 of Lecture Notes in Computer Science, pages 796– 807. Springer, 1998. 4.1.1, 4.1.4 [26] Mathieu Hoyrup and Cristóbal Rojas. Computability of probability measures and Martin-Löf randomness over metric spaces. Information and Computation, 207(7):830–847, 2009. 4.1.2 [27] A.N. Kolmogorov. Three approaches to the quantitative deﬁnition of information. Problems of Inform. Transmission, 1(1):1–7, 1965. 1.1, 1.1.3 [28] Andrei N. Kolmogorov. Foundations of the Theory of Probability. Chelsea, New York, 1956. 1.1 [29] Andrei N. Kolmogorov. A logical basis for information theory and probability theory. IEEE Transactions on Information Theory, IT-14:662–664, 1968. 1.1, 1.1.3 [30] Leonid A. Levin. On the notion of a random sequence. Soviet Math. Dokl., 14(5):1413–1416, 1973. 1.1.3, 4.1.1, 4.1.1 [31] Leonid A. Levin. Laws of information conservation (nongrowth) and as- pects of the foundations of probability theory. Problems of Inform. Transm., 10(3):206–210, 1974. 1.1.2, 1.1.3, 1.6.5, 3.1.2 [32] Leonid A. Levin. Uniform tests of randomness. Soviet Math. Dokl., 17(2):337–340, 1976. 4.1.1,2, 4.3.5 [33] Leonid A. Levin. Randomness conservation inequalities: Information and independence in mathematical theories. Information and Control, 61(1):15–37, 1984. 1.1.2, 1.1.3, 2.2.3, 3.1.11, 3.1.2, 4.1.1,1,2, 4.3.5 [34] Leonid A. Levin. One-way functions and pseudorandom generators. Com- binatorica, 4, 1987. 1.1.2 [35] Leonid A. Levin and V.V. V’yugin. Invariant properties of information bulks. In Proceedings of the Math. Found. of Comp. Sci. Conf., volume 53 of Lecture Notes on Comp.Sci., pages 359–364. Springer, 1977. 1.1.3 [36] Ming Li and Paul M. B. Vitányi. Introduction to Kolmogorov Complexity and its Applications (Third edition). Springer Verlag, New York, 2008. 1.1.3, 2.2.1

173 Bibliography

[37] Donald W. Loveland. A variant of the Kolmogorov concept of complexity. Information and Control, 12:510, 1969. 1.1.3, 3.2 [38] Per Martin-Löf. The definition of random sequences. Information and Con- trol, 9:602–619, 1966. 1.1.3, 4.1.1, 4.1.1, 4.2.3 [39] Wofgang J. Paul, Joel Seiferas, and Janos Simon. An information- theoretical approach to time-bounds for on-line computation. J. Computer and System Sciences, 23:108–126, 1981. 1.1.2 [40] David Pollard. A User’s Guide to Measure-Theoretic Probability. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, U.K., 2001. A.2, A.2.5, A.2.6 [41] Claus Peter Schnorr. Zufälligkeit und Wahrscheinlichkeit. In A. Dold and B. Eckmann, editors, Lecture Notes in Mathematics 218, pages 1–213. Springer Verlag, New York, 1971. 1.1.3 [42] Claus Peter Schnorr. Process complexity and effective random tests. J. Comput. Syst. Sci, 7(4):376–388, 1973. Conference version: STOC 1972, pp. 168-176. 1.1.3 [43] Claus Peter Schnorr. A review of the theory of random sequences. In Proc.5-th Int.Congr. of Logic, Meth. and Phil. of Sci., London, Ontario, 1975. 1.1.3 [44] Claus Peter Schnorr. General random sequences and learnable sequences. J. Symb. Logic, 42:329–340, 1977. 1.1.3 [45] Raymond J. Solomonoff. A formal theory of inductive inference I. Informa- tion and Control, 7:1–22, 1964. 1.1.3 [46] Raymond J. Solomonoff. Complexity-based induction systems: Compar- isons and convergence theorems. IEEE Transactions on Information Theory, IT-24(4):422–432, July 1978. 1.1.2 [47] Robert Solovay. Unpublished manuscript. 1976. 1.1.3, 1.7.2, 1.7.4 [48] Robert Solovay. On random r.e. sets. In A. I. Arruda et al., editor, Non-Clas- sical Logic, Model Th. and Computability, pages 283–307. North Holland, 1977. 1.1.3 [49] Tom Stoppard. Rosencrantz and Guildenstern are Dead. A play. 1.1 [50] Volker Strassen. The existence of probability measures with given marginals. Annals of Mathematical Statistics, 36:423–439, 1965. A.2.40

174 Bibliography

[51] Cedric Villani. Optimal Transport, Old and New. Springer, New York, 2008. A.2.44 [52] J. Ville. Etude Critique de la Notion de Collectif. Gauthier-Villars, Paris, 1939. 1.1.3 [53] Richard von Mises and H. Geiringer. The Mathematical Theory of Probability and Statistics. Academic Press, New York, 1964. 1.1, 1.1.3 [54] Vladimir G. Vovk and V. V. Vyugin. On the empirical validity of the Bayesian method. Journal of the Royal Statistical Society B, 55(1):253–266, 1993. 4.5.3 [55] V. V. V’yugin. On Turing-invariant sets. Soviet Math. Doklady, 229(4):1090–1094, 1976. 1.1.3 [56] V. V. V’yugin. Ergodic theorems for individual random sequences. Theo- retical Computer Science, 207(2):343–361, 1998. 4.2.2 [57] Klaus Weihrauch. Computable Analysis. Springer, 2000. 4.1.1, B.1.1, B.1.1 [58] D. G. Willis. Computational complexity and probability constructions. J. ACM, pages 241–259, April 1970. 1.1.3 [59] Andrew C. Yao. Theory and applications of trapdoor functions. In IEEE Symp. on Foundations of Computer Science, pages 80–91, 1982. 1.1.2 [60] Ann Yasuhara. Recursive Function Theory and Logic. Academic Press, New York, 1971. 1.1.1 [61] Alexander K. Zvonkin and Leonid A. Levin. The complexity of ﬁnite objects and the development of the concepts of information and randomness by means of the theory of algorithms. Russian Math. Surveys, 25(6):83–124, 1970. 1.1.2, 1.1.3, 3.2

175