Algorithmic Information Theory The Basics



Adam Elga

IAP January

Preliminaries

Turing machines

Turing machine An idealized computing device attached to a tape each

square of which is capable of holding a symbol We write a program

p a nite binary string on the tap e and start the machine If the

machine halts with string o written at a designated place on the tap e

then o is the output

Universal Turing machine A Turing machine capable of simulating any

other Turing machine in the following sense Given an input that

enco des the command Simulate machine k on program p a universal

machine will output whatever machine k would output if given program

p

Denition of complexity

The complexity of string s relative to machine T is dened to b e the length

of the shortest program that gets T to pro duce s as output

We choose some universal Turing machine U and dene the complexity of

string s to b e the complexity of s relative to U

How much do es the choice of universal machine matter



adamphilosophersnet

Lots For any string s there is a universal machine U such that the com

s

plexity of s relative to U is zero

s

0

Not much Supp ose that U and U are b oth universal machines Then there

is a constant k such that for any s the complexity of s relative to U

0

never diers from the complexity of s relative to U by more than k So

0

for increasingly long strings the dierence b etween U and U b ecomes

less and less signicant

Upshot this complexity measure allows us to b o otstrap a single qualitative

simplicity judgment into many quantitative simplicity judgments

Basic facts

Strings never have complexities much greater than their lengths There

is a constant k such that no strings complexity is more than k greater

than its length

Lowcomplexity strings are relatively rare Example of the strings of

length only a tiny fraction have complexity less than

A minimal program has a complexity approximately equal to its length

The complexity function C
is not computable

Hint on how to prove this Berry paradox

D The smallest number not describable in fewer than twenty syllables

D describ es some number since there are only nitely many under

twentysyllable descriptions

But supp ose that D refers to n Then n is describable in nineteen

syllables so D do esnt refer to n Contradiction

Randomness

Two approaches to the question of whether a string is random

Consider what sort of pro cess produced the string Call the string

processrandom if it was pro duced by the right sort of chancy pro cess

Pay no attention to what pro duced the string Instead call the string

productrandom if it is appropriately unpatterned

Well fo cus on the second approach as applied to innite binary strings

Think of a minimallength program for a nite string as a compressed version

of the string The compressibility of a nite string s is dened to b e the length

of s minus the length of a minimallength program for s

Intuition an innite string that is pro ductrandom ought to have initial seg

ments that arent very compressible That motivates the following denition

An innite string is productrandom if and only if there is an upp er b ound

B such that no initial segment of the string has compressibility greater than

B

Tweak require programs to b e selfdelimiting

In the long run pro ductrandom sequences have just as many 1s as 0s Also

pro ductrandom sequences cannot b e exploited by gambling machines

Incompleteness

The language of arithmetic contains logical symbols expressions that denote

numbers and the addition and multiplication signs

In the language of arithmetic one can make numerical assertions such as

 For all x for all y x y y x

 There exists an x such that for all y x  y y  y

Supp ose that weve got some axioms that are such that some Turing machine

prints the axioms out in order

Supp ose that weve got some rules of inference for deriving consequences

of the axioms rules such that there is a machine M that prints out every

sentence derivable from the axioms using the rules and prints no other sen

tences

Question Can it b e that our axioms and rules satisfy b oth of the following

conditions

Every sentence derivable from the axioms on the basis of the rules is

true In other words every sentence that M prints is true

Every true arithmetical sentence is derivable from the axioms on the

basis of the rules In other words M eventually prints every true

arithmetical sentence

References

Gregory Chaitin Algorithmic information theory Cambridge Uni

versity Press This b o ok develops algorithmic prex complex

ity emphasizing its analogies with the entropy of classical information

theory Chapter investigates innite random sequences Available at

httpwwwcsaucklandacnzCDMTCSchaitincuppdf

Ming Li and Paul Vitanyi An introduction to Kolmogorov complexity and

its applications Springer A comprehensive textb o ok This b o ok

has everything including extensive notes on the history of the various

ideas

Michiel van Lambalgen Von mises denition of random sequences recon

sidered Journal of Symbolic Logic A technical article

comparing various criteria for randomness

Michiel van Lambalgen Algorithmic information theory Journal of Sym

bolic Logic Argues against some of Chaitins

claims ab out the relationship b etween algorithmic information theory and

Go dels incompleteness theorems If you read anything by Chaitin you

should also read this article