The Limits of Computation

1 The Limits of Computation

The next time a bug freezes your computer, you can take solace in the deep mathematical truth that there is no way to eliminate all bugs -Seth Lloyd

The 1999 science ﬁction movie, ‘The Matrix’, explored this idea. In this world, humans are plugged into a virtual reality, referred to as ‘The Matrix’. They go about their daily lives, unaware that the sensory inputs that they receive do not originate from their per- ceived reality. When a person, Alice, within the matrix observes a watermelon falling from a skyscraper, there is no skyscraper, nor watermelon, nor even gravity responsi- ble for the watermelon’s fall. Instead a complex computer program works silently in the background. The initial state of the watermelon, the location of the observer, are encoded in a sequence of bits. The computer takes these bits, processes according to a predetermined algorithm, and outputs the electrical signals that dictate what the observer should see.

To people in the twenty ﬁrst century, whose lives are enmeshed in various information processors, the eventual plausibility of the Matrix does not appear as radical as it once did. One by one, the photos we watch, and the mail we send, have been converted to digital form. Common questions such as “How many megabytes does that song take up?” reﬂect a society that is becoming increasingly accepting of the possibility that the observable qualities of every object can be represented by bits, and physical processes by how they manipulate these bits. While most people regard this as a representa- tion of reality, some scientists have even gone as far to speculate that the universe is perhaps just a giant information processor, programmed with the laws of physics we know [Llo06].

Why should a humble computer, that operates on the laws of electromagnetism, ever be capable of simulating all observable effects of gravity? How is it that we may use them to predict the ﬂuctuations of gravitational waves or the reliability of a Boeing

1 MILE GU

747, when it neither detects gravity waves nor ﬂies? A priori, there exists no reason why there should exist universal systems, computers capable of mimicking the output of any feasible physical experiment in the known universe short of the entire universe itself.

The synonymity of universal systems and universal computers highlights the close links between the theories of computation, and that of physical reality. All practical computational devices must ultimately be implemented by some physical law, and all physical laws discovered so far may be simulated by a standard computer1. Not only does this suggest that universal devices allow us to simulate and thus better understand reality, but also that discoveries made in the science of computation could naturally lead to insights about physical reality, and vice versa.

The goal of this course is explore introduce this synergy. We consider three rapidly developing scientific fields linked to this central theme. The first demonstrates that understanding the limits and capabilities of computational devices can lead to further insight about our universe. The second motivates further development in theoretical computer science by consideration of how the difficulty of computational problems fundamentally depends on the laws of physics. The final topic revisits the Second Law of Thermodynamics from the perspective of information theory, giving profound demonstration of the idiom, ’Knowledge is Power.’

In this chapter, we begin this journey with a discussion on the ultimate limits of computation, and show how this can lead us to new understanding about universal principles of our reality. Section 1.1 launches the chapter in highlighting the connections between theorem’s in computer science and and understanding of reality. Section 1.2 introduces variations of the Church-Turing thesis [Tur36], a universal principle that roughly speaking, postulates that the computers we used today are such universal players. Section 1.3 describes the limitations of these universal computers exhibit, and 1.4 constructs formal models of universal computers that recasts these limitations to more concrete settings. In the next chapter, we combine these results with a proof that the ability to derive macroscopic laws from microscopic laws would neces- sitate the ability to do something that no universal computer can do, in violation of the Church-Turing thesis.

1.1 Emergent Laws, Universal Principles and Non-computability

Much of fundamental physics has been motivated by a search for a ‘Theory of Every- thing’, a set of principles that govern all known dynamics of the universe. This ‘holy

1All currently known models of quantum computation can also be simulated by classical computers, albeit with exponential overhead. There does, however, exist a minority of scientists who believe there may exists undiscovered physical laws that cannot be simulated. For example, see [Pen89].

2 PHYSICS OF INFORMATION LECTURE NOTES grail’ has motivated the construction of particle accelerators and many of the more es- oteric ﬁelds of theoretical physics. The standard reductionist rationale is that since all physical systems are built from conglomerations of fundamental particles, the principles governing these such particles will also apply to all such systems.

The plausible existence of universal systems suggests a complementary approach to the understanding of universal principles. If all observable qualities of any physical process may be simulated by a single system, then the limitations on that system will also be universal limitations that allow us to make generic statements about what we can observe within our universe.

Indeed, there exists many tasks that universal computers cannot perform. The canon- ical example is the Halting problem [Tur36], which asks a computer to decide whether a given computer algorithm will eventually output some number, or be trapped in an inﬁnite loop. A systematic solution of this problem would be of intense interest not only to software engineers, but also to any PhD student tired of spontaneous computer crashes when writing their thesis. The Halting problem is far from an isolated case, there exist many interesting problems that have no systematic solution. This immediately leads to a somewhat depressing universal principle (See Box 1) .

”The universe allows no systematic method to eliminate all bugs.”

Philosophically, such principles may appear puzzling from a reductionist perspective. While most existing computers are electronic, such devices can in principle be constructed from almost any physical medium. Numerous physical systems of surpris- ing simplicity, including even colliding billiard balls, have demonstrated universality [FT82]. The rather fundamental statement about arbitrary macroscopic systems is thus independent of microscopic composition. This motivates the idea of emergence; principles that govern macroscopic systems are not entirely predictable by the laws that govern their microscopic constituents [And72].

When we look at the physical universe around us, we often observe some sort of ‘macroscopic order’. When we analyze the ﬂow of water, or the dynamics of a glacier, we do not need to compute the exact motion of every atom. The trick here is that when we observe the macroscopic world, we generally neglect the microscopic details. We see and measure macroscopic features, such as pressure, density and stress. The earli- est scientists like Galileo and Archimedes observed relations between such quantities, without a care for the existence of atoms or quantum mechanics.

When scientists write down equations relating such quantities, one should note that the quantities themselves implicitly assume a continuity limit. Pressure gradient, stress etc., are only formally defined in the limit where the medium is continuous, i.e., where it contains an infinite number of infinitesimal particles.

3 MILE GU

Box 1 (Universal Principles) :

”The law that entropy always increases, holds, I think, the supreme position among the laws of Nature. If someone points out to you that your pet theory of the universe is in disagreement with Maxwell’s equations then so much the worse for Maxwell’s equations. If it is found to be contradicted by observation well, these experimentalists do bungle things sometimes. But if your theory is found to be against the second law of thermodynamics I can give you no hope; there is nothing for it but to collapse in deepest humiliation.” Sir Arthur Stanley Eddington, The Nature of the Physical World (1927)

Any primary school student could tell you that should you drop a watermelon from the Eiffel Tower, it will plummet to the ground. Any claims otherwise would be met with disbe- lief. Yet, the kid probably never went to the Eiffel tower, and almost certainly was not there when a watermelon plummeted to its fruity demise. Without ﬁrst hand experience, how is it that a child could rule out the inﬁnite other possibilities... a watermelon hovering in midair, or spontaneous turning into a banana, perhaps?

A key component of predicting natural phenomena is the knowledge of underlying rules of thumb that you expect very general classes of objects to obey. In this case, the child knows that heavier than air objects fall. This rule allows a child to rule out limitless number of scenarios, and hence have a good idea of what happens in scenarios that he or she has never observed. When these rules of thumb are thought to work for any physical system, they are regarded as universal principles.

The truth of universal principles cannot be proven. There is nothing that prevents some exotic phenomena to be discovered in future, that demonstrably violates a given principle. We believe in the conservation of energy, because all observations and experiments so far indicate that it is true.

Such universal principles not only give insight on how the universe behaves, but also help rule out proposals that carelessly violate them without experimental evidence or good reason. Should an overexcited undergrad propose a machine capable of generating energy from nothing, one’s skepticism is justiﬁed by the law of conservation of energy. Similarly, the second law of thermodynamics allows one to rule out the countless proposals of perpetual motion machines that invariably surface each year [Llo06].

4 PHYSICS OF INFORMATION LECTURE NOTES

We call these equations macroscopic laws. A reductionist view is that all macroscopic laws are logically implied by microscopic laws. Since a metal bar is made out of atoms, then the microscopic laws governing the dynamics of these atoms would allow you to systematically derive the laws that govern how the bar behaves under stress. Here ‘systematically derive’ implies formal, mathematical implication. That is, feed the microscopic laws that governed each atom and their interactions into a computer, along with formal deﬁnitions of the macroscopic observables, and it would eventually be able to output any law that governs those macroscopic observables. The derivation of macroscopic laws requires no additional ‘insight’ or ‘creativity’.

Here, we will see that this connection between the Halting problem and emergence is not unjustiﬁed. The universal no-go theorems of computer science each give us universal principles that govern all of physical reality. The trick then, is to understand how computation relates with physics.

1.2 The Road to Universality

The ﬁrst step of our strategy is to formally understand the Church-Turing thesis, and gain intuition on why it is generally regarded to be quite possibly true. A priori, it is not obvious why this should be the case. The universe is ﬁlled with hundreds of different fundamental particles, and it seems quite plausible that to replicate your observations on each of these systems, you’re going to need all those particles with you. Why should a select few of such particles be capable of simulating any experiment you can could perform on the rest?

Surprisingly, observations so far suggests that the universe could actually operate this way. The existence of computers we operate each day lends credible evidence that universal computers can be constructed without appealing to all possible physical interactions. In fact, we have become so used to the idea of universality that we exploit it daily without second thought. Given a piece of software designed for PC, we know that in principle, a MAC version for it can be written.

Whether a PC or a Mac, computers operate by manipulating information, stored in the form of strings of bits 2. Programs on either platform operate identically provided when fed coinciding input, they give identical strings of bits as output. It matters not how or on what physical medium such bits are encoded. In physics lingo, any experiment you can perform on a MAC can be replicated on a PC, and vice versa.

2A bit is any physical system with at least two different conﬁgurations, two of which are selected and denoted 0 and 1.

5 MILE GU

1.2.1 Experiments as Computation

Computation and physics, however, are not at all that different. When we perform an experiment in the physical world, we are essentially programming. When we pub- lish the design of a new experiment, we do not need to send a perfect replica of the experimental set-up to every lab. Instead, a schematic of the experiment would be distributed, represented by a ﬁnite sequence of bits 3. It matters not whether this sequence of bits was stored in a magnetic tape, a CD, or a solid state device. All information about the experiment is encoded within the bits, regardless of the physical medium used to store those bits.

Similarly, any observations on the results of the given experiment is nothing more than another ﬁnite string of bits 4, regardless of the nature of the underlying physical system 5. Any experiment can be thought of as a computation. The initial conﬁguration of the system dictates the input; the physical laws operating on the system dictates the program; and anything that we choose to observe the output. As David Deutsch, one of the pioneers of quantum computing, points out [Deu85]:

”The reason why we ﬁnd it possible to construct, say, electronic calculators, and indeed why we can perform mental arithmetic, cannot be found in mathematics or logic. The reason is that the laws of physics “happen” to permit the existence of physical models for the operations of arithmetic such as addition, subtraction and multiplication.”

Every physical experiment is a computation, and every computation is facilitated by some physical experiment. The class of all possible experiments deﬁne the class of Computable functions all computable functions. A universal computer is a system capable of computing all functions that can be computed using any physically realizable system.

Consider evaluating the action of a binary function f on an input x. Regardless of which programming language or method we use, we must rely on the dynamics of some underlying physical system S. Prior to computation, the definition of a function, together with input of interest x, are encoded within the initial configuration of S. We allow the system to evolve according to a specific physical process, such that after some specific time, a particular observable of the system encodes the desired output Physically Reasonable f(x). The computation is completed by measurement of this observable. Computation This motivates us to only consider models of computation that are physically reasonable. Any algorithm specified within such a model should admit a physical implemen-

3The sequence is always guaranteed to be finite, even if a given experiment involves parameters that vary over continuous quantities. This is because it is impossible to set-up anything to perfect precision. 4Again, the finite precision of measurement ensures that the meaningful output can be encoded in a finite number of bits. 5Even in cases where the underlying system is quantum, and wavefunction evolves continuously, the final measurement result is still encoded within a string of bits.

6 PHYSICS OF INFORMATION LECTURE NOTES tation that is experimentally possible, at least in principle. One postulate on the set of computations we would consider to be physically reasonable are:

1. Only a ﬁnite subsystem of the entire computer is ‘in motion’ at any particular time.

2. The motion depends only on the state of a ﬁnite subsystem.

3. The amount of information used to to specify this motion is ﬁnite.

This is not a mathematical definition, and is entirely dictated by our current understanding of what operations can be physically realized within the universe. Currently, for, example, we expect a finitely realizable physical process to manipulate only a bounded amount of information, based on the intuition that it is impossible to measure a real number to infinite precision.

If any physical process were ever discovered to allow such an operation, then the set of models we take to be physically reasonable will necessarily need to be revised. There already exists many exotic models of computation, and a brief survey is available in Box 2.

1.2.2 Universal Computers

We deﬁne a system to be a universal computer if it is capable of disguising itself com- pletely as physical reality. In short, the system is capable of replicating the effects of the matrix, such that a person (who we refer to as Neo) is incapable of determining whether he is interacting with the real world, or that of a simulation provided by this system.

We assume that Neo is armed with all the knowledge of how the real world actually behaves. To determine whether he is living in reality or the Matrix, Neo is capable of performing any physically reasonable experiment. He can set up any conﬁgura- tion of physical objects, and measure any of their resulting outputs. Thus, a universal computer must be capable of computing any function that may be computed on a reasonable model of computation.

Such systems not only exists, but can be remarkably simple. In 1936, an English math- ematician Alan Turing was exploring the Entscheidungsproblem, a challenge by David Hilbert that asked whether there exists an algorithm that can decide whether any given formal mathematical proposition was either true or false. To formalize the notion of algorithm, Alan Turing proposed the Turing machine [Tur36], a simple device which he claimed to be capable of performing any algorithmic process (See Box 3).

7 MILE GU

Box 2 (Models of Hypercomputation) :

The ﬁeld of hypercomputation is devoted to the study of computational devices cannot be realized by physical processes that we currently know [Pot06]. A quick survey includes

Zeno Machines: Named after the Greek philosopher Zeno for presenting a paradox where no one ever catches up to anybody else (Zeno’s Paradox). A Zeno machine is capable of making each successive computation twice as fast as the previous one [Wey49]. For example, at each fixed timestep, each Zeno machine would create a clone of itself. This allows it to potentially perform an infinite number of tasks in a finite amount of time.

For example, if Moore’s Law, where the speed of computers doubled every 18 months, were to precede indeﬁnitely, the system of a person who upgrades his computer every two years would be a Zeno machine in spirit.

Real Computers: Real computers are idealistic analogue computers whose systems can store and output real numbers [BCSS98]. Thus, each finitely realizable physical process effectively operates on an infinite number of bits. The infeasibility of this model is due to the observation that there exists no known method to measure any continuous variable quantity to infinite precision.

Infinite Time Computers: These are computers that are physically realizable, with the ex- ception that they are allowed an infinite amount of time to compute the output [Pot06]. Possible proposals of such computers have involved the use of time dilation, where a person accelerates himself to the speed of light while he leaves his humble desktop running. Unfortunately, such proposals involve infinite energy, and do not appear very reasonable outside of Star Trek.

While the study of such unrealistic models of computation may seem esoterical, they yield insight on what computers potentially more powerful than any realizable in our universe can do. There are also a minority of scientists who believe that such computers already exist, in for example, human consciousness [Pen89].

But, what is an algorithmic process? From an operational perspective, an algorithm is a set of instructions that a person or indeed, a sufﬁcient sophisticated machine, can follow, without additional insight. Thus, such a process can be regarded as a physically realizable experiment, and corresponds to some reasonable model of computation. This leads to the Church-Turing thesis:

Church-Turing Thesis: Any function that can be computed by a reasonable model of computation can be computed by an universal Turing machine.

The Church-Turing thesis is not a theorem, but rather a universal principle whose va- lidity is based on observation. Since its original inception over eight decades ago, no physical processes that can compute a larger class of functions than Turing machines have been discovered. Every reasonable model of computation that seemed more powerful, such as Turing machines with 2-dimensional tapes, multiple robots, or high level programming languages with conditional statements, can all be simulated by the humble implementation above.

8 PHYSICS OF INFORMATION LECTURE NOTES

Box 3 (Turing Machines) :

While the technical description of Turing machines suffers from cumbersome notation, the concept behind its mechanics are intuitive. Imagine a one dimensional world that consists of an inﬁnite sequence of square cells connected together to form a tape. Each cell may be of k different colors. In practice, three colors is sufﬁcient.

Living in this world is a single robot, who is always in one of n internal states, for example the happy, sad and halting states. At each time step, the behavior of the robot is entirely characterized by his internal state, si, and the color of the square he is on (ki). If the robot is in the halting state, he does nothing, and the system reaches steady state. Otherwise the robot will (a) paint the square he resides in one of the k possible colors, (b) move either up or down the tape and (c) change to one of the n possible internal states, based on his two input states.

Formally, let Σ be the set of possible colors, and Q be the set of internal states of the robot, then the behavior of the robot is entirely deﬁned by a mapping f :Σ × Q → {0, 1} × Σ × Q, where {0, 1} is used to represent which way the robot moves. Mathematically, this mapping is represented by a look-up table. Each row of the look-up table consists of an order pair of pair separate entries: (si, ki, sf , kf , b) (1.1)

Where si, ki are the states of the robot and the color of the tape the robot is currently on; and sf , kf are what these states change to after application of f. The binary bit, b ∈ {0, 1} represents the direction of the robot will move. To execute f, we can algorithmically follows the look-up table, and executes the line that matches his current state, and the color of the tape he sees.

A Turing machine is defined as the collective system that consists of the robot and the tape. Any specific Turing machine is then entirely specified by:

1. The set of internal states of the robot, Q. 2. The set of colors on the tape σ 3. The behavior of the robot, f.

A given Turing machine computes by taking as input a tape, such that all but a finite number of cells are of some default color, say ‘white’. The robot then modifies the tape according to the prescribed behavior till it reaches the halting state, whereby the set of non-white cells on the modified tape encodes the output.

The Church-Turing thesis implies that for any physically computable function, we may define a Turing Machine that computes this desired function. In fact, there exists specific constructions of Turing machines, universal Turing machines, that may simulate all other Turing machines and hence any possible computation. In [Wol02], Wolfram proposes one such construction, consisting of a robot with just two states and a tape with only five colors.

9 MILE GU

1.3 The Limits of Universal Systems

If a Turing machine is capable of emulating the computational capabilities of an arbitrary physical system, are there any computational tasks that it cannot do? When Alan Turing conjectured such machines, this was a question he was attempting to answer. Since any algorithm may be implemented by a Turing machine, any decision problem that no Turing machine can solve would not be solvable by any reasonable physical process (assuming the truth of the Church-Turing thesis). In this section, we explore such non-computable problems.

Box 4 (Turing machines as Pseudo-code) :

To make our lives easier, we won’t be specifying Turing machines by the behaviorial function f of their corresponding robots. Instead, each Turing machine may be deﬁned by the pseudo-code that describes the algorithm it implements, with the understanding that each basic operation (loops, addition, condition statements, etc) within the code may be implemented by a Turing machine [NC00]. For example

if n mod 2 = 0 return true; else return false;

represents a Turing machine that tests whether a given input n is even.

1.3.1 The Halting Problem

The Halting problem is a classical example of a non-computable problem [Tur36]. No Turing machine can take an arbitrary Turing machine T and some number x as input, and output with certainty whether or not T will halt on input x. There exists no algorithm, and hence no physical process, that would allow us to determine whether a given algorithm will ever end or be trapped in an inﬁnite loop. In relation to our previous analogy, there is no way to predict whether a given robot will toil away at his trip for all eternity, or come to halt in a ﬁnite amount of time.

Of course, there will exist many cases where this is possible; a Turing machine with a single line of code Return 4 will always halt and output 4 by simple inspection6. Rather, this result tells us that no algorithm can do so for arbitrary inputs. The only sure way to find out whether a general Turing machine will halt is to run it for all eternity; no physically realizable experiment in a finite timeframe is sufficient.

There exists no Turing machine that can decide whether or not a general Turing Machine will halt in ﬁnite time.

The proof of this statement is done through contradiction. The existence of Turing machine the computes the halting problem, H, allows us to construct a second Turing

6We represent all Turing machines by pesudo-code for convenience. See Box 4.

10 PHYSICS OF INFORMATION LECTURE NOTES

Figure 1.1: The non-computability of the Halting problem and the truth of Rice’s theorem can both be proved through the construction of hypothetical Turing machines. The ability to decide whether any given Turing machine halts implies the existence of a Turing machine T that implements (A), which leads to a contradiction when one considers the value of T (T ). In (B), the Turing machine Q would allow us to decide whether X halts provided a given black box property P is decidable.

machine T (See Fig. 1.1.A). Consider the action of T when given itself as input, i.e., the evaluation of T (T ). Regardless of whether T halts, a logical contradiction results. Explicitly, if we assume T halts, then the algorithm take T as input. Upon verifying that it does indeed halt, T would would then loop forever, leading to a contradiction. Com- bining this with the Church Turing thesis gives the problem a physical interpretation.

There exists no physically realizable process that can decide whether or not a Turing machine will halt in ﬁnite time

1.3.2 Rice’s Theorem

Closer inspection of the proof reveals that there is nothing special about the property, ‘will a given Turing machine halt?’ Here, the world halt could have been replaced with any number of other actions, such as ‘output 2’ or ’double all its inputs’ and a slight modiﬁcation of proof would lead to an identical contradiction. Despite the universality of Turing machines, there are a lot of decision problems they cannot compute.

11 MILE GU

The properties aforementioned are all examples of non-trivial7 black box properties 8. Suppose any such property P may be computed by some physical process for any Tur- ing Machine, then it would in particular be able to compute P for the Turing machine Q (See Fig 1.1.B), and thus solve the Halting problem. This contradiction necessarily implies that P is non-computable. This observation is captured by Rice’s theorem [Ric53].

Theorem 1 (Rice’s Theorem) Any non-trivial black box property of a Turing machine is non- computable.

Box 5 (Black Box Properties) :

One way of thinking about black box properties is to imagine that you are given a Turing machine that is entirely encased in a black box. For any particular input, the box spits out a particular output, but the internal workings of the box remain invisible. A black box property of such a Turing machine is a property of this box, i.e., it does not refer to the internal workings of the Turing machine.

Mathematically, a property of a Turing machine is a function f that maps each Turing machine to a real number. We say that f is a black box property if f(T1) = f(T2) for any two Turing machines T1 and T2 that are functionally equivalent. That is, T1 and T2 halts for the same set of inputs, and for all inputs in which they halt, T1(x) = T2(x).

Examples:

1. Does T output 4 on input 2? 2. Does T output only even numbers?

Counter Examples:

1. Does T halt before 1000 timesteps? 2. Does T feature more than 200 integer additions? Undecidable Properties

In mathematics, non-computable properties are referred to as being undecidable. We will use the two terms interchangeably, depending on context. The origin on the word is from Godel’s Incompleteness theorem [God92], which states that for any sufﬁciently

7The word ‘non-trivial’ used in this context refers to properties that do not take on the same value for every possible Turing machine. Clearly trivial properties are not useful. 8Black-box properties are properties that depend only on the functional relationship between the input and output of a given Turing machine. See Box. 5) for details

12 PHYSICS OF INFORMATION LECTURE NOTES complex mathematical system, there exists mathematical propositions that cannot be proven. Any proposition concerning the black box property of a Turing machine would be an example. Any universal computer is generally unable to predict its own long term behavior.

1.4 Towards Natural Universal Systems

Rice’s theorem implies that no physical process is capable of predicting the long term behavior of any universal system. While this fundamental statement limits the computational power of any physical process, it has little physical implication if all physical systems capable of universal computation are nothing more than an exotic curios- ity.

How prevalent are such systems in our universe? Given that this thesis will most likely be distributed electronically, you are most likely starring at a good approximation of one right now 9. Any programmer would know that writing a program to simulate the actions of a Turing machine would be a rather simple task, and any computer would be able to faithfully execute the program.

Universal systems are in fact, not very rare at all. In this section, we will review the circuit model10 and cellular automata [Wol86], both of which bear a closer relation to existing computers than the Turing machine.

1.4.1 The Circuit Model

An algorithm that acts on at most n bits of information can be deﬁned by a binary n n function fn : {0, 1} → {0, 1} . Thus, for a model of computation to be universal, it is Elementary logic gates sufﬁcient to show that model can evaluate all binary functions, fn.

Any binary function may be decomposed into logical operations that act on at most two bits (Fig. 1.2). In fact, a subset of such operations, referred to as elementary logic gates can be concatenated to evaluate any fn. One such universal gate set involves the operations (1) FANOUT, (2) SWAP, (3) AND and (4) NOT, whose deﬁnitions are outlined in (Fig. 1.2 A).

From a physical perspective, consider a system that is capable of

1. Initializing bits of data (creating bits),

2. Implementing each of the elementary logic gates (processing bits),

9The approximation comes about only because desktop computers have ﬁnite storage, which is a mi- nor technicality. Hard drive space can always be upgraded, and for all practical purposes, computers are universal. 10See Chapter 3 of [NC00] for further information.

13 MILE GU

Figure 1.2: Any binary function f on n-bits may be decomposed into a sequence of elementary logic gates (B). One possible universal gate set involves the operations listed in (C). FANOUT duplicates an input bit; SWAP switches its two inputs; NOT ﬂips its input while AND performs addition modulo 2.

3. Transferring the bits from one logical operation to another (storing bits).

These operations can be concatenated into a circuit-like structure (Fig 1.2 B) to construct an arbitrary binary function11. Thus, the said system is universal.

The major difference between the circuit model and the standard Turing machine is that while the input size to any particular circuit is fixed, a Turing machine may potentially receive input of unbounded size. This reflects the fact that the physical universe is finite, and thus any physical system may not a emulate system that contains a greater number of configurations.

The two models relate to each other through the notion of uniform circuit families 12. Given any particular algorithm deﬁned by a Turing machine T , we may deﬁne a family

of circuits {Cn} where the action of Cn coincides with the action T on input of size n. The set of circuits, when taken together, equate to the action of T .

While all Turing machines may be implemented by an appropriate circuit family, not all circuit families correspond to Turing machines. For example, we can construct

11A proof of this statement is available in many elementary books on mathematical logic or computer science. For example, [Man77]. 12A formal deﬁnition is available in Section 2.4 of [NC00].

14 PHYSICS OF INFORMATION LECTURE NOTES

a Hn that solves the Halting problem for each ﬁxed n, and thus a circuit family that solves the Halting problem! This discrepancy is resolved by noting that there exists no algorithm to construct each Hn. Thus, we limit ourselves to circuit families that are uniform, i.e., there exists an algorithm that on input n outputs the information needed to construct Hn. It can be shown that uniform circuit families and Turing machines are of equivalent computational power: Any function computable in one model is computable in the other.

1.4.2 Game of Life

The circuit model is a tool that can be used to prove the universality of models of computation that are more likely to occur in the natural world. The Game of Life is an interesting example. First devised by John Conway and published in the October 1970 issue of Scientiﬁc American under the column ‘Mathematical Games’[Gar70], the Game of Life soon gained a cult following due to its seemingly limitless possibilities despite a very simple set of rules. Later, this complexity was conﬁrmed when it was shown that the Game of life could simulate the behavior of any Turing machine [BCG82].

Figure 1.3: Gliders in the Game of Life (A) are persistent conﬁgurations of cells that evolve diagonally at the speed of 1 cell every four iterations. Easily created by glider guns (B), they can be used to encode bits, and their interactions with each other allow the construction of a universal gate set. The exact process of glider creation can best be viewed in 3 dimensions (C), where each layer represents the state of the system at a speciﬁc timestep.

15 MILE GU

The universe of the Game of life consists of an inﬁnite two-dimensional grid, resem- bling a chess board of inﬁnite size. Each square cell interacts with its eight neighbors, which can be viewed as the eight squares that correspond to plausible moves for a King in chess. The input to the system is encoded within the initial conditions of the board, where each individual cell may be either dead (0), or alive (1). At each timestep, Update Rules the grid evolves according the following update rules:

1. Any live cell with fewer than two or more than three live neighbors dies, due to either loneliness or overcrowding. Otherwise, it remains alive.

2. Any dead cell with exactly three alive adjacent cells becomes alive, and remains dead otherwise.

The universality of the Game of the Life results from its capacity to implement each of the operations necessary to construct an arbitrary logic circuit. In particular, one can demonstrate the existence of gliders, stables configurations of cells that propagate in some fixed sketched direction. The proof involves designating specific parts of the grid as wires, and encoding 1(0) as the existence (absence) of gliders on those particular wires [BCG82]. A series of rather ingenious constructions allow one create gliders and collide these gliders in specific fashions so that their interaction results the exact logic interactions required for universality (See Fig. 1.3)13.

While the circuit model appeared rather artiﬁcial since it required a careful concatena- tions of speciﬁc interactions in an exact order, the Game of Life appears more natural. The update rules of the Game of Life are isotropic, independent of location in the grid. The update rule may even be interpreted as a primitive model of evolving biological communities (and hence where the model got its name from). The universality of such a deceptively simple system suggests that universality may not be a rare trait [Wol02].

1.4.3 General Cellular Automata

The Game of Life is a specific example of a cellular automata (CA), a general class of discrete dynamical systems that have the potential to model many physical systems [Wol94, CD88, TM87, CM97]. A CA is essentially a discretization of the universe. Spa- tially, we consider an n dimensional grid of rectangular cells instead of continuous space, where each cell may be in one of a number of possible states Σ. A CA is n dimensional, if it consists of an n dimensional grid of cells. Time in this model flows discretely, so that the states of each cell changes at discrete time steps. Like the Game of Life, a general CA is initialized at time t = 0 in some particular configuration.

As in the physical universe, non-local interactions are restricted. Each cell s has a local

13The 3-d visualizations are generated courtesy of Adam’s Life Tower Program: http://itp.nyu.edu/ ap1607/glart/

16 PHYSICS OF INFORMATION LECTURE NOTES neighborhood N(s), which represents the set of cells that are close enough to s to directly affect its evolution. In the Game of Life, this would consist of the eight cells that lie in the 3 × 3 block directly centered on s. The interactions are deﬁned by a update rule f, a function that takes as input the states of each cell in N(s), and outputs a state in Σ. At each time step t, the state of each cell s(t) is computed by acting f on the neighborhood of s at time t − 1.

The isotropy of CAs makes them an advantageous candidate for modeling natural phenomena. Since the update rule is applied identically to each cell, much as the laws of physics are identical at each point in space time, Indeed, Wolfram points out that any patterns in nature may be directly explained by as those resulting from simple CAs [Wol85]. Rule 110

Many CAs are suspected to be universal, including numerous whose universality has been formally demonstrated. Rule 110 presents one of the simplest examples of such CAs [Coo04]. In contrast to the Game of Life, this CA is one-dimensional, its configuration at each timestep is entirely defined by an infinite one dimensional strip of cells, each of which may be either 0 (black) or 1 (white). At each time step, the state of each cell is dependent on its own state and the state of its two directly adjacent neighbors. The update function f is best described graphically:

In analogy with the Game of Life, gliders, stable structures that propagate either left or right along the strip exist in the Rule 110. These gliders can be engineered to interact in a way that allows universal computation [Coo04].

Rule 110 provides compelling evidence that universal systems can have remarkably simple descriptions. In particular, the dynamics of Rule 110 is entirely deﬁned by a boolean function on 3 bits, which can potentially implemented by fairly simply physical processes. If a physical system were to feature such a process, tesselated evenly through a two dimensional plane, universality would result. While the existence of such physical systems in nature is not immediately clear, it appears much more plausible.

1.5 Rice’s Theorem as a Limiting Principle

Rice’s Theorem presents an entirely new set of principles that govern what any physical system can and cannot do. While conservation laws restrict the creation of spon-

17 MILE GU

taneous energy, and the second law of thermodynamics eliminates perpetual motion machines, Rice’s theorem, together with the Church-Turing thesis, prohibits the existence of any physical device that could predict the long term behavior of any other device of sufﬁcient complexity.

Ever since the notion of universal computers was ﬁrst formalized, a growing number of physical systems have been shown to be universal. These include billiard balls [FT82], simple dynamical systems [Moo90] and the dynamics of 3-dimensional majority vot- ing cellular automata [CM97]. The standard strategy for such proofs often involves an explicit demonstration that the system may be used to implement a model of computation that is known to be universal, such as a universal set of gates or a particular universal CA.

It was traditionally believed that once the interactions of a physical system is com- pletely understood, its dynamics may always be predicted. The advent of Rice’s theorem suggests otherwise. There exists no device capable of predicting the long term behavior of an arbitrary system capable of universality. With a wide variety of physical systems in nature ﬁtting this role, Rice’s theorem posits a signiﬁcant limitation of what is knowable about the universe around us. In the next chapter, we will exploit this to show that not only is steady state behavior non-computable, but so are many interesting macroscopic properties.

18 References

[And72] P.W. Anderson. More Is Different. Science, 177(4047):393–396, 1972.

[BCG82] E. Berlekamp, J. Conway, and R. Guy. Winning Ways for your mathematical plays. Academic Press, Oxford, 1982.

[BCSS98] Lenore Blum, Felipe Cucker, Michael Shub, and Steve Smale. Complexity and real computation. Springer-Verlag New York, Inc., Secaucus, NJ, USA, 1998.

[CD88] B Chopard and M Droz. Cellular automata modeling of physical systems. Cambridge University Press, 1988.

[CM97] Cristopher C. Moore. Majority-vote cellular automata, ising dynamics, and p-completeness. J. Stat. Phys., 88(1/2):795–805, 1997.

[Coo04] M. Cook. Universality in elementary cellular automata. Complex Systems, 15, 2004.

[Dav65] M. D. Davis. The Undecidable. Raven Press, Hewlett, New York, 1965.

[Deu85] D. Deutsch. Quantum Theory, the Church-Turing Principle and the Univer- sal Quantum Computer. Proc. R. Soc. Lond. A, 400:97, 1985.

[FT82] Edward Fredkin and Tommaso Toffoli. Conservative Logic. International Journal of Theoretical Physics, 21(3/4):219–253, 1982.

[Gar70] Martin Gardner. Mathematical games. Sci. Am., 223:120–123, 1970.

[God92] Kurt Godel. On Formally Undecidable Propositions of Principia Mathematica and Related Systems. Springer, Berlin, 1992. Translated by B. Meltzer.

19 MILE GU

[Llo06] Seth Lloyd. Programming the Universe: A Quantum Computer Scientist Takes On the Cosmos. Alfred A. Knopf, Berlin, 2006. Translated by E. T. Beyer, Math- ematical Foundations of Quantum Mechanics (Princeton University Press, Princeton, NJ, 1955).

[Man77] Y. Manin. A course in mathematical logic. Springer-Verlag, Berlin, 1977.

[Moo90] Cristopher Moore. Unpredictability and undecidability in dynamical systems. Phys. Rev. Lett., 64(20):2354–2357, May 1990.

[NC00] M. A. Nielsen and I. L. Chuang. Quantum computation and quantum information. Cambridge University Press, Cambridge, 2000.

[Pen89] R. Penrose. The emperor’s new mind. Oxford University Press, Oxford, 1989.

[Pot06] Petrus H. Potgieter. Zeno machines and hypercomputation. Theoretical Computer Science, 358(1):23 – 33, 2006.

[Ric53] H. G Rice. Rice’s theorem. Trans. Amer. Math. Soc., 74(1):358–366, 1953.

[TM87] T.Toffoli and N. Margolus. Cellular automata machines. The MIT Press, Cam- bridge, Massachusetts, 1987.

[Tur36] A. M. Turing. On computable numbers, with an application to the Entschei- dungsproblem. Proc. Lond. Math. Soc. 2 (reprinted in [?]), 42:230, 1936.

[Wey49] H. Weyl. Philosophy of mathematics and natural science. Princeton: Prince- ton University Press, —c1949, 1949.

[Wol85] Stephen Wolfram. Undecidability and intractability in theoretical physics. Phys. Rev. Lett., 54(8):735–738, Feb 1985.

[Wol86] S. Wolfram. Theory and applications of cellular automata. World Scientiﬁc, Singapore, 1986.

[Wol94] S. Wolfram. Cellular automata and complexity. Addison-Wesley, Redwood City, CA, 1994.

[Wol02] S. Wolfram. A New Kind of Science. Wolfram Media, Redwood City, CA, 2002.