1 The Limits of Computation The next time a bug freezes your computer, you can take solace in the deep mathe- matical truth that there is no way to eliminate all bugs -Seth Lloyd The 1999 science fiction movie, ‘The Matrix’, explored this idea. In this world, humans are plugged into a virtual reality, referred to as ‘The Matrix’. They go about their daily lives, unaware that the sensory inputs that they receive do not originate from their per- ceived reality. When a person, Alice, within the matrix observes a watermelon falling from a skyscraper, there is no skyscraper, nor watermelon, nor even gravity responsi- ble for the watermelon’s fall. Instead a complex computer program works silently in the background. The initial state of the watermelon, the location of the observer, are encoded in a sequence of bits. The computer takes these bits, processes according to a predetermined algorithm, and outputs the electrical signals that dictate what the observer should see. To people in the twenty first century, whose lives are enmeshed in various information processors, the eventual plausibility of the Matrix does not appear as radical as it once did. One by one, the photos we watch, and the mail we send, have been converted to digital form. Common questions such as “How many megabytes does that song take up?” reflect a society that is becoming increasingly accepting of the possibility that the observable qualities of every object can be represented by bits, and physical processes by how they manipulate these bits. While most people regard this as a representa- tion of reality, some scientists have even gone as far to speculate that the universe is perhaps just a giant information processor, programmed with the laws of physics we know [Llo06]. Why should a humble computer, that operates on the laws of electromagnetism, ever be capable of simulating all observable effects of gravity? How is it that we may use them to predict the fluctuations of gravitational waves or the reliability of a Boeing 1 MILE GU 747, when it neither detects gravity waves nor flies? A priori, there exists no reason why there should exist universal systems, computers capable of mimicking the output of any feasible physical experiment in the known universe short of the entire universe itself. The synonymity of universal systems and universal computers highlights the close links between the theories of computation, and that of physical reality. All practical computational devices must ultimately be implemented by some physical law, and all physical laws discovered so far may be simulated by a standard computer1. Not only does this suggest that universal devices allow us to simulate and thus better un- derstand reality, but also that discoveries made in the science of computation could naturally lead to insights about physical reality, and vice versa. The goal of this course is explore introduce this synergy. We consider three rapidly developing scientific fields linked to this central theme. The first demonstrates that understanding the limits and capabilities of computational devices can lead to further insight about our universe. The second motivates further development in theoretical computer science by consideration of how the difficulty of computational problems fundamentally depends on the laws of physics. The final topic revisits the Second Law of Thermodynamics from the perspective of information theory, giving profound demonstration of the idiom, ’Knowledge is Power.’ In this chapter, we begin this journey with a discussion on the ultimate limits of com- putation, and show how this can lead us to new understanding about universal prin- ciples of our reality. Section 1.1 launches the chapter in highlighting the connections between theorem’s in computer science and and understanding of reality. Section 1.2 introduces variations of the Church-Turing thesis [Tur36], a universal principle that roughly speaking, postulates that the computers we used today are such universal players. Section 1.3 describes the limitations of these universal computers exhibit, and 1.4 constructs formal models of universal computers that recasts these limita- tions to more concrete settings. In the next chapter, we combine these results with a proof that the ability to derive macroscopic laws from microscopic laws would neces- sitate the ability to do something that no universal computer can do, in violation of the Church-Turing thesis. 1.1 Emergent Laws, Universal Principles and Non-computability Much of fundamental physics has been motivated by a search for a ‘Theory of Every- thing’, a set of principles that govern all known dynamics of the universe. This ‘holy 1All currently known models of quantum computation can also be simulated by classical computers, albeit with exponential overhead. There does, however, exist a minority of scientists who believe there may exists undiscovered physical laws that cannot be simulated. For example, see [Pen89]. 2 PHYSICS OF INFORMATION LECTURE NOTES grail’ has motivated the construction of particle accelerators and many of the more es- oteric fields of theoretical physics. The standard reductionist rationale is that since all physical systems are built from conglomerations of fundamental particles, the princi- ples governing these such particles will also apply to all such systems. The plausible existence of universal systems suggests a complementary approach to the understanding of universal principles. If all observable qualities of any physical process may be simulated by a single system, then the limitations on that system will also be universal limitations that allow us to make generic statements about what we can observe within our universe. Indeed, there exists many tasks that universal computers cannot perform. The canon- ical example is the Halting problem [Tur36], which asks a computer to decide whether a given computer algorithm will eventually output some number, or be trapped in an infinite loop. A systematic solution of this problem would be of intense interest not only to software engineers, but also to any PhD student tired of spontaneous com- puter crashes when writing their thesis. The Halting problem is far from an isolated case, there exist many interesting problems that have no systematic solution. This immediately leads to a somewhat depressing universal principle (See Box 1) . ”The universe allows no systematic method to eliminate all bugs.” Philosophically, such principles may appear puzzling from a reductionist perspective. While most existing computers are electronic, such devices can in principle be con- structed from almost any physical medium. Numerous physical systems of surpris- ing simplicity, including even colliding billiard balls, have demonstrated universality [FT82]. The rather fundamental statement about arbitrary macroscopic systems is thus independent of microscopic composition. This motivates the idea of emergence; principles that govern macroscopic systems are not entirely predictable by the laws that govern their microscopic constituents [And72]. When we look at the physical universe around us, we often observe some sort of ‘macro- scopic order’. When we analyze the flow of water, or the dynamics of a glacier, we do not need to compute the exact motion of every atom. The trick here is that when we observe the macroscopic world, we generally neglect the microscopic details. We see and measure macroscopic features, such as pressure, density and stress. The earli- est scientists like Galileo and Archimedes observed relations between such quantities, without a care for the existence of atoms or quantum mechanics. When scientists write down equations relating such quantities, one should note that the quantities themselves implicitly assume a continuity limit. Pressure gradient, stress etc., are only formally defined in the limit where the medium is continuous, i.e., where it contains an infinite number of infinitesimal particles. 3 MILE GU Box 1 (Universal Principles) : ”The law that entropy always increases, holds, I think, the supreme position among the laws of Nature. If someone points out to you that your pet theory of the universe is in disagreement with Maxwell’s equations then so much the worse for Maxwell’s equations. If it is found to be contradicted by observation well, these experimentalists do bungle things sometimes. But if your theory is found to be against the second law of thermodynamics I can give you no hope; there is nothing for it but to collapse in deepest humiliation.” Sir Arthur Stanley Eddington, The Nature of the Physical World (1927) Any primary school student could tell you that should you drop a watermelon from the Eiffel Tower, it will plummet to the ground. Any claims otherwise would be met with disbe- lief. Yet, the kid probably never went to the Eiffel tower, and almost certainly was not there when a watermelon plummeted to its fruity demise. Without first hand experience, how is it that a child could rule out the infinite other possibilities... a watermelon hovering in midair, or spontaneous turning into a banana, perhaps? A key component of predicting natural phenomena is the knowledge of underlying rules of thumb that you expect very general classes of objects to obey. In this case, the child knows that heavier than air objects fall. This rule allows a child to rule out limitless number of scenarios, and hence have a good idea of what happens in scenarios that he or she has never observed. When these rules of thumb are thought to work for any physical system, they are regarded as universal principles. The truth of universal principles cannot be proven. There is nothing that prevents some exotic phenomena to be discovered in future, that demonstrably violates a given principle. We believe in the conservation of energy, because all observations and experiments so far indicate that it is true. Such universal principles not only give insight on how the universe behaves, but also help rule out proposals that carelessly violate them without experimental evidence or good rea- son.