Church-Turing Thesis: Every function f: {0,1}* -> {0,1}* which is "effectively computable" is computable by a Turing Machine. Observation: Every Turing Machine (equivalently, every python program, or evey program in your favorite language) M can be encoded by a finite binary string #M. Moreover, our encoding guarantees #M != #M' whenever M and M' are different Turing Machines (different programs). Recall: Every finite binary string can be thought of as a natural number written in binary, and vice versa. Therefore, we can think of #M as a natural number. E.g. If #M = 1057, then we are talking about the 1057th Turing Machine, or 1057th python program. Intuitive takeaway: At most, there are as many different Turing Machines (Programs) as there are natural numbers. We will now show that some (in fact most) decision problems f: {0,1}* -> {0,1} are not computable by a TM (python program). If a decision problem is not comptuable a TM/program then we say it is "undecidable". Theorem: There are functions from {0,1}* to {0,1} which are not computable by a Turing Machine (We say, undecidable). Corollary: Assuming the CT thesis, there are functions which are not "effectively computable". Proof of Theorem: - Recall that we can think of instead of of {0,1}* - "How many" functions f: -> {0,1} are there? - We can think of each each such function f as an infinite binary string - For each x in (0,1), we can write its binary expansion as an infinite binary string to the right of a decimal point - The string to the right of the decimal point can be interpreted as a function f_x from natural numbers to {0,1} -- f_x(i) = ith bit of x to the right of the decimal point - In other words: for each x in (0,1) we get a different function f_x from natural numbers to {0,1}, so there are at least as many decision problems as there are numbers in (0,1). So, there are at least as many decision problems as there are numbers in (0,1). How many different Turing Machines (python programs) are there? - Recall: Each Turing machine (python program) can be encoded as a finite string. Also, finite binary strings are in a bijection with the natural numbers. - At most, there are as many Turing Machines (python programs) as there are natural numbers. -- We say: Turing machines are "countable". Suppose for a contradiction that each decision problem can be solved by a TM (or, a python program). - We have an injective map from decision problems to Turing Machines. Composing the two injective maps, we have an injective map from (0,1) to the natural numbers - I.e., (0,1) are countable. Theorem (Cantor): (0,1) are uncountable. (There does not exist an injection from (0,1) to the natural numbers). Note: Often this is stated as the real numbers are uncountable. This is equivalent, because We arrived at a contradiction. Upshot: There are as many decision problems as there are real numbers, but there are only as many algorithms as there are whole numbers. There are way more problems than there are algorithms. Recall: Turing Machines === Python programs === C programs === Algorithms in your favorite language. - Historically, we used to think of each algorithm as a different "machine". In other words, algorithms were thought of as "hardware", while inputs/outputs/intermediate results were thought of as the "software". - One of Turing's great insights, which is obvious to us but was revolutionary at the time, is that you can think of algorithms as "software". -- Each Turing Machine, or Algorithm in your favorite language, can be written as a finite binary string. -- There exists a "Universal Turing Machine" (today we call it "interpreter") which can take the description of another TM / Algorithm as input, and "simulate it". Definition: A Universal Turing Machine U is a Turing Machine which takes as input a string #M describing a different Turing machine M, and another string x, and "simulates" M on x. Nowadays we call U an "interpreter" (software perspective), or a "general purpose computer" (hardware perspective) Theorem (Turing): A Universal Turing Machine exists. Great insight at the time, and the proof was quite technical. Interpretation in modern lingo: --- Interpreters exist (software perspective). In other words, you can write an interpreter for your favorite language in your favorite language. --- General purpose computers exist (Hardware perspective). A computer can run virtual machines simulating other computers. Important to remember: An algorithm can take in another algorithm as input, simulate it on whatever input it wants, postprocess the output, etc. Input: - Algorithm M, written as a finite string in your favorite programming language. (e.g. Python, C, TM) - Binary string x Output: Does M halt on input x? Return H (meaning, "halts") if yes, otherwise return L (meaning "loops forever). Theorem: The halting problem is undecidable. In other words, there is no algorithm, no matter its runtime, which solves the halting problem, assuming the CT thesis. .