BOAZBARAK INTRODUCTIONTO THEORETICAL COMPUTERSCIENCE TEXTBOOK IN PREPARATION. AVAILABLEON HTTPS://INTROTCS.ORG Text available on https://github.com/boazbk/tcs - please post any issues there - thank you! This version was compiled on Wednesday 26th August, 2020 18:10 Copyright © 2020 Boaz Barak This work is licensed under a Creative Commons “Attribution-NonCommercial- NoDerivatives 4.0 International” license. To Ravit, Alma and Goren. Contents Preface 9 Preliminaries 17 0 Introduction 19 1 Mathematical Background 37 2 Computation and Representation 73 I Finite computation 111 3 Defining computation 113 4 Syntactic sugar, and computing every function 149 5 Code as data, data as code 175 II Uniform computation 205 6 Functions with Infinite domains, Automata, and Regular expressions 207 7 Loops and infinity 241 8 Equivalent models of computation 271 9 Universality and uncomputability 315 10 Restricted computational models 347 11 Is every theorem provable? 365 Compiled on 8.26.2020 18:10 6 III Efficient algorithms 385 12 Efficient computation: An informal introduction 387 13 Modeling running time 407 14 Polynomial-time reductions 439 15 NP, NP completeness, and the Cook-Levin Theorem 465 16 What if P equals NP? 483 17 Space bounded computation 503 IV Randomized computation 505 18 Probability Theory 101 507 19 Probabilistic computation 527 20 Modeling randomized computation 539 V Advanced topics 561 21 Cryptography 563 22 Proofs and algorithms 591 23 Quantum computing 593 VI Appendices 625 Contents (detailed) Preface 9 0.1 To the student ........................ 10 0.1.1 Is the effort worth it? . 11 0.2 To potential instructors ................... 12 0.3 Acknowledgements ..................... 14 Preliminaries 17 0 Introduction 19 0.1 Integer multiplication: an example of an algorithm . 20 0.2 Extended Example: A faster way to multiply (optional) 22 0.3 Algorithms beyond arithmetic . 27 0.4 On the importance of negative results . 28 0.5 Roadmap to the rest of this book . 29 0.5.1 Dependencies between chapters . 30 0.6 Exercises ........................... 32 0.7 Bibliographical notes .................... 33 1 Mathematical Background 37 1.1 This chapter: a reader’s manual . 37 1.2 A quick overview of mathematical prerequisites . 38 1.3 Reading mathematical texts . 39 1.3.1 Definitions ...................... 40 1.3.2 Assertions: Theorems, lemmas, claims . 40 1.3.3 Proofs ......................... 40 1.4 Basic discrete math objects . 41 1.4.1 Sets .......................... 41 1.4.2 Special sets ...................... 42 1.4.3 Functions ....................... 44 1.4.4 Graphs ........................ 46 1.4.5 Logic operators and quantifiers . 49 1.4.6 Quantifiers for summations and products . 50 1.4.7 Parsing formulas: bound and free variables . 50 1.4.8 Asymptotics and Big- notation . 52 푂 8 1.4.9 Some “rules of thumb” for Big- notation . 53 1.5 Proofs ............................. 54 1.5.1 Proofs and programs . .푂 . 55 1.5.2 Proof writing style . 55 1.5.3 Patterns in proofs . 56 1.6 Extended example: Topological Sorting . 58 1.6.1 Mathematical induction . 60 1.6.2 Proving the result by induction . 61 1.6.3 Minimality and uniqueness . 63 1.7 This book: notation and conventions . 65 1.7.1 Variable name conventions . 66 1.7.2 Some idioms ..................... 67 1.8 Exercises ........................... 69 1.9 Bibliographical notes .................... 71 2 Computation and Representation 73 2.1 Defining representations . 75 2.1.1 Representing natural numbers . 76 2.1.2 Meaning of representations (discussion) . 78 2.2 Representations beyond natural numbers . 78 2.2.1 Representing (potentially negative) integers . 79 2.2.2 Two’s complement representation (optional) . 79 2.2.3 Rational numbers, and representing pairs of strings ......................... 80 2.3 Representing real numbers . 82 2.4 Cantor’s Theorem, countable sets, and string represen- tations of the real numbers . 83 2.4.1 Corollary: Boolean functions are uncountable . 89 2.4.2 Equivalent conditions for countability . 89 2.5 Representing objects beyond numbers . 90 2.5.1 Finite representations . 91 2.5.2 Prefix-free encoding . 91 2.5.3 Making representations prefix-free . 94 2.5.4 “Proof by Python” (optional) . 95 2.5.5 Representing letters and text . 97 2.5.6 Representing vectors, matrices, images . 99 2.5.7 Representing graphs . 99 2.5.8 Representing lists and nested lists . 99 2.5.9 Notation . 100 2.6 Defining computational tasks as mathematical functions 100 2.6.1 Distinguish functions from programs! . 102 2.7 Exercises . 104 2.8 Bibliographical notes . 108 9 I Finite computation 111 3 Defining computation 113 3.1 Defining computation . 115 3.2 Computing using AND, OR, and NOT. 116 3.2.1 Some properties of AND and OR . 118 3.2.2 Extended example: Computing XOR from AND, OR, and NOT . 119 3.2.3 Informally defining “basic operations” and “algorithms” . 121 3.3 Boolean Circuits . 123 3.3.1 Boolean circuits: a formal definition . 124 3.3.2 Equivalence of circuits and straight-line programs 127 3.4 Physical implementations of computing devices (di- gression) . 130 3.4.1 Transistors . 131 3.4.2 Logical gates from transistors . 132 3.4.3 Biological computing . 132 3.4.4 Cellular automata and the game of life . 132 3.4.5 Neural networks . 132 3.4.6 A computer made from marbles and pipes . 133 3.5 The NAND function . 134 3.5.1 NAND Circuits . 135 3.5.2 More examples of NAND circuits (optional) . 136 3.5.3 The NAND-CIRC Programming language . 138 3.6 Equivalence of all these models . 140 3.6.1 Circuits with other gate sets . 141 3.6.2 Specification vs. implementation (again) . 142 3.7 Exercises . 143 3.8 Biographical notes . 146 4 Syntactic sugar, and computing every function 149 4.1 Some examples of syntactic sugar . 151 4.1.1 User-defined procedures . 151 4.1.2 Proof by Python (optional) . 153 4.1.3 Conditional statements . 154 4.2 Extended example: Addition and Multiplication (op- tional) .............................157 4.3 The LOOKUP function . 158 4.3.1 Constructing a NAND-CIRC program for LOOKUP . 159 4.4 Computing every function . 161 4.4.1 Proof of NAND’s Universality . 162 4.4.2 Improving by a factor of (optional) . 163 푛 10 4.5 Computing every function: An alternative proof . 165 4.6 The class SIZE . 167 4.7 Exercises . 170 4.8 Bibliographical(푇 notes ) . 174 5 Code as data, data as code 175 5.1 Representing programs as strings . 177 5.2 Counting programs, and lower bounds on the size of NAND-CIRC programs . 178 5.2.1 Size hierarchy theorem (optional) . 180 5.3 The tuples representation . 182 5.3.1 From tuples to strings . 183 5.4 A NAND-CIRC interpreter in NAND-CIRC . 184 5.4.1 Efficient universal programs . 185 5.4.2 A NAND-CIRC interpeter in “pseudocode” . 186 5.4.3 A NAND interpreter in Python . 187 5.4.4 Constructing the NAND-CIRC interpreter in NAND-CIRC . 188 5.5 A Python interpreter in NAND-CIRC (discussion) . 191 5.6 The physical extended Church-Turing thesis (discus- sion) .............................193 5.6.1 Attempts at refuting the PECTT . 195 5.7 Recap of Part I: Finite Computation . 199 5.8 Exercises . 201 5.9 Bibliographical notes . 203 II Uniform computation 205 6 Functions with Infinite domains, Automata, and Regular expressions 207 6.1 Functions with inputs of unbounded length . 208 6.1.1 Varying inputs and outputs . 209 6.1.2 Formal Languages . 211 6.1.3 Restrictions of functions . 211 6.2 Deterministic finite automata (optional) . 212 6.2.1 Anatomy of an automaton (finite vs. unbounded) 215 6.2.2 DFA-computable functions . 216 6.3 Regular expressions . 217 6.3.1 Algorithms for matching regular expressions . 221 6.4 Efficient matching of regular expressions (optional) . 223 6.4.1 Matching regular expressions using DFAs . 227 6.4.2 Equivalence of regular expressions and automata 228 6.4.3 Closure properties of regular expressions . 230 11 6.5 Limitations of regular expressions and the pumping lemma ............................231 6.6 Answering semantic questions about regular expres- sions .............................236 6.7 Exercises . 239 6.8 Bibliographical notes . 240 7 Loops and infinity 241 7.1 Turing Machines . 242 7.1.1 Extended example: A Turing machine for palin- dromes . 244 7.1.2 Turing machines: a formal definition . 245 7.1.3 Computable functions . 247 7.1.4 Infinite loops and partial functions . 248 7.2 Turing machines as programming languages . 249 7.2.1 The NAND-TM Programming language . 251 7.2.2 Sneak peak: NAND-TM vs Turing machines . 254 7.2.3 Examples . 254 7.3 Equivalence of Turing machines and NAND-TM pro- grams .............................257 7.3.1 Specification vs implementation (again) . 260 7.4 NAND-TM syntactic sugar . 261 7.4.1 “GOTO” and inner loops . 261 7.5 Uniformity, and NAND vs NAND-TM (discussion) . 264 7.6 Exercises . 265 7.7 Bibliographical notes . 268 8 Equivalent models of computation 271 8.1 RAM machines and NAND-RAM . 273 8.2 The gory details (optional) . 277 8.2.1 Indexed access in NAND-TM . 277 8.2.2 Two dimensional arrays in NAND-TM . 279 8.2.3 All the rest . 279 8.3 Turing equivalence (discussion) . 280 8.3.1 The “Best of both worlds” paradigm . 281 8.3.2 Let’s talk about abstractions . 281 8.3.3 Turing completeness and equivalence, a formal definition (optional) . 283 8.4 Cellular automata . 284 8.4.1 One dimensional cellular automata are Turing complete . 286 8.4.2 Configurations of Turing machines and the next-step function . 287 12 8.5 Lambda calculus and functional programming lan- guages ............................290 8.5.1 Applying functions to functions .