COMP 531 Advanced Complexity Theory Lecture Notes Hamed Hatami 2 1 Background: Before we start5 10 Randomized Complexity Classes 45 Decidable and Undecidable Languages:......5 Probabilistic Turing machines........... 45 Exercises......................6 Polynomial time Randomized complexity classes 46 The complexity class ZPP:.......... 46 2 Background: Basic Analysis7 The complexity class RP ........... 47 Some basic inequalities...............7 The complexity class BPP .......... 47 Measure spaces...................9 The complexity class PP:............. 48 Probability Spaces.............. 10 The relation between the classes......... 49 Normed spaces................... 11 How large is PP?............... 49 Hilbert Spaces................ 12 Randomness as advice and RP ⊆ NP .... 50 How large is BPP?.............. 50 The L spaces................. 13 p Exercises...................... 51 Exercises...................... 13 11 Non-uniform complexity: Circuits 53 3 Time complexity 15 What is a Boolean circuit?............ 53 Our computational model............. 15 The class P=Poly .................. 54 Time complexity.................. 16 Karp-Lipton Theorem............... 55 Time Hierarchy theorems............. 17 Shannon-Muller Lower bound........... 55 Exercises...................... 18 Adleman's theorem and advice: BPP ⊆ P=Poly . 56 Exercises...................... 57 4 Polynomial Hierarchy 19 Exercises...................... 21 12 AC0: Bounded Depth Alternating Circuits 59 Lower-bounds for AC circuits........... 60 5 Space complexity 23 Decision tree complexity........... 61 Time and Space.................. 23 Hastad's switching lemma............. 61 Logarithmic Space L and NL ........... 24 Influences in bounded depth circuits....... 64 Exercises...................... 25 Exercises...................... 65 6 Savitch's Theorem 27 13 Razborov-Smolensky 67 0 Savitch's theorem.................. 28 The class AC (⊕): Algebraic techniques..... 68 Exercises...................... 29 Razborov-Smolensky over F2 ........... 69 concluding remarks................. 71 7 NL versus coNL: Immerman-Szelepcsenyis Exercises...................... 71 Theorem 31 14 Razborov's monotone circuit lower-bound 75 NL versus coNL: Immerman-Szelepcsenyis Theo- Approximating the monotone circuit.... 76 rem...................... 31 The preliminary lemmas.............. 77 Exercises...................... 32 Approximation by a small number of cliques... 78 Concluding remarks................ 79 8 Ladner's Theorem: NP-completeness 35 Exercises...................... 79 Warm-up: Proof assuming the Exponential Time Hypothesis.................. 35 15 Razborov-Rudich: Natural Proof Barrier 81 The full proof of Ladner's Theorem........ 36 What are natural proofs?............. 81 Exercises...................... 37 The Razborov-Rudich Theorem.......... 82 Preparing for the proof: Pseudo-random Gener- 9 Oracles and relativization 39 ators...................... 83 Relativization................. 40 The proof of Razborov-Rudich Theorem..... 84 Relativization as a proof-barrier......... 41 Concluding Remarks................ 84 Exercises...................... 42 Exercises...................... 85 3 16 Fourier analysis and Polynomial Represen- Rank Lower-Bound................. 107 tations 87 Non-deterministic Communication complexity.. 107 Polynomial representations of f : {−1; 1gn ! R . 88 Exercises...................... 107 Fourier analysis of Finite Abelian Groups.... 90 Basic Fourier Theory................ 90 19 Randomized Communication complexity 109 Infinite Ableian groups and beyond..... 96 Yao's Min-Max Theorem.............. 110 Concluding remarks:................ 97 Exercises...................... 112 Exercises...................... 97 20 Discrepancy and Sign-rank 113 Discrepancy..................... 113 17 Learning and AC0 circuits 99 Other formulations of discrepancy..... 115 PAC learning from uniform samples.... 99 Unbounded-Error model.............. 115 Decision Trees................... 100 Sign-rank................... 116 Linial-Mansour-Nisan............... 101 Exercises...................... 117 Mansour-Nisan conjecture............. 102 Final remarks about Circuits........... 102 21 Machine Learning: Dimension and Margin 119 Exercises...................... 103 Sign-rank in Machine Learning: Dimension 119 Margin and Discrepancy.............. 121 18 Communication complexity 105 The Matrix Theory Background......... 122 Monochromatic rectangles, and Fooling sets... 105 Forster's Sign-rank Lower-bound...... 124 Rectangle Size Bounds............... 106 Exercises...................... 125 4 Chapter 1 Background: Before we start This is an advanced course on complexity theory, and thus we will assume that the reader is familiar with the basics of the theory of computation; topics such as Turing Machines, Non-determinism, Decidability, Undecidability, Turing reductions. We will also assume that the reader has some familiarity with the basic complexity theory: the complexity classes P, NP, coNP, the definition of NP through efficient verifiers, the notion of NP-completeness, and the Cook-Levin theorem. Similar to any theoretical computer science course, we shall use the asymptotic notations of O(·); Θ(·); Ω(·); o(·);!(·). In this short lecture, we quickly remind the reader of some of these concepts. Decidable and Undecidable Languages: An alphabet Σ is just a finite set, where the elements are interpreted as \symbols". In this course, almost always Σ = f0; 1g. We denote by Σ∗ the set of all finite strings constructed from the symbols in Σ. For example, if Σ = f0; 1g, then Σ∗ = f"; 0; 1; 00; 01; 10; 11;:::g. Here " denotes the empty string of length zero. We emphasis that all the elements in Σ∗ are of finite length. Every subset L ⊆ Σ∗ is called a language. Languages correspond to decision problems, i.e. YES/NO problems in a natural way: Given an input x, we would like to know whether x 2 L or x 62 L. A language is called decidable if and only if there is a Turing Machine that always halts and accepts x if x 2 L and rejects x if x 62 L. The Turing Machines that always halt are called algorithms. Since the set of the languages is uncountable, and the set of Turing Machines over any fixed alphabet is countable, there are languages that are undecidable. Recall that P is the class of the languages that can be decided in polynomial time, and NP is the class of the problems that can be decided in polynomial non-deterministic time. It is also useful to define NP through the notions of the verifiers. A verifier for a language L is an algorithm V that takes a pair (x; y) as input, and has the property that x 2 L () 9y; V (x; y) = Accept: Here we can think of y as a certificate/proof for the fact that x 2 L. That is if x 2 L, then there is a certificate for it and our algorithm can verify it. However if x 62 L, then such a certificate should not exist, and V should reject (x; y) for every y. A verifier is called efficient if its running time is bounded by a polynomial p in the length of x. Note that this is stronger than saying that V runs in polynomial time because the input of V is (x; y), but we bound its running time by p(jxj) rather than p(j(x; y)j). It is easy to see that NP is the set of the languages with efficient verifiers. An important problem in NP is the SAT problem, which in a certain sense is as hard as any other problem in the class NP. First we define the notion of a polynomial time reduction. Definition 1.1. A language A is polynomial time mapping reducible, or simply polynomial time reducible to a language ∗ ∗ B, written as A 6p B, if a polynomial time computable function f :Σ ! Σ exists, where for every w, w 2 A () f(w) 2 B: A Boolean formula over variables x1; : : : ; xn, is a formula consisting of variables and logical operators ^; _; : corresponding to AND, OR, and negation, respectively. For example φ(x1; x2; x3) := (x1 ^ x2) _ (x1 ^ x3) _ (x2 ^ x3) is a Boolean formula over the variables x1; x2; x3, and one can easily check that φ evaluates to True if and only if majority of the variables x1; x2; x3 are True. We will identify 1 with True and 0 with False, so for example φ(1; 1; 0) = 1. 5 Definition 1.2 (Conjunctive Normal Form (CNF)). A Boolean formula over variables x1; : : : ; xn is in CNF form if it is an AND of ORs of variables or their negations. We call formulas that are in CNF form, simply CNF formulas. The terms xi or :xi appearing in a formula are called terms or literals. For a CNF formula m 0 k 1 ^ _i φ = @ tij A ; i=1 j=1 ki where tij are terms, we call each (_j=1tij ) a clause of the formula. For example (x1 _:x2 _ x3) ^ (x2 _ x3) is a CNF formula with two clauses (x1 _:x2 _ x3) and (x2 _ x3). Theorem 1.3 (Cook-Levin Theorem, Cook [Coo71], Levin [Lev73]). The language of all satisfiable CNF formulas SAT = fφ j φ is a CNF, and there exists y such that φ(y) = Trueg; is NP-complete. That is it belongs to NP and furthermore X 6p SAT for every X 2 NP. It is easy to see that SAT is in fact in NP. To see this, note that a y that satisfies φ(y) = True can be used as a certificate to verify that φ is satisfiable. In other words an efficient certifier V takes the pair (φ, y), plugs y into φ and checks to see if φ is satisfied. If it is, then it accepts the pair, otherwise it rejects. Note that φ 2 SAT () 9y; V (φ, y) = Accept: The interesting and difficult part of the Cook-Levin theorem is the proof that every language in NP reduces to SAT. The proof is by encoding the computation of a Turing machine as a CNF. Big O notation : Finally let us finish this lecture by recalling the asymptotic notations. Let f; g : N ! N. We say • f(n) = O(g(n) if there exists n0; c > 0 such that f(n) < cg(n) for all n > n0. • f(n) = Ω(g(n) if there exists n0; c > 0 such that f(n) > cg(n) for all n > n0. • f(n) = Θ(g(n)) if there exists n0; c1; c2 > 0 such that c1g(n) < f(n) < c2g(n) for all n > n0. f(n) • f(n) = o(g(n)) if limn!1 g(n) = 0. f(n) • f(n) = !(g(n)) if limn!1 g(n) = 1.