Complexity Theory Lectures 1–6
Total Page:16
File Type:pdf, Size:1020Kb
Complexity Theory 1 Complexity Theory Lectures 1–6 Lecturer: Dr. Timothy G. Griffin Slides by Anuj Dawar Computer Laboratory University of Cambridge Easter Term 2009 http://www.cl.cam.ac.uk/teaching/0809/Complexity/ Cambridge Easter 2009 Complexity Theory 2 Texts The main texts for the course are: Computational Complexity. Christos H. Papadimitriou. Introduction to the Theory of Computation. Michael Sipser. Other useful references include: Computers and Intractability: A guide to the theory of NP-completeness. Michael R. Garey and David S. Johnson. Structural Complexity. Vols I and II. J.L. Balc´azar, J. D´ıaz and J. Gabarr´o. Computability and Complexity from a Programming Perspective. Neil Jones. Cambridge Easter 2009 Complexity Theory 3 Outline A rough lecture-by-lecture guide, with relevant sections from the text by Papadimitriou (or Sipser, where marked with an S). Algorithms and problems. 1.1–1.3. • Time and space. 2.1–2.5, 2.7. • Time Complexity classes. 7.1, S7.2. • Nondeterminism. 2.7, 9.1, S7.3. • NP-completeness. 8.1–8.2, 9.2. • Graph-theoretic problems. 9.3 • Cambridge Easter 2009 Complexity Theory 4 Outline - contd. Sets, numbers and scheduling. 9.4 • coNP. 10.1–10.2. • Cryptographic complexity. 12.1–12.2. • Space Complexity 7.1, 7.3, S8.1. • Hierarchy 7.2, S9.1. • Descriptive Complexity 5.6, 5.7. • Cambridge Easter 2009 Complexity Theory 5 Complexity Theory Complexity Theory seeks to understand what makes certain problems algorithmically difficult to solve. In Data Structures and Algorithms, we saw how to measure the complexity of specific algorithms, by asymptotic measures of number of steps. In Computation Theory, we saw that certain problems were not solvable at all, algorithmically. Both of these are prerequisites for the present course. Cambridge Easter 2009 Complexity Theory 6 Algorithms and Problems Insertion Sort runs in time O(n2), while Merge Sort is an O(n log n) algorithm. The first half of this statement is short for: If we count the number of steps performed by the Insertion Sort algorithm on an input of size n, taking the largest such number, from among all inputs of that size, then the function of n so defined is eventually bounded by a constant multiple of n2. It makes sense to compare the two algorithms, because they seek to solve the same problem. But, what is the complexity of the sorting problem? Cambridge Easter 2009 Complexity Theory 7 Lower and Upper Bounds What is the running time complexity of the fastest algorithm that sorts a list? By the analysis of the Merge Sort algorithm, we know that this is no worse than O(n log n). The complexity of a particular algorithm establishes an upper bound on the complexity of the problem. To establish a lower bound, we need to show that no possible algorithm, including those as yet undreamed of, can do better. In the case of sorting, we can establish a lower bound of Ω(n log n), showing that Merge Sort is asymptotically optimal. Sorting is a rare example where known upper and lower bounds match. Cambridge Easter 2009 Complexity Theory 8 Review The complexity of an algorithm (whether measuring number of steps, or amount of memory) is usually described asymptotically: Definition For functions f : IN IN and g : IN IN, we say that: → → f = O(g), if there is an n0 IN and a constant c such that for • ∈ all n > n0, f(n) cg(n); ≤ f = Ω(g), if there is an n0 IN and a constant c such that for • ∈ all n > n0, f(n) cg(n). ≥ f = θ(g) if f = O(g) and f = Ω(g). • Usually, O is used for upper bounds and Ω for lower bounds. Cambridge Easter 2009 Complexity Theory 9 Lower Bound on Sorting An algorithm A sorting a list of n distinct numbers a1,...,an. ai < aj ? ap < aq ? ak < al? . ar < a.s? . done done done done done To work for all permutations of the input list, the tree must have at least n! leaves and therefore height at least log2(n!) = θ(n log n). Cambridge Easter 2009 Complexity Theory 10 Travelling Salesman Given V — a set of nodes. • c : V V IN — a cost matrix. • × → Find an ordering v1,...,vn of V for which the total cost: n−1 c(v , v )+ c(v , v ) n 1 X i i+1 i=1 is the smallest possible. Cambridge Easter 2009 Complexity Theory 11 Complexity of TSP Obvious algorithm: Try all possible orderings of V and find the one with lowest cost. The worst case running time is θ(n!). Lower bound: An analysis like that for sorting shows a lower bound of Ω(n log n). Upper bound: The currently fastest known algorithm has a running time of O(n22n). Between these two is the chasm of our ignorance. Cambridge Easter 2009 Complexity Theory 12 Formalising Algorithms To prove a lower bound on the complexity of a problem, rather than a specific algorithm, we need to prove a statement about all algorithms for solving it. In order to prove facts about all algorithms, we need a mathematically precise definition of algorithm. We will use the Turing machine. The simplicity of the Turing machine means it’s not useful for actually expressing algorithms, but very well suited for proofs about all algorithms. Cambridge Easter 2009 Complexity Theory 13 Turing Machines For our purposes, a Turing Machine consists of: K — a finite set of states; • Σ — a finite set of symbols, including . • ⊔ s K — an initial state; • ∈ δ :(K Σ) (K a, r ) Σ L,R,S • × → ∪ { } × × { } A transition function that specifies, for each state and symbol a next state (or accept acc or reject rej), a symbol to overwrite the current symbol, and a direction for the tape head to move (L – left, R – right, or S - stationary) Cambridge Easter 2009 Complexity Theory 14 Configurations A complete description of the configuration of a machine can be given if we know what state it is in, what are the contents of its tape, and what is the position of its head. This can be summed up in a simple triple: Definition A configuration is a triple (q,w,u), where q K and w,u Σ⋆ ∈ ∈ The intuition is that (q,w,u) represents a machine in state q with the string wu on its tape, and the head pointing at the last symbol in w. The configuration of a machine completely determines the future behaviour of the machine. Cambridge Easter 2009 Complexity Theory 15 Computations Given a machine M =(K, Σ,s,δ) we say that a configuration (q,w,u) yields in one step (q′, w′,u′), written ′ ′ ′ (q,w,u) M (q , w ,u ) → if w = va ; • δ(q, a)=(q′,b,D); and • either D = L and w′ = v u′ = bu • or D = S and w′ = vb and u′ = u or D = R and w′ = vbc and u′ = x, where u = cx. If u is empty, then w′ = vb and u′ is empty. ⊔ Cambridge Easter 2009 Complexity Theory 16 Computations ⋆ The relation is the reflexive and transitive closure of M . →M → A sequence of configurations c1,...,cn, where for each i, ci M ci+1, is called a computation of M. → The language L(M) Σ⋆ accepted by the machine M is the set of ⊆ strings x (s, ⊲, x) ⋆ (acc,w,u) for some w and u { | →M } A machine M is said to halt on input x if for some w and u, either (s, ⊲, x) ⋆ (acc,w,u) or (s, ⊲, x) ⋆ (rej,w,u) →M →M Cambridge Easter 2009 Complexity Theory 17 Decidability A language L Σ⋆ is recursively enumerable if it is L(M) for some ⊆ M. A language L is decidable if it is L(M) for some machine M which halts on every input. A language L is semi-decidable if it is recursively enumerable. A function f : Σ⋆ Σ⋆ is computable, if there is a machine M, → such that for all x, (s, ⊲, x) ⋆ (acc, f(x), ε) →M Cambridge Easter 2009 Complexity Theory 18 Example Consider the machine with δ given by: ⊲ 0 1 ⊔ s s,⊲,R s, 0,R s, 1,R q, , L ⊔ q acc,⊲,R q, , L rej, ,R q, , L ⊔ ⊔ ⊔ This machine will accept any string that contains only 0s before the first blank (but only after replacing them all by blanks). Cambridge Easter 2009 Complexity Theory 19 Multi-Tape Machines The formalisation of Turing machines extends in a natural way to multi-tape machines. For instance a machine with k tapes is specified by: K, Σ, s; and • δ :(K Σk) K a, r (Σ L,R,S )k • × → ∪ { }× × { } Similarly, a configuration is of the form: (q, w1,u1,...,wk,uk) Cambridge Easter 2009 Complexity Theory 20 Complexity For any function f : IN IN, we say that a language L is in → TIME(f(n)) if there is a machine M =(K, Σ,s,δ), such that: L = L(M); and • The running time of M is O(f(n)). • Similarly, we define SPACE(f(n)) to be the languages accepted by a machine which uses O(f(n)) tape cells on inputs of length n. In defining space complexity, we assume a machine M, which has a read-only input tape, and a separate work tape. We only count cells on the work tape towards the complexity. Cambridge Easter 2009 Complexity Theory 21 Nondeterminism If, in the definition of a Turing machine, we relax the condition on δ being a function and instead allow an arbitrary relation, we obtain a nondeterministic Turing machine.