Introduction to the Analysis of Complexity Chapter 27 1 Complexity Theory Are all decidable languages equal? ● (ab)* ● WWR = {wwR : w Î {a, b}*} ● WW = {ww : w Î {a, b}*} ● SAT = {w : w is a wff in Boolean logic and w is satisfiable} Computability: classifies problems into: solvable and unsolvable Complexity: classifies solvable problems into: easy ones and hard ones • Easy: tractable • Hard: intractable • Complexity zoo: hierarchy of classes 2 The Traveling Salesman Problem 15 25 10 28 20 4 40 8 9 7 3 23 Given n cities and the distances between each pair of them, find the shortest tour that returns to its starting point and visits each other city exactly once along the way. 3 The Traveling Salesman Problem 15 25 10 28 20 4 40 8 9 7 3 23 Given n cities: Choose a first city n Choose a second n-1 Choose a third n-2 … n! 4 The Growth Rate of n! 2 2 11 479001600 3 6 12 6227020800 4 24 13 87178291200 5 120 14 1307674368000 6 720 15 20922789888000 7 5040 16 355687428096000 8 40320 17 6402373705728000 9 362880 18 121645100408832000 10 3628800 19 2432902008176640000 11 39916800 36 3.6×1041 5 Tackling Hard Problems 1. Use a technique that is guaranteed to find an optimal solution. 2. Use a technique that is guaranteed to run quickly and find a “good” solution. – The World Tour Problem Does it make sense to insist on true optimality if the description of the original problem was approximate? 6 Modern TSP From: http://xkcd.com/399/ 7 The Complexity Zoo The attempt to characterize the decidable languages by their complexity: http://qwiki.stanford.edu/wiki/Complexity_Zoo Notable ones: P: solvable (decidable) by deterministic TM in polynomial time • tractable • context-free (including regular) languages are in P NP: solvable (decidable) by nondeterministic TM in polynomial time • given solution can be verified by DTM in polynomial time NP-complete: as hard as any one in NP & in NP (hardest ones in NP) • no efficient algorithm is known • require non-trivial search, as in TSP NP-hard: as hard as any one in NP (not necessarily in NP) • Every problem in NP are reducible to it in polynomial time • L is NP-complete if it is in NP + it is NP-hard • Intractable = not in P. But since it’s believed P ¹ NP, loosely intractable = NP hard 8 Note • NP is a set of decision problems. Computational complexity primarily deals with decision problems. • There is a link between the "decision" and "optimization" problems in that if there exists a polynomial algorithm that solves the "decision" problem, then one can find the maximum value for the optimization problem in polynomial time by applying this algorithm iteratively while increasing the value of k . • On the other hand, if an algorithm finds the optimal value of the optimization problem in polynomial time, then the decision problem can be solved in polynomial time by comparing the value of the solution output by this algorithm with the value of k . • Thus, both versions of the problem are of similar difficulty. • Note that, NP-hard is not restricted to decision problems, it also includes optimization problems. • When the decision problem is NP-complete, the optimization problem is NP-hard 9 Methodology • Characterizing problems as languages to be comparable – SAT = {w : w is a wff in Boolean logic and w is satisfiable} – only applies to decidable languages. nothing to say about H – Including optimization problems • TSP-DECIDE = {<G, cost> : <G> encodes an undirected graph with a positive distance attached to each of its edges and G contains a Hamiltonian circuit whose total cost is less than <cost>}. • minimizing -> at most k • maximizing -> at least k • All problems are decision problems – requires at least enough time to write the solution – By restricting our attention to decision problems, the length of the answer is not a factor • Encoding matters as complexity is w.r.t. problem size – L is regular for one encoding, nonregular for another – E.g., for integers, we should use any base other than 1 111111111111 vs 1100 111111111111111111111111111111 vs 11110 10 Measuring Time and Space Complexity • Model of computation: TM • timereq(M) is a function of n: – If M is deterministic timereq(M) = f(n) = the maximum number of steps that M executes on any input of length n. – If M is nondeterministic timereq(M) = f(n) = the number of steps on the longest path that M executes on any input of length n. • spacereq(M) is a function of n: – If M is deterministic spacereq(M) = f(n) = the maximum number of tape squares that M reads on any input of length n. – If M is nondeterministic spacereq(M) = f(n) = the maximum number of tape squares that M reads on any path that it executes on any input of length n. • Focus on worst-case performance – Interested in upper bound – Easy to determine – Beware: could be quite different from average-case – for most real problems, worst case is rare 11 Growth Rates of Functions 12 Asymptotic upper bound - O f(n) Î O(g(n)) iff there exists a positive integer k and a positive constant c such that: "n ³ k (f(n) £ c g(n)). In other words, ignoring some number of small cases (all those of size less than k), and ignoring some constant factor c, f(n) is bounded from above by g(n). f (n) Alternatively, if the limit exists: lim < ¥ n®¥ g(n) In this case, we’ll say that f is “big-oh” of g or g asymptotically dominates f or g grows at least as fast as f does 13 Asymptotic upper bound - O • n3 Î O(n3) • n3 Î O(n4) • 3n3 Î O(n3) • n3 Î O(3n) • n3 Î O(n!) • log n Î O(n) 14 Summarizing O b n O(c) Í O(loga n) Í O(n ) Í O(d ) Í O(n!) 15 Asymptotic strong upper bound o f(n) Î o(g(n)) iff, for every positive c, there exists a positive integer k such that: "n ³ k (f(n) < c g(n)) Alternatively, if the limit exists: f (n) lim = 0 n®¥ g(n) In this case, we’ll say that f is “little-oh” of g or that g grows strictly faster than f does. 16 Asymptotic lower bound - W f(n) Î W(g(n)) iff there exists a positive integer k and a positive constant c such that: "n ³ k (f(n) ³ c g(n)) In other words, ignoring some number of small cases (all those of size less than k), and ignoring some constant factor c, f(n) is bounded from below by g(n). Alternatively, if the limit exists: f (n) lim > 0 n®¥ g(n) In this case, we’ll say that f is “big-Omega” of g or that g grows no faster than f. 17 Asymptotic strong lower bound w f(n) Î w(g(n)) iff, for every positive c, there exists a positive integer k such that: "n ³ k (f(n) > c g(n)) Alternatively, if the required limit exists: fn() lim =¥ n®¥ gn() In this case, we’ll say that f is “little-omega” of g or that g grows strictly slower than f does. 18 Asymptotic tight bound Q f(n) Î Q(g(n)) iff there exists a positive integer k and positive constants c1, and c2 such that: "n ³ k (c1 g(n) £ f(n) £ c2 g(n)) Or: Or: f(n) Î Q(g(n)) iff: f(n) Î Q(g(n)) iff: f(n) Î O(g(n)), and f(n) Î O(g(n)), and g(n) Î O(f(n)). f(n) Î W(g(n)). Is n3 Î Q(n3)? Is n3 Î Q(n4)? 3 5 Is n Î Q(n )? 19 Asymptotic Dominance 20 Algorithmic Gaps We’d like to show: 1. Upper bound: There exists an algorithm that decides L and that has complexity C1. 2. Lower bound: Any algorithm that decides L must have complexity at least C2. 3. C1 = C2 If C1 = C2, we are done. Often, we’re not done. 21 Time-Space Tradeoffs - Search Depth-first search: small space, big time • potential problem: get stuck in one path Breadth-first search: big space, small time Iterative-deepening search: compromise • depth-first on length 1 paths, length 2 paths and so on • Space same as depth-first, time slightly worse than breath-first 22 Time Complexity Classes Chapter 28 23 The Language Class P L Î P iff ● there exists some deterministic Turing machine M that decides L, and ● timereq(M) Î O(nk) for some k. We’ll say that L is tractable iff it is in P. To show a language is in P: • describe a one-tape, deterministic Turing machine • state an algorithm that runs on a regular random-access computer – easier 24 Languages That Are in P • Every regular language • Every context-free language since there exist context-free parsing algorithms that run in O(n3) time. • Others: – AnBnCn 25 The Language Class NP Nondeterministic deciding: L Î NP iff: ● there is some NDTM M that decides L, and ● timereq(M) Î O(nk) for some k. NDTM deciders: longest path is polynomial s,!abab q2,#abab q1,!abab q1,!abab q3,!bbab 26 Deterministic Verifying A Turing machine V is a verifier for a language L iff: w Î L iff $c (<w, c> Î L(V)). We’ll call c a certificate. An alternative definition for the class NP: L Î NP iff there exists a deterministic TM V such that: • V is a verifier for L, and • timereq(V) Î O(nk) for some k.