Time Complexity (1) CSCI 2670 Original Slides were written by Dr. Frederick W Maier Spring 2014 CSCI 2670 Time Complexity (1) Time Complexity I So far we've dealt with determining whether or not a problem is decidable. I But even if it is, it might be \too difficult” in practice to decide. I For a given string w and language L, it might require too much time or too much memory to determine whether or not w 2 L. I The time required to solve a problem is called its time complexity. I The memory required to solve a problem is called its space complexity. I Computational complexity theory is the study of the time and space complexity of problems. I Chapter 7 deal with time complexity. CSCI 2670 Time Complexity (1) Time Complexity k k I Suppose we have a TM to decide A = f0 1 jk ≥ 0g. I This language is context free and so decidable. I Informally, the time complexity is the number of steps required by the TM, as a function of the input size. I We want to know the number of steps needed to determine whether w 2 A. I We usually express time complexity as a function of the length of the input string w. I Note that if different input strings u and v both have length n, it might take more time to process u than v. I In worst case analysis, we are interested in the maximum number of steps required for an input string of length n. I In average case analysis, we are interested in the average number of steps required for an input string of length n. CSCI 2670 Time Complexity (1) Time Complexity Definition I If M is a deterministic TM that halts on all inputs, then the time complexity (running time) of M is the function f : N ! N, where f (n) is the maximum number of steps M uses on an input of length n. I We say that M runs in time f (n) and M is an f (n) Turing machine. I It is often more convenient to use estimates of f (n) rather than f (n) itself to describe the running time of a TM. I In asymptotic analysis, we estimate the running time of the algorithm when it is run on large inputs. I Not all terms of the function contribute very much to the running time and so can be ignored. I The most common estimates are big-O and small-O (little-O) estimates. CSCI 2670 Time Complexity (1) Big O Notation Definition If f and g are functions such that f ; g : N ! R+, then f (n) is O(g(n)) iff there exist positive integers c and n0 such that f (n) ≤ c · g(n) for all n ≥ n0. g(n) is an asymptotic upper bound (cg(n) is an upper bound of f (n)). \f (n) is O(g(n))" means that if we ignore constant factors, f (n) ≤ g(n). Example 3 2 3 I 5n + 2n + 22n + 6 is O(n ): Let c = 6 and n0 = 10. 2 I n is O(n ). Let c = 1 and n0 = 1. 2 2 I n is not O(n): Not matter what c and n0 are chosen, n cn for some n ≥ n0. CSCI 2670 Time Complexity (1) Big O Notation and Logarithms I The base of a logarithm doesn't matter when using Big-O notation. Note that for any bases a and b, log (n) = loga(n) . I b loga(b) loga(n) So, if f (n) ≤ clogb(n), then f (n) ≤ c . I loga(b) c Letting c1 = d e, it follows that f (n) ≤ c1loga(n). I loga(b) I So, if f (n) is O(logb(n)), f (n) is O(loga(n)). I We don't even bother with the base: f (n) is O(log(n)). Example If f (n) = 3nlog2(n) + 5nlog2(log2(n)) + 2, then f (n) is O(nlog(n)). CSCI 2670 Time Complexity (1) Arithmetic and Big O Notation I If f1(n) is O(g1(n)) and f2(n) is O(g2(n)), then I f1(n) + f2(n) is O(g1(n)) + O(g2(n)). I f1(n) + f2(n) is max(O(g1(n)); O(g2(n))). I If f (n) appears in an exponent, we can use the Big-O estimate there: 3n3+2n2+n+6 O(n3) I 2 is 2 . c I Frequently we derive bounds of the form n for c > 0. Such bounds are called polynomial bounds. (nδ ) I Bounds of the form 2 are called exponential bounds when δ is a real number greater than 0. CSCI 2670 Time Complexity (1) Small-O Notation I In a way, Big-O notation says that f (n) is less than or equal to g(n). I Small-O notation says that f (n) is less than g(n). Definition If f and g are functions such that f ; g : N ! R+, then f (n) is o(g(n)) iff f (n) lim = 0. n!1 g(n) Alternatively, f (n) is o(g(n)) iff for all real constants c > 0, there is an n0 such that f (n) < cg(n) for all n ≥ n0. Example p I n is o(n). I n is o(nlog(log(n)). 2 I nlog(n) is o(n ). 2 3 I n is o(n ). CSCI 2670 Time Complexity (1) Analyzing Algorithms k k I Consider TM M1 which decides A = f0 1 jk ≥ 0g. It works in 4 phases. On input w: 1. Scan the tape, rejecting if a 0 is found to the right of a 1. 2. While both 0s and 1s are still on the tape: 3. Scan the tape, marking off a single 0 and 1. 4. Reject if a 0 remains but all 1s are marked, or vice versa. If not, accept. I What is the running time of M1 as a function of n? CSCI 2670 Time Complexity (1) Analyzing Algorithms k k I Consider TM M1 which decides A = f0 1 jk ≥ 0g. It works in 4 phases. On input w: 1. Scan the tape, rejecting if a 0 is found to the right of a 1. 2. While both 0s and 1s are still on the tape: 3. Scan the tape, marking off a single 0 and 1. 4. Reject if a 0 remains but all 1s are marked, or vice versa. If not, accept. I Phase 1 scans once through the tape, taking O(n) steps, where jwj = n. The tapehead returns left|another n steps. Phase 1 takes O(n) steps. I In Phase 2-3, the tape is scanned to check that both 1s and 0s appear; another scan marks off a single 0 and 1. I In each cycle, 2 symbols are marked, and so the total number of cycles is O(n=2). Phases 2 and 3 together take O(n2) steps. I In Phase 4, we check to see that all 0s and 1s are marked off. This takes only a single scan of the tape: O(n). 2 2 I And so the running time of M1 is O(n) + O(n ) + O(n) = O(n ). CSCI 2670 Time Complexity (1) Complexity Classes: TIME(t(n)) 2 I Observe that M1 ran in time O(n ), and it decides A. I We can classify languages by the algorithms that decide them. Definition Let t : N ! R+ be a function. The time complexity class TIME(t(n)) is the set of all languages that can be decided in time O(t(n)). 2 3 I Observe, e.g., if L 2 TIME(n ), then L 2 TIME(n ). I If a decider M for L runs in time O(t(n)), then L 2 TIME(t(n)). k k 2 I So, A = f0 1 jk ≥ 0g is in time TIME(n ). I Failing to find a O(t(n))-time decider doesn't imply L 2= TIME(t(n)). CSCI 2670 Time Complexity (1) Analyzing Algorithms k k I Consider TM M2 which decides A = f0 1 jk ≥ 0g. It works in 5 phases. On input w: 1. Scan the tape, rejecting if a 0 is found to the right of a 1. 2. While some 0s and some 1s are on the tape: 3. Scan the tape. Reject if the number of unmarked symbols is odd. 4. Scan the tape, crossing off every other 0 and every other 1. 5. Scan the tape. If all symbols are marked, accept. Otherwise reject. I What's the running time of M2? CSCI 2670 Time Complexity (1) Analyzing Algorithms k k I Consider TM M2 which decides A = f0 1 jk ≥ 0g. It works in 5 phases. On input w: 1. Scan the tape, rejecting if a 0 is found to the right of a 1. 2. While some 0s and some 1s are on the tape: 3. Scan the tape. Reject if the number of unmarked symbols is odd. 4. Scan the tape, crossing off every other 0 and every other 1. 5. Scan the tape. If all symbols are marked, accept. Otherwise reject. I Phase 1 again takes O(n) steps, as does phase 5. I To check that some 0s and 1s appear (phase 2) takes O(n) steps. I Each execution of phase 3 and 4 takes O(n) steps. I Each execution of phase 4 cuts the number of 0s and 1s by half.