sign in sign up
<<
Home , Ackermann function, Large numbers

15-251 This is The Big Oh! Great Theoretical Ideas Lecture 21, November 4, 2008 in Computer Science

Counting Algebra

The Power of One The Power of X Asymptotics How to add 2 n-

The Power of O * * * * * * * * * * * + * * * * * * * * * * *

How to add 2 n-bit numbers How to add 2 n-bit numbers

* * * * * * * * * * * * * * * * * * * * * * * * * + * * * * * * * * * * * + * * * * * * * * * * *

* * *

How to add 2 n-bit numbers How to add 2 n-bit numbers

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * + * * * * * * * * * * * + * * * * * * * * * * *

* * * * * * *

How to add 2 n-bit numbers of grade school * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * T(n) = amount of time + * * * * * * * * * * * + * * * * * * * * * * grade school * * * * * * * * * * addition uses to add two n-bit numbers * * * * * * * * * * * * * * * * * * * * * * * What do we mean by “time”? “Grade school addition” Roadblock ??? Our Goal A given will take different We want to define “time” in a way amounts of time on the same inputs that transcends implementation depending on such factors as: details and allows us to make – Processor speed assertions about grade school – Instruction set addition in a very general yet useful way. – Disk speed – Brand of compiler

On any reasonable computer, adding 3 and writing down the two bit answer On another computer M’, the time can be done in constant time to perform may be c’.

Pick any particular computer M and define c Total time to add two n-bit numbers to be the time it takes to perform on using grade school addition: that computer. c’n [c’ time for each of n columns] Total time to add two n-bit numbers using grade school addition:

cn [i.e., c time for each of n columns] Thus we arrive at an t implementation-independent i Machine M: cn m insight: Machine M’: c’n e Grade School Addition is a linear time algorithm # of bits in the numbers

This process of abstracting away details The fact that we get a line is invariant and determining the rate of resource under changes of implementations. usage in terms of the problem size n is Different machines result in different one of the fundamental ideas in slopes, but the time taken grows computer science. linearly as input size increases.

Time vs Input Size How to multiply 2 n-bit numbers.

For any algorithm, define X * * * * * * * * Input Size = # of bits to specify its inputs. * * * * * * * *

Define * * * * * * * * * * * * * * * * TIMEn = the worst-case amount of time used by the algorithm * * * * * * * * n2 * * * * * * * * on inputs of size n * * * * * * * * * * * * * * * * We often ask: What is the growth rate of * * * * * * * * Timen ? * * * * * * * * * * * * * * * * * * * * * * * * How to multiply 2 n-bit numbers. Grade School Addition: Linear time Grade School : Quadratic time X * * * * * * * * * * * * * * * * t The total time is* *bounded * * * * * by* i cn2 (abstracting * * * *away * * * the* m * * * * * * * * e implementation details). n2 * * * * * * * * * * * * * * * * * * * * * * * * # of bits in the numbers * * * * * * * * * * * * * * * * No matter how dramatic the difference in the constants, the quadratic curve will * * * * * * * * * * * * * * * * eventually dominate the linear curve

Grade School Multiplication: Quadratic time

t i m e How much time does it take to square the n using # of bits in numbers grade school multiplication? c(log n)2 time to square the number n Input size is measured in bits, unless we say otherwise. How much time does it take? Worst Case Time Nursery School Addition Input: Two n-bit numbers, a and b Worst Case Time T(n) for algorithm A: Output: a + b T(n) = Max[all permissible inputs X of size n] Start at a and increment (by 1) b times (Running time of algorithm A on input X). T(n) = ? If b = 000…0000, then NSA takes almost no time If b = 1111…11111, then NSA takes cn2n time

What is T(n)? Thus, Nursery School adding and Kindergarten Multiplication Kindergarten multiplication are exponential time. Input: Two n-bit numbers, a and b Output: a * b They scale HORRIBLY as input size Start with a and add a, b-1 times grows.

Remember, we always pick the WORST Grade school methods scale CASE input for the input size n. polynomially: just linear and quadratic. Thus, we can add and multiply fairly Thus, T(n) = cn2n . If T(n) is not polynomial, the algorithm Multiplication is efficient, what about is not efficient: the run time scales too “reverse multiplication”? poorly with the input size.

Let’s define FACTORING(N) to be any This will be the yardstick with which method to produce a non-trivial factor of we will measure “efficiency”. N, or to assert that N is prime.

Factoring The Number N By Trial Trial division up to !N for k = 2 to ! N do Can we do better? if k | N then return “N has a non-trivial factor k” We know of methods for FACTORING return “N is prime” that are sub-exponential (about 2n1/3 time) c !N (logN)2 time if division is c (logN)2 time but nothing efficient. Is this efficient? No! The input length n = log N. Hence we’re using c 2n/2 n2 time. Notation to Discuss Growth Rates f = O(n) means that there is a line For any monotonic function f from the that can be drawn that stays above positive to the positive integers, f from some point on we say “f = O(n)” or “f is O(n)”

If some constant times n eventually t dominates f i m e [Formally: there exists a constant c such that for all sufficiently large n: f(n) ! cn ] # of bits in numbers

Other Useful Notation: ! f = !(n) means that there is a line For any monotonic function f from the that can be drawn that stays below positive integers to the positive integers, f from some point on we say “f = !(n)” or “f is !(n)”

If f eventually dominates some constant t times n i m [Formally: there exists a constant c such e that for all sufficiently large n: f(n) " cn ]

# of bits in numbers f = "(n) means that f can be Yet More Useful Notation: " sandwiched between two lines from some point on. For any monotonic function f from the positive integers to the positive integers, we say “f = "(n)” or “f is "(n)” t i if: f = O(n) and f = !(n) m e

# of bits in numbers

Notation to Discuss Growth Rates f = O(g) means that there is some For any two monotonic functions f and g constant c such that c g(n) stays from the positive integers to the positive above f(n) from some point on. integers, we say “f = O(g)” or “f is O(g)” 1.5g If some constant times g eventually t i dominates f g m f [Formally: there exists a constant c such e that for all sufficiently large n: f(n) ! c g(n) ] # of bits in numbers Other Useful Notation: ! For any two monotonic functions f and g Yet More Useful Notation: " from the positive integers to the positive integers, we say For any two monotonic functions f and g from the positive integers to the positive “f = !(g)” or “f is !(g)” integers, we say If f eventually dominates some constant “f = "(g)” or “f is "(g)” times g If: f = O(g) and f = !(g) [Formally: there exists a constant c such that for all sufficiently large n: f(n) " c g(n) ]

• n = O(n2) ? Yes! • n = O(n2) ? Yes!

• n = O(#n) ? No Take c = 1 For all n " 1, it holds that n ! cn2 Suppose it were true that n ! c #n for some constant c and large enough n Cancelling, we would get #n ! c. Which is false for n > c2 • n = O(n2) ? Yes! • f = O(g) and g = O(h) then f = O(h) ? Yes! • n = O(#n) ? No

f(n) ! c g(n) for all n " n0. 2 2 • 3n + 4n + 4 = O(n ) ? Yes! and g(n) ! c’ h(n) for all n " n0’.

• 3n2 + 4n + 4 = !(n2) ? Yes! So f(n) ! (cc’) h(n) for all n " max(n0, n0’)

2 • n = !(n log n) ? Yes! • f = O(g) then g = !(f) Yes! • n2 log n = "(n2) ? No

Names For Some Growth Rates Large Growth Rates

Exponential Time: Linear Time: T(n) = O(n) for some constant k, T(n) = O(kn) Quadratic Time: T(n) = O(n2) Example: T(n) = n2n = O(3n) Cubic Time: T(n) = O(n3)

Polynomial Time: for some constant k, T(n) = O(nk). Example: T(n) = 13n5 Small Growth Rates Some Big Ones

Logarithmic Time: T(n) = O(logn) Doubly Exponential Time means Example: T(n) = 15log2(n) that for some constant k 2k n Polylogarithmic Time: T(n) = 2 for some constant k, T(n) = O(logk(n)) Triply Exponential Note: These kind of can’t 2k n possibly read all of their inputs. T(n) = 22 A very common example of logarithmic time is looking up a word in a sorted dictionary (binary search)

Faster and Faster: 2STACK And the inverse of 2STACK: log* 2STACK(0) = 1 2STACK(0) = 1 2STACK(n) = 22STACK(n-1) 2STACK(n) = 22STACK(n-1)

2STACK(1) = 2 2STACK(1) = 2 log*(2) = 1 2STACK(n) = 2STACK(2) = 4 2 2STACK(2) = 4 log*(4) = 2 2: : : 2STACK(3) = 16 22 2STACK(3) = 16 log*(16) = 3 2STACK(4) = 65536 2STACK(4) = 65536 log*(65536) = 4 2 “tower of n 2’s” 2STACK(5) " 1080 2STACK(5) " 1080 log*() = 5 = atoms in = atoms in universe log*(n) = # of times you have to apply the log function to n to make it ! 1 Ackermann’s Function

A(0, n) = n + 1 for n " 0 A(m, 0) = A(m - 1, 1)$for m " 1 So an algorithm that can be shown to run in O(n log*n) Time is Linear Time A(m, n) = A(m - 1, A(m, n - 1))$for m, n " 1 for all practical purposes!! n=0 1 2 3 4 5 m=0 1 2 3 4 5

Ackermann’s Function Ackermann’s Function

A(0, n) = n + 1 for n " 0 A(0, n) = n + 1 for n " 0 A(m, 0) = A(m - 1, 1)$for m " 1 A(m, 0) = A(m - 1, 1)$for m " 1 A(m, n) = A(m - 1, A(m, n - 1))$for m, n " 1 A(m, n) = A(m - 1, A(m, n - 1))$for m, n " 1

A(4,2) > # of particles in universe

A(5,2) can’t be written out as decimal in this universe Ackermann’s Function

A(0, n) = n + 1 for n " 0 The inverse – in A(m, 0) = A(m - 1, 1)$for m " 1 fact, "(n #(n)) arises in the seminal A(m, n) = A(m - 1, A(m, n - 1))$for m, n " 1 paper of:

D. D. Sleator and R. E. Tarjan. A data structure for dynamic trees. Journal of Define: A’(k) = A(k,k) Computer and System Sciences, 26(3): 362-391, 1983. Inverse Ackerman #(n) is the inverse of A’ Practically speaking: n % #(n) ! 4n

• How is “time” measured • Definitions of: • O, !, " • linear, quadratic time, etc • log*(n) Here’s What • Ackerman Function You Need to Know…