Analysis of Algorithms

Home , Analysis of algorithms

CS 440 Theory of Algorithms / CS 468 Algor ithms in Bio in forma tics

Fundamentals of Analysis of Alggyorithm Efficiency

Issues: • Correctness • Time efficiency • Space e ffic iency • Optimality

Approaches: • Theoretical analysis • Empirical analysis

Time efficiency is analyzed by determining the number of repetitions of the basic operation as a function of input size

Basic operation: the operation that contributes most towards the running time of the algorithm.

input size

T(n) copC(n)

running time execution time Number of times for basic operation basic operation is executed

Input Size and Basic Operation Examples

Problem Input size measure Basic operation

Search for key in list of Number of items in list, n Key comparison n items Multiply two matrices of floating point numbers

Compute an

Checking primality of a given integer n

Graph problem

Select a specific (typical) sample of inputs

Use physical unit of time (e.g., milliseconds) OR Count actual number of basic operations

Analyypze the empirical data

Best-Case, Average-Case, Worst-Case

For some algorithms efficiency depends on form of input:

Worst case: Cworst(n) – maximum over inputs of size n

Best case: Cbest(n) – minimum over inputs of size n

Average case: Cavg(n) – “average” over inputs of size n • Number of times the basic operation will be executed on typical input • NOT the average of worst and best case • EtdbfbiExpected number of basic opera tiiddtions considered as a ran dom variable under some assumption about the probability distribution of all possible inputs

Problem: Given a list of n elements and a search key K, find an element equal to K, if any. Algorithm: Scan the list and compare its successive elements with K untiiil either a matchi ng element i ifs found ( successflful search) of the list is exhausted (unsuccessful search)

Example: Sequential Search

Worst case • What is the worst case / what are the worst cases?

• #bas ic opera tions, i .e., compari sons = ? Best case • What is the best case / what are the best cases?

• #basic operations = ? Average case • How many cases are there? • Assume each case is equally likely to occur, #basic operations = ?

Exact formula e.g., C(n) = n(n-1)/2

Formula indicating order of growth with specific multiplicative constant e.g., C(n) 0.5 n2

Formula indicating order of growth with unknown multiplicative constant e.g., C(n) cn2

Order of Growth

Most important: Order of growth within a constant multiple as n

Example: • How much faster will algorithm run on computer that is twice as fast?

• How much longer does it take to solve problem of double input size?

See table 2.1

Asymptotic Growth Rate

A way of comparing functions that ignores constant factors and small input sizes

O(g(n)): class of functions f(n) that grow no faster than g(n)

(g(n)): class of functions f(n) that grow at same rate as g(n)

(g(n)): class of functions f(n) that grow at least as fast as g(n) see figures 2.1, 2.2, 2.3

Big-Omega

Establishing Order of Growth Using the Definition

Definition: f(n) is in O(g(n)) if order of growth of f(n) order of growth of g(n) (within constant multiple), i.e., there exist positive constant c and non-negative integer

n0 such that

f(n) c g(n) for every n n0

It’s a game of finding Examples: c n the right and 0! 10n iO(is O(n2)

5n+20 is O(n)

Definition: f(n) is in (g(n)) if order of growth of f(n) order of growth of g(n) (within constant multiple), i.e., there exist positive constant c and non-negative integer

n0 such that

f(n) c g(n) for every n n0

It’s a game of finding Examples: c n the right and 0! 10n2 is (n)

5n+20 is (n)

Establishing Order of Growth Using the Definition

Definition: f(n) is in (g(n)) if order of growth of f(n) = order of growth of g(n) (within constant multiple), i.e., f(n) + O(g(n)) and f(n) + (g(n))

Examples: It’s a game of finding 10n2 is (n2) c n the right and 0!

5n+20 is (n)

f(n) + (g(n)) iff f(n) + O(g(n)) and f(n) + (g(n)) Theorem: f(n) + O(f(n)) Theorem: f(n) + O(g(n)) iff g(n) + (f(n)) Theorem: If f (n) + O(g (n)) and g(n) + O(h(n)) , then f(n) + O(h(n)) (Note similarity with a b)

+ + Theorem: If f1(n) O(g1(n)) and f2(n) O(g2(n)) , then + f1(n) + f2(n) O(max{g1(n), g2(n)})

Establishing Order of Growth Using Limits

0,,g order of growth of T(n) __ order of growth of g(n)

c > 0, order of growth of T(n) __ order of growth of g(n) limn T(n)/g(n) =

, order of growth of T(n) ___ order of growth of g(n)

Examples: •10n vs. 2n2

• n(n+1)/2 vs. n2

•logb n vs. logc n

0, T(n) = O(g(n)) but not (g(n)), i.e., o(g(n))

c > 0, T(n) = (g(n)) limn T(n)/g(n) =

, T(n) = (g(n)) but not (g(n)), i.e., (g(n))

Examples: •10n vs. 2n2

• n(n+1)/2 vs. n2

•logb n vs. logc n

L’Hôpital’s Rule and Stirling’s Formula

L’Hôpital’s rule: If limn f(n) = limn g(n) = and the derivatives f´, g´exist, then

f(n) f ´(n) lim = lim n g(n) n g ´(n) Example: log n vs. n

Stirling’s formula: n! (2n)1/2 (n/e)n Example: 2n vs. n!

All logarithmic functions loga n belong to the same class (log n) no matter w hat the logarithm’s base a >1is> 1 is

All polynomials of the same degree k belong to the same class: k k-1 + k akn + ak-1n + … + a0 (n )

Exponential functions an have different orders of growth for different a’s: order an < order bn , where a < b

order log n < order n ( >0) < order an < order n! < order nn

Basic Asymptotic Efficiency Classes

1 constant log n logarithmic n linear n log nn log n n2 quadratic n3 cubic 2n exponential n! factorial

General Plan for Analysis

Decide on parameter n indicating input size

Identify algorithm’s basic operation

Determine worst, average, and best cases for input of size n

Set up a sum for the number of times the basic operation is executed

Simplify the sum using standard formulas and rules (see Appendix A)

Useful Summation Formulas and Rules

l i u1 = 1+1+…+1 = u - l + 1 + In particular, l i u1 = n - 1 + 1 = n (n)

2 + 2 1 i n i = 1+2+…+n = n(n+1)/2 n /2 (n )

2 2 2 2 3 + 3 1 i n i = 1 +2 +…+n = n(n+1)(2n+1)/6 n /3 (n )

i n n+1 0 i n a = 1 + a +…+ a = (a -1)/(a - 1) for any a 1 i 0 1 n n+1 + n In particular, 0 i n 2 = 2 + 2 +…+ 2 = 2 -1 (2 )

(ai ± bi ) = ai ± bi cai = c ai l i uai = l i mai + m+1 i uai

Maximum element

Element uniqueness problem

Matrix multiplication

Countinggyg binary digits

Myyygstery Algorithm

Example 1: Maximum Element

Example 3: Matrix Multiplication

It cannot be investigated the way the previous examples are. (Why?)

Example 5: Mystery Algorithm for i := 1 to n -1 do max := i ; for j := i + 1 to n do if |A[ j, i ]| > |A[ max, i ]| then max := j ; for k := i to n + 1 do swap A[ i, k ] with A[ max, k ]; for j := i + 1to1 to n do for k := n + 1 downto i do A[ j, k ]:] := A[ j, k ] - A[ i, k ] * A[ j, i ]/A[] / A[ i, i ];] ;

Decide on a parameter indicating an input’s size.

Identify the algorithm’s basic operation.

Check whether the number of times the basic op. is executed may vary on different inputs of the same size. (If it may, the worst, average, and best cases must be investigated separately.)

Set up a recurrence relation with an appropriate initial condition expressing the number of times the basic op. is executed.

Example 1: Recursive Evaluation of n!

Definition: n ! = 1 2 … (n-1) n for n 1 and 0! = 1

Recursive definition of n!: F(n) = F(n-1) n for n 1 and F(0) = 1

Trace the algorithm to set up recurrence!

Size: Basic operation: Recurrence relation:

M(n))( = M(n-1)),() + 1, M(0) = 0

Example 2: The Tower of Hanoi Puzzle

1 3

Recurrence for number of moves:

M(n))( = 2M(n-1)),() + 1, M(1) = 1

Tree of Calls for the Tower of Hanoi Puzzle

n-1 n-1

n-2 n-2 n-2 n-2 ...... 222 2

111111 11

Important Recurrence Types:

One (constant) operation reduces problem size by one. T(n)=T() = T(n-1) + c T(1) = d Solution: T(n) = (n-1)c + d linear

A pass through input reduces problem size by one. T(n) = T(n-1) + cn T(1) = d Solution: T(n) = [n(n+1)/2 – 1] c + d quadratic

One (constant) operation reduces problem size by half. T(n) = T(n/2) + c T(1) = d Solution: T(n) = c lg n + d logarithmic

A ppgppass through input reduces problem size b y,y half, and both halves are needed. T(n) = 2T(n/2) + cn T(1) = d Solution: T(n) = cn lg n + d n n log n

The Fibonacci sequence: 011230, 1, 1, 2, 3, 5, 8 81321, 13, 21, …

Fibonacci recurrence: F(n) = F(n-1) + F(n-2) F(0) = 0 F(1) = 1

General 2nd order linear homogeneous recurrence with constant coefficients (LHRCCs): aX(n) + bX(n-1) + cX(n-2) = 0

Another example: A(n))( = 3A(n-1) -2(n-2))()() A(0) = 1 A(1) = 3

Solving aX(n) + bX(n-1) + cX(n-2) = 0

Set up the characteristic equation (quadratic) ar2 + br + c=0

Solve to obtain roots r1 and r2

General solution to the recurrence n n if r1 and r2 are two distinct real roots: X(n) = r1 + r2 n n if r1 = r2 = r are two equal real roots: X(n) = r + nr

Particular solution can be found by using initial conditions

A(n))( = 3A(n-1) -2(n-2), where A ()(0) = 1 A ()(1) = 3

Characteristic eqq(q)uation (quadratic):

Solve to obtain roots r1 and r2:

n n General solution to RR: linear combination of r1 and r2

Particular solution using initial conditions:

Application to the Fibonacci Numbers

F(n) = F(n-1) + F(n-2) or F(n) - F(n-1) - F(n-2) = 0

Characteristic equation:

Roots of the characteristic equation:

General solution to the recurrence:

Particular solution for F(0) =0, F(1)=1:

1. Definition based recursive algorithm

2. Nonrecursive definition-based algorithm

3. Explicit formula algorithm

1 n 1 5 F(n) round to the nearest integer, where , the gloden ratio 5 2 4. Logarithmic algorithm based on formula: n F(n-1) F(n) 0 1 = F(n) F(n+1) 1 1 for n11, assuminggypgp an efficient way of computing matrix powers