Analysis of Algorithms Recurrence Relations Acknowledgements • The first 12 slides are from the course taught by Prof. Yun Peng at UMBC. – http://www.csee.umbc.edu/~ypeng/S03203/lecture-notes/Ch06.ppt Recursive Algorithms: Analysis • We have already learnt to analyze the running time of iterative algorithms by considering the number of instructions executed in the Word RAM Model. • In case of Recursive algorithms, it is not easy to compute the number of instructions by looking at code. – The number of instructions in one instance of function call depends on the number of instructions executed when recursive calls are made. – Hence to analyze recursive algorithms, we require more sophisticated techniques • Specifically, we need to solve recurrence relations Recurrence Relations • A recurrence relation for the sequence {an} is an equation that expresses an in terms of one or more of the previous terms of the sequence, namely, a0, a1, …, an-1, for all integers n with n n0, where n0 is a nonnegative integer. • A sequence is called a solution of a recurrence relation if its terms satisfy the recurrence relation. Recurrence Relations • In other words, a recurrence relation is like a recursively defined sequence, but without specifying any initial values (initial conditions). • Therefore, the same recurrence relation can have (and usually has) multiple solutions. • If both the initial conditions and the recurrence relation are specified, then the sequence is uniquely determined. Recurrence Relations • Example: Consider the recurrence relation an = 2an-1 – an-2 for n = 2, 3, 4, … • Is the sequence {an} with an=3n a solution of this recurrence relation? • For n 2 we see that 2an-1 – an-2 = 2(3(n – 1)) – 3(n – 2) = 3n = an. • Therefore, {an} with an=3n is a solution of the recurrence relation. • Is the sequence {an} with an=5 a solution of the same recurrence relation? • For n 2 we see that 2an-1 – an-2 = 2*5 - 5 = 5 = an. • Therefore, {an} with an=5 is also a solution of the recurrence relation. Modeling with Recurrence Relations • Example: • Someone deposits 10,000 in a savings account at a bank yielding 5% per year with interest compounded annually. How much money will be in the account after 30 years? • Solution: • Let Pn denote the amount in the account after n years. • How can we determine Pn on the basis of Pn-1? Modeling with Recurrence Relations • We can derive the following recurrence relation: • Pn = Pn-1 + 0.05Pn-1 = 1.05Pn-1. • The initial condition is P0 = 10,000. • Then we have: • P1 = 1.05P0 2 • P2 = 1.05P1 = (1.05) P0 3 • P3 = 1.05P2 = (1.05) P0 • … n • Pn = 1.05Pn-1 = (1.05) P0 • We now have a formula to calculate Pn for any natural number n and can avoid the iteration. Modeling with Recurrence Relations • Let us use this formula to find P30 under the initial condition P0 = 10,000: 30 • P30 = (1.05) * 10,000 = 43,219.42 • After 30 years, the account contains Rs. 43,219.42. Modeling with Recurrence Relations • Another example: • Let an denote the number of bit strings of length n that do not have two consecutive 0s (“valid strings”). Find a recurrence relation and give initial conditions for the sequence {an}. • Solution: • Idea: The number of valid strings equals the number of valid strings ending with a 0 plus the number of valid strings ending with a 1. Modeling with Recurrence Relations • Let us assume that n 3, so that the string contains at least 3 bits. • Let us further assume that we know the number an-1 of valid strings of length (n – 1). • Then how many valid strings of length n are there, if the string ends with a 1? – There are an-1 such strings, namely the set of valid strings of length (n – 1) with a 1 appended to them. • Note: Whenever we append a 1 to a valid string, that string remains valid. Modeling with Recurrence Relations • Now we need to know: How many valid strings of length n are there, if the string ends with a 0? • Valid strings of length n ending with a 0 must have a 1 as their (n – 1)th bit (otherwise they would end with 00 and would not be valid). • And what is the number of valid strings of length (n – 1) that end with a 1? – We already know that there are an-1 strings of length n that end with a 1. • Therefore, there are an-2 strings of length (n – 1) that end with a 1. Modeling with Recurrence Relations • So there are an-2 valid strings of length n that end with a 0 (all valid strings of length (n – 2) with 10 appended to them). • As we said before, the number of valid strings is the number of valid strings ending with a 0 plus the number of valid strings ending with a 1. • That gives us the following recurrence relation: an = an-1 + an-2 Modeling with Recurrence Relations • What are the initial conditions? • a1 = 2 (0 and 1) • a2 = 3 (01, 10, and 11) • a3 = a2 + a1 = 3 + 2 = 5 • a4 = a3 + a2 = 5 + 3 = 8 • a5 = a4 + a3 = 8 + 5 = 13 • … • This sequence satisfies the same recurrence relation as the Fibonacci sequence. • Since a1 = f3 and a2 = f4, we have an = fn+2. Example: Factorial • Recall the factorial function: 1 if n= 1 n! = n.(n-1) ! if n > 1 • Consider the following (recursive) algorithm for computing n! int Factorial (int n) if (n=1) or (n=0) return 1 else return n Factorial (n-1) Factorial: Analysis • How many multiplications T(x) does factorial function perform? • When n=1 or n = 0 we don’t perform any • Otherwise, we perform one explicitly in the code, and some number of multiplications are performed in the recursive call to Factorial (n-1) • The number of multiplications can be expressed as a formula (similar to the definition of n!) • T(0) = 0 • T(n) = 1 + T(n-1) • To know the time complexity of factorial function, we will have to solve this recurrence relation. Factorial complexity T(n) = T(n – 1) + 1, T(0) = 0 T(n) = T(n – 1 ) + 1 T(n – 1 ) = T(n – 2 ) + 1 = [T(n – 2 ) + 1 ] + 1 = T(n – 2 ) + 2 T(n – 2 ) = T(n – 3 ) + 1 = T(n – 3 ) + 3 .. = T(n – k ) + k To use Boundary condition let T(n – k ) = T(0) k = n T(n) = 0 + n = n T(n) = O(n) Recurrence for Linear Search • Problem: Given an array containing n elements, search if a value x is present in the array. • Linear Search: Examine each element of the array one by one. Total number of operations T(n) to handle n elements can be broken down in two parts. • One part is to examine the first element ( 1 operation), and the other part is to examine remaining elements which requires T(n – 1 ) operations. • We can write a recurrence relation for T(n) T(n) = T(n-1) + 1 Recurrence for Binary Search • Problem: Given an array containing n elements, search if a value x is present in the array. • Binary Search: Examine the middle element of the array ( 1 operation), and then examine either the left half or the right half of the array which requires T(n/2) operations. • Thus the recurrence relation for T(n) takes the form T(n) = T(n/2) + 1 Binary Search Complexity T(n) = T(n/2) + 1, T(1) = 1 T(n/2) = T(n/4) + 1 T(n) = T(n/2) + 1 T(n/4) = T(n/8) + 1 = [T(n/4) + 1 ] + 1 = [T(n/8) + 1 ] + 2 = T(n/8) + 3 = T(n/23) + 3 .. = T(n/2k) + k To use Boundary condition let us try to bring it to form T(1) Binary Search Complexity let T(n/2k) = T(1) n/2k = 1 2k = n Take log of both sides k log 2 = log n k = log n Substituting back in T(n) =T(n/2k) + k T(n) = T(1) + log n = 1 + log n = O(log n) Solving Recurrence Relations - Iteration method • Steps: ▪ Expand the recurrence ▪ Express the expansion as a summation by plugging the recurrence back into itself until you see a pattern. ▪ Evaluate the summation • In evaluating the summation one of the following formulae may be used: • Arithmetic series: •Special Cases of Geometric Series: • Geometric Series: Solving Recurrence Relations - Iteration method • Harmonic Series: • Others: Analysis of Towers of Hanoi Algorithm public static void hanoi(int n, char from, char to, char temp){ if (n == 1) Move (from, to); else{ hanoi(n - 1, from, temp, to); Move (from, to); hanoi(n - 1, temp, to, from); } } • The recurrence relation for the running time of the method hanoi is: T(n) = a if n = 1 T(n) = 2T(n - 1) + b if n > 1 Analysis of Towers of Hanoi Algorithm Expanding: T(n) = 2T(n – 1) + b = 2[2T(n – 2) + b] + b = 22 T(n – 2) + 2b + b = 22 [2T(n – 3) + b] + 2b + b = 23 T(n – 3) + 22b + 2b + b = 23 [2T(n – 4) + b] + 22b + 2b + b = 24 T(n – 4) + 23 b + 22b + 21b + 20b = …… = 2k T(n – k) + b[2k- 1 + 2k– 2 + . 21 + 20] When k = n – 1, we have: Therefore, The method hanoi is O(2n).