Recursive Algorithms

A recursive algorithm is one in which objects are defined in terms Recurrences of other objects of the same type.

Computer Science & Engineering 235: Discrete Mathematics Advantages:

I Simplicity of code Christopher M. Bourke I Easy to understand [email protected] Disadvantages:

I Memory

I Speed

I Possibly redundant work

Tail recursion offers a solution to the memory problem, but really, do we need recursion?

Recursive Algorithms Motivating Example Analysis Factorial Recall the factorial function. 1 if n = 1 n! = n (n 1)! if n > 1  · − Consider the following (recursive) algorithm for computing n!: We’ve already seen how to analyze the running time of algorithms. However, to analyze recursive algorithms, we require more Algorithm (Factorial) sophisticated techniques.

Input : n N Specifically, we study how to define & solve recurrence relations. ∈ Output : n! 1 if n = 1 then 2 return 1 3 end 4 else 5 return Factorial(n 1) n − × 6 end

Motivating Example Recurrence RelationsI Factorial - Analysis? Definition

How many multiplications M(n) does Factorial perform? Definition I When n = 1 we don’t perform any. A recurrence relation for a sequence a is an equation that I Otherwise we perform 1. { n} expresses an in terms of one or more of the previous terms in the I Plus how ever many multiplications we perform in the sequence, recursive call, Factorial(n 1). − a0, a1, . . . , an 1 − I This can be expressed as a formula (similar to the definition of for all integers n n where n is a nonnegative integer. n!. ≥ 0 0 M(0) = 0 A sequence is called a solution of a recurrence relation if its terms M(n) = 1 + M(n 1) − satisfy the recurrence relation. I This is known as a recurrence relation. Recurrence RelationsII Recurrence RelationsIII Definition Definition

Example More generally, recurrences can have the form The Fibonacci numbers are defined by the recurrence,

F (n) = F (n 1) + F (n 2) T (n) = αT (n β) + f(n),T (δ) = c − − − F (1) = 1 F (0) = 1 or n T (n) = αT + f(n),T (δ) = c β The solution to the Fibonacci recurrence is   n n Note that it may be necessary to define several T (δ), initial 1 1 + √5 1 1 √5 fn = − conditions. √5 2 ! − √5 2 !

(your book derives this solution).

Recurrence RelationsIV Recurrence RelationsV Definition Definition

The initial conditions specify the value of the first few necessary Recurrence relations have two parts: recursive terms and terms in the sequence. In the Fibonacci numbers we needed two non-recursive terms. initial conditions, F (0) = F (1) = 1 since F (n) was defined by the two previous terms in the sequence. T (n) = 2T (n 2) + n2 10 − − Initial conditions are also known as boundary conditions (as recursive non-recrusive opposed to the general conditions). | {z } | {z } From now on, we will use the subscript notation, so the Fibonacci Recursive terms come from when an algorithm calls itself. numbers are Non-recursive terms correspond to the “non-recursive” cost of the fn = fn 1 + fn 2 − − algorithm—work the algorithm performs within a function. f1 = 1 f0 = 1 We’ll see some examples later. First, we need to know how to solve recurrences.

Solving Recurrences Linear Homogeneous Recurrences

There are several methods for solving recurrences. Definition

I Characteristic Equations A linear homogeneous recurrence relation of degree k with I Forward Substitution constant coefficients is a recurrence relation of the form I Backward Substitution an = c1an 1 + c2an 2 + + ckan k I Recurrence Trees − − ··· −

I Maple! with c1, . . . , ck R, ck = 0. ∈ 6 Linear Homogeneous Recurrences Solving Linear Homogeneous RecurrencesI Examples

n Examples We want a solution of the form an = r where r is some (real) constant.

n The Fibonacci sequence is a linear homogeneous recurrence We observe that an = r is a solution to a linear homogeneous relation. As are the following. recurrence if and only if

n n 1 n 2 n k an = 4an 1 + 5an 2 + 7an 3 r = c r − + c r − + + c r − − − − 1 2 ··· k an = 2an 2 + 4an 4 + 8an 8 − − − n k We can now divide both sides by r − , collect terms, and we get a How many initial conditions do we need to specify for these? As k-degree polynomial. many as the degree, k = 3, 8 respectively. k k 1 k 2 r c1r − c2r − ck 1r ck = 0 So, how do we solve linear homogeneous recurrences? − − − · · · − − −

Solving Linear Homogeneous RecurrencesII Second Order Linear Homogeneous Recurrences

A second order linear homogeneous recurrence is a recurrence of the form an = c1an 1 + c2an 2 − −

k k 1 k 2 r c1r − c2r − ck 1r ck = 0 − − − · · · − − − Theorem (Theorem 1, p414)

This is called the characteristic equation of the recurrence relation. 2 Let c1, c2 R and suppose that r c1r c2 = 0 is the ∈ − − The roots of this polynomial are called the characteristic roots of characteristic polynomial of a 2nd order linear homogeneous 1 the recurrence relation. They can be used to find solutions (if they recurrence which has two distinct roots, r1, r2. exist) to the recurrence relation. We will consider several cases. Then a is a solution if and only if { n} n n an = α1r1 + α2r2

for n = 0, 1, 2,... where α1, α2 are constants dependent upon the initial conditions. 1we discuss how to handle this situation later.

Second Order Linear Homogeneous Recurrences Second Order Linear Homogeneous Recurrences Example Example Continued

Example I Using the 2nd-order theorem, we have a solution,

Find a solution to n n an = α1(2 ) + α2(3 ) an = 5an 1 6an 2 − − − I Now we can plug in the two initial conditions to get a system with initial conditions a0 = 1, a1 = 4 of linear equations.

0 0 I The characteristic polynomial is a0 = α1(2) + α2(3) 1 1 a1 = α1(2) + α2(3) r2 5r + 6 −

I Using the quadratic formula (or common sense), the root can 1 = α1 + α2 (1) be found; 4 = 2α1 + 3α2 (2) r2 5r + 6 = (r 2)(r 3) − − − so r1 = 2, r2 = 3 Second Order Linear Homogeneous Recurrences Second Order Linear Homogeneous Recurrences Example Continued Another Example

I Solving for α = (1 α ) in (1), we can plug it into the 1 − 2 second. Example 4 = 2α1 + 3α2 4 = 2(1 α2) + 3α2 Solve the recurrence − 4 = 2 2α + 3α − 2 2 2 = α2 an = 2an 1 + 15an 2 − − − I Substituting back into (1), we get with initial conditions a0 = 0, a1 = 1.

α1 = 1 − If we did it right, we have I Putting it all back together, we have 1 n 1 n an = (3) ( 5) n n 8 8 an = α1(2 ) + α2(3 ) − − = 1 2n + 2 3n − · · How can we check ourselves?

Single Root Case Single Root Case Example Recall that we can only apply the first theorem if the roots are Example distinct, i.e. r = r . 1 6 2 What is the solution to the recurrence relation If the roots are not distinct (r1 = r2), we say that one characteristic root has multiplicity two. In this case we have to an = 8an 1 16an 2 apply a different theorem. − − −

with initial conditions a0 = 1, a1 = 7? Theorem (Theorem 2, p416)

2 I The characteristic polynomial is Let c1, c2 R with c2 = 0. Suppose that r c1r c2 = 0 has ∈ 6 − − only one distinct root, r . Then a is a solution to 0 { n} r2 8r + 16 an = c1an 1 + c2an 2 if and only if − − − n n I Factoring gives us an = α1r0 + α2nr0 r2 8r + 16 = (r 4)(r 4) for n = 0, 1, 2,... where α1, α2 are constants depending upon the − − −

initial conditions. so r0 = 4

Single Root Case General Linear Homogeneous Recurrences Example

I By Theorem 2, we have that the solution is of the form

n n There is a straightforward generalization of these cases to higher an = α14 + α2n4 order linear homogeneous recurrences. I Using the initial conditions, we get a system of equations; Essentially, we simply define higher degree polynomials. a0 = 1 = α1 The roots of these polynomials lead to a general solution. a1 = 7 = 4α1 + 4α2 The general solution contains coefficients that depend only on the 3 initial conditions. I Solving the second, we get that α2 = 4 I And so the solution is In the general case, however, the coefficients form a system of 3 linear inequalities. a = 4n + n4n n 4

I We should check ourselves... General Linear Homogeneous RecurrencesI General Linear Homogeneous Recurrences Distinct Roots Any Multiplicity

Theorem (Theorem 3, p417)

Let c1, . . . , ck R. Suppose that the characteristic equation ∈ k k 1 Theorem (Theorem 4, p418) r c1r − ck 1r ck = 0 − − · · · − − − has k distinct roots, r , . . . , r . Then a sequence a is a solution Let c1, . . . , ck R. Suppose that the characteristic equation 1 k { n} ∈ of the recurrence relation k k 1 r c1r − ck 1r ck = 0 − − · · · − − − an = c1an 1 + c2an 2 + + ckan k − − ··· − has t distinct roots, r1, . . . , rt with multiplicities m1, . . . , mt. if and only if

a = α rn + α rn + + α rn n 1 1 2 2 ··· k k

for n = 0, 1, 2,..., where α1, α2, . . . , αk are constants.

General Linear Homogeneous Recurrences Linear Nonhomogeneous Recurrences Any Multiplicity

Theorem (Continued)

Then a sequence an is a solution of the recurrence relation For recursive algorithms, cost functions are often not homogenous { } because there is usually a non-recursive cost depending on the an = c1an 1 + c2an 2 + + ckan k input size. − − ··· − if and only if Such a recurrence relation is called a linear nonhomogeneous recurrence relation. m1 1 n an = (α1,0 + α1,1n + + α1,m1 1n − )r1 + ··· − m2 1 n Such functions are of the form (α2,0 + α2,1n + + α2,m2 1n − )r2 + ··· − . . an = c1an 1 + c2an 2 + + ckan k + f(n) − − − mt 1 n ··· (αt,0 + αt,1n + + αt,mt 1n − )rt + ··· − for n = 0, 1, 2,..., where α are constants for 1 i t and i,j ≤ ≤ 0 j m 1. ≤ ≤ i −

Linear Nonhomogeneous Recurrences Linear Nonhomogeneous Recurrences

Theorem (Theorem 5, p420) Here, f(n) represents a non-recursive cost. If we chop it off, we (p) are left with If an is a particular solution of the nonhomogeneous linear { } recurrence relation with constant coefficients an = c1an 1 + c2an 2 + + ckan k − − ··· − an = c1an 1 + c2an 2 + + ckan k + f(n) which is the associated homogenous recurrence relation. − − ··· − (p) (h) (h) then every solution is of the form an + an , where an is a Every solution of a linear nonhomogeneous recurrence relation is { } { } the sum of a particular solution and a solution to the associated solution of the associated homogenous recurrence relation linear homogeneous recurrence relation. an = c1an 1 + c2an 2 + + ckan k − − ··· − Linear Nonhomogeneous Recurrences Linear Nonhomogeneous Recurrences

Theorem (Theorem 6, p421)

Suppose that a satisfies the linear nonhomogeneous recurrence There is no general method for solving such relations. However, we { n} can solve them for special cases. relation

In particular, if f(n) is a polynomial or exponential function (or an = c1an 1 + c2an 2 + + ckan k + f(n) more precisely, when f(n) is the product of a polynomial and − − ··· − exponential function), then there is a general solution. where c1, . . . , ck R and ∈ t t 1 n f(n) = (btn + bt 1n − + + b1n + b0) s − ··· ·

where b0, . . . , bn, s R. ∈

Linear Nonhomogeneous Recurrences Linear Nonhomogeneous RecurrencesI

Theorem (Continued) Examples (Rosen): an = 3an 1 + 2n When s is not a root of the characteristic equation of the − associated linear homogeneous recurrence relation, there is a particular solution of the form (h) n I Homogenous solution form: an = α3 t t 1 n (ptn + pt 1n − + + p1n + p0) s I Hetrogeneous form: p = cn + d since f(n) = 2n is a linear − ··· · n function

When s is a root of this characteristic equation and its multiplicity I Test: cn + d = 3(c(n 1) + d) + 2n is m, there is a particular solution of the form − I Collect terms, factor out a negation: (2c + 2)n + (2d 3c) = 0 − m t t 1 n (h) (p) n n (ptn + pt 1n − + + p1n + p0) s I Total solution: an = an + an = α3 + ( n 3/2) − ··· · − −

Linear Nonhomogeneous RecurrencesII Linear Nonhomogeneous RecurrencesI Application Recall that the number of additions performed by the recursive fibonacci code can be characterized as: n an = 5an 1 6an 2 + 7 − − − A(n) = A(n 1) + A(n 2) + 1 − − We now recognize this as a linear, nonhomogeneous recurrence (h) n n I Homogenous solution form: an = α12 + α23 relation of degree 2. n n I Hetrogeneous form: p = c 7 since f(n) = 7 is an n · Solving the homogenous part we get: exponential funciton n n 1 n 2 n n 2 n n I c7 = 5c7 6c7 + 7 (divide by 7 and solve for c) a = α ( .61803) + α (1.61803) − − − − n 1 − 2 (h) (p) n n 49 n I Total solution: a = an + an = α 2 + α 3 + 7 n 1 2 20 · Note that φ = 1.61803 ... is the golden ratio. To solve the non-homogeneous part (a constant function), we assume it has a solution of the form an = c and so:

c = c + c + 1 Linear Nonhomogeneous RecurrencesII Other Methods Application

When analyzing algorithms, linear homogenous recurrences of order greater than 2 hardly ever arise in practice. thus c = 1; altogether: − We briefly describe two “unfolding” methods that work for a lot of a = a(h) + a(p) = α ( φ + 1)n + α (φ)n 1 cases. n n n 1 − 2 − Backward substitution – this works exactly as its name implies: For A(0) = A(1) = 0 we get the regular solution to the fibonacci starting from the equation itself, work backwards, substituting recurrence minus 1: values of the function for previous ones. an = fn 1 − Recurrence Trees – just as powerful but perhaps more intuitive, which is exponential in n. this method involves mapping out the recurrence tree for an equation. Starting from the equation, you unfold each recursive call to the function and calculate the non-recursive cost at each level of the tree. You then find a general formula for each level and take a summation over all such levels.

Backward Substitution Backward Substitution Example Example – Continued Example If we continue to do this, we get the following.

Give a solution to T (n) = T (n 2) + 2(n 1) + 2n − − = T (n 3) + 2(n 2) + 2(n 1) + 2n T (n) = T (n 1) + 2n − − − − = T (n 4) + 2(n 3) + 2(n 2) + 2(n 1) + 2n . − − − − where T (1) = 5. . i 1 = T (n i) + − 2(n j) − j=0 − We begin by unfolding the recursion by a simple substitution of the P function values. Observe that I.e. this is the function’s value at the i-th iteration. Solving the sum, we get T (n 1) = T ((n 1) 1) + 2(n 1) = T (n 2) + 2(n 1) − − − − − − (i 1)(i 1 + 1) T (n) = T (n i) + 2n(i 1) + 2 − − + 2n − − 2 Substituting this into the original equation gives us T (n) = T (n 2) + 2(n 1) + 2n − −

Backward Substitution Recurrence Trees Example – Continued

When using recurrence trees, we graphically represent the We want to get rid of the recursive term. To do this, we need to recursion. know at what iteration we reach our base case; i.e. for what value Each node in the tree is an instance of the function. As we of i can we use the initial condition, T (1) = 5? progress downward, the size of the input decreases. We can easily see that when i = n 1, we get the base case. − The contribution of each level to the function is equivalent to the Substituting this into the equation above, we get number of nodes at that level times the non-recursive cost on the size of the input at that level. T (n) = T (n i) + 2n(i 1) i2 + i + 2n − − − The tree ends at the depth at which we reach the base case. = T (1) + 2n(n 1 1) (n 1)2 + (n 1) + 2n − − − − − = 5 + 2n(n 2) (n2 2n + 1) + (n 1) + 2n As an example, we consider a recursive function of the form − − − − = n2 + n + 3 n T (n) = αT + f(n),T (δ) = c β   Recurrence Trees Recurrence Trees Example

Iteration Cost The total value of the function is the summation over all levels of 0 T (n) f(n) the tree: logβ n i n T (n) = α f i 1 T (n/β) α T (n/β) α f n · β ··· ··· · β i=0     X We consider the following concrete example. 2 T (n/β2) α T (n/β2) T (n/β2) α T (n/β2) α2 f n ··· ··· ··· ··· · β2   ...... Example

i αi f n · βi   . . . . n . . T (n) = 2T + n, T (1) = 4 log n 2 logβ n α β T (δ) ·  

Recurrence Trees Recurrence Trees Example – Continued Example – Continued

Iteration Cost 0 T (n) n

n + n The value of the function then, is the summation of the value of 1 T (n/2) T (n/2) 2 2 all levels. We treat the last level as a special case since its non-recursive cost is different. 2 T (n/4) T (n/4) T (n/4) T (n/4) 4 n ·  4  (log n) 1 2 − i n 3 T (n/8) T (n/8) T (n/8) T (n/8) T (n/8) T (n/8) T (n/8) T (n/8) 8 n T (n) = 4n + 2 = n(log n) + 4n ·  8  2i i=0 . . . . X . .

i 2i n ·  2i    ......

log2 n 2log2 n T (1) ·

Smoothness RuleI Smoothness RuleII

In the previous example we make the following assumption: that n was a power of two; n = 2k. This was necessary to get a nice depth of log n and a full tree. Theorem We can restrict consideration to certain powers because of the smoothness rule. For a smooth function f(n) and a fixed constant b Z such that ∈ b 2, ≥ Definition f(bn) Θ(f(n)) ∈ A function f : N R is called smooth if it is monotonically 7→ nondecreasing and Thus the order of growth is preserved. f(2n) Θ(f(n)) ∈

Most “slow” growing functions (logarithmic, polylogarithmic, polynomial) are smooth while exponential functions are not. How To Cheat With MapleI How To Cheat With MapleII

> rsolve( T(n) = T(n-1) + 2*n, T(1) = 5 , T(n)); { } Maple and other math tools are great resources. However, they are 1 not substitutes for knowing how to solve recurrences yourself. 1 + 2(n + 1) n + 1 2n 2 −   As such, you should only use Maple to check your answers. Recurrence relations can be solved using the rsolve command You can clean up Maple’s answer a bit by encapsulating it in the and giving Maple the proper parameters. simplify command: The arguments are essentially a comma-delimited list of equations: > simplify(rsolve( T(n) = T(n-1) + 2*n, T(1) = 5 , { } general and boundary conditions, followed by the “name” and T(n))); variable of the function. 3 + n2 + n