3. RECURSIVE ALGORITHMS AND RECURRENCE RELATIONS
In discussing the example of finding the determinant of a matrix an algorithm was outlined that defined det(M) for an nxn matrix in terms of the determinants of n matrices of size (n-1)x(n-1).
If D(n) is the work required to evaluate the determinant of an nxn matrix using this method then
D(n)=n . D(n-1)
To solve this -- in the sense of ending up with an expression for D(n) that does not have reference to other occurrences of the function D() on the right hand side -- we can use progressive substitutions as follows:
D(n)= n. D(n-1) = n. (n-1). D(n-2) = n. (n-1). (n-2). … x 3 x 2 x D(1)
a constant, the cost of returning a number
n!
⇒ D(n)∈O(n!)
COMP1004: Part 2.2 27 Now consider
ALGORITHM Factorial(n) // Recursively calculates n! for positive integer n if n=1 return 1 else return n*Factorial(n-1)
The only possible choice for the elementary operation here is multiplication, so we choose as a cost function
F(n) = number of multiplications to evaluate Factorial(n)
The algorithm as defined then implies that
F(1) = 0 F(n) = F(n-1) + 1, n>1
The general-n form can be rewritten as
F(n)-F(n-1)=1
Letting n → n-1 in the above, we also have
F(n-1)-F(n-2)=1
We can continue to do this until the row beginning with F(2), which contains the first mention of the base case (n-1 = 2-1 = 1).
F(n) - F(n-1) = 1 + F(n-1) – F(n-2) = 1 + F(n-2) – F(n-3) = 1 n-1 equations …. …. … + F(2) - F(1) = 1 F(n) - F(1) = n-1
('Method of differences’ or ‘ladder method'.)
COMP1004: Part 2.2 28 Adding up the rungs of the ladder there are cancellations on the left hand side for every term except F(n) and F(1), and for the right hand side the sum is just n-1.
→ F(n) = F(1) + n - 1 → F(n) = n-1, since F(1)=0
→ F(n) ∈ O(n).
Aside:
Is this reasonable, is evaluating n! really only in O(n)?
The analysis ignored the fact that large values will build up quickly and so it was not in this case ideal to count integer multiplications as elementary. However it was a simple example that showed the techniques that can also be used in much more complex cases.
* * *
Expressions like ‘D(n) = n D(n-1)’ ‘F(n) = F(n-1) +1’ are known as recurrence relations. A recurrence relation is a formula that allows us to compute the members of a sequence one after the other, given one or more starting values.
These examples are first order recurrence relations because a reference is made to only one smaller sized instance.
A first order recurrence requires only one ‘starting value’ -- in the first of these cases D(1), in the second F(1) -- in order to obtain a unique solution for general n.
Recurrence relations arise naturally in the analysis of recursive algorithms, where the starting values are the work required to compute base cases of the algorithm.
COMP1004: Part 2.2 29 Consider as another example of a first order recurrence relation f(0) = 1 (the base case is for n=0) f(n) = c. f(n-1) n > 0
(It doesn't matter here what hypothetical algorithm may have generated the definition of f(n). Most of the examples in this section will be starting directly from a recurrence relation and will be focused primarily on developing the mathematical tools for solving such relations.)
By inspection, f(n) = c. f(n-1) = c2. f(n-2) … = cn. f(0) = cn f(n)=cn is the solution of this recurrence relation.
It is purely a function of the input size variable n, it does not make reference on the right hand side to the original cost function applied to any other input sizes.
The expression
D(n)= n. D(n-1) is clearly also a first order recurrence of the same kind, but here the multiplying coefficient depends on n. This is the next step up in complication, where for a general n-dependent multiplier b(n) f(n) = b(n)xf(n-1) (1)
(defined for n > a with base case value f(a) given)
Again by inspection f(n) = b(n)xf(n-1) = b(n)xb(n-1)xf(n-2) = b(n)xb(n-1)x...xb(a+1)xf(a)
COMP1004: Part 2.2 30 (note it was not possible to go any further because there would then be a reference to f(a-1), which is not defined) giving as a general solution to (1)
$" n $& f(n) = #!b(i)'xf(a) %$i=a+1 ($
in the determinant example n a=1, f(1)=1, b(i)=i, giving f(n) = !b(i) = 2x3x...xn = n! i=a+1
(1) is the most general case of a first order (it makes reference to only one smaller sized instance), homogeneous (there is nothing on the r.h.s. that doesn't multiply a smaller sized instance, such as a polynomial in n) recurrence relation.
What about the factorial function recurrence
F(n) = F(n-1) + 1
? This is a simple example of a first order, inhomogeneous recurrence relation, for which the general form (with non-constant coefficients b(n), c(n)) is f(n) = b(n)xf(n-1) + c(n) (n > a, f(a) given) (2)
If we try to do this one by progressive substitution as for the determinant case we quickly get into difficulties:
f(n) = b(n)xb(n-1)x f(n-2) + b(n)xc(n-1) + c(n) = b(n)xb(n-1)xb(n-2)xf(n-3) + b(n)xb(n-1)xc(n-2) + b(n)xc(n-1)+ c(n) = ......
COMP1004: Part 2.2 31 In cases like this we need to define instead a new function g(n) by f(n) = b(a+1)xb(a+2)x…xb(n)x g(n) , n>a f(a) ≡ g(a) (the two functions agree on the base case)
Substituting in this way for f(n) in (2): f(n) = b(n)xf(n-1) + c(n) b(a+1)xb(a+2)x…xb(n)x g(n) = b(n)xb(a+1)xb(a+2)x… xb(n-1)x g(n-1) + c(n)
same coefficient
Then dividing by the product b(a+1)xb(a+2)x…xb(n), the common multiplier of both instances of the function g(), gives g(n) = g(n-1) + d(n) (3)
c(n) where d(n) = . b(a +1)xb(a + 2)x...xb(n)
(3) is now in a much simpler form (the important thing is that the multiplier of the smaller-input instance of the function on the r.h.s. is now 1) and can be solved by the ladder method we used in the case of F(n) (where d(n) =1, independent of n) to give the solution
n $" n $& f(n) = )b(i)#f(a) + ! d(j)' i=a+1 %$ j=a+1 ($
in the factorial example a=1, f(1)=0, b(i)=1, n n d(j) = 1, giving f(n) = n-1 (as !d(j) = !1= n "1) j=2 j=2
COMP1004: Part 2.2 32 It's possible, and not incorrect, to memorise and quote the above formula and substitute into it in order to solve a given recurrence relation, as the formula can be applied to inhomogeneous first order recurrences of any degree of complexity.
However it’s more instructive, and often easier, to consider inhomogeneous recurrences on a case-by-case basis, noting that in the examples you would be given the base case input ‘a’ will normally be 0 or 1, and the d-summation above -- which could in principle be hard and require an approximation technique -- will use just the series summation formulae you have already seen.
Example: f(0)=0 (base case a=0)
f(n)= 3f(n-1) + 1, n > 0
Change variable:
n f(n) = 3n g(n) (b(i) = 3 for all i, so !b(i) = 3n ) i=1 This gives
3n g(n) = 3. 3n-1 g(n-1) + 1
3n
→ g(n) = g(n-1) +1/3n
Now use the method of differences (ladder method) to obtain
n 1 g(n) = ∑ i i=1 3
COMP1004: Part 2.2 33 n In detail: g(n) – g(n-1) = 1/3
n-1 g(n-1) – g(n-2) = 1/3
......
1 g(1) – g(0) = 1/3
n 1 g(n) – g(0) = ∑ i i=1 3
(When you are familiar with these techniques you don’t need to show all the details.)
Multiplying both sides of the solution for g(n) by 3n gives the solution for f(n)
n 1 f(n) 3n = ∑ i i=1 3
We evaluate the sum using the formula for a geometric series:
n n 1 ⎡ i ⎤ i = a ∑ ⎢∑ ⎥ 1 i=1 3 ⎣ i=1 ⎦a= 3 ⎡a(1− an )⎤ = ⎢ ⎥ 1 ⎣ 1− a ⎦a= 3 1 ⎛ 1 ⎞ = ⎜1− n ⎟ 2 ⎝ 3 ⎠ Hence
n 3 " 1 % 1 n n f(n) = $1! ' = 3 !1 ( O(3 ) 2 # 3n & 2 ( )
COMP1004: Part 2.2 34 SOLVING HIGHER ORDER RECURRENCES USING THE CHARACTERISTIC EQUATION
This is a technique which can be applied to linear recurrences of arbitrary order but is less general than some of the techniques illustrated for first order recurrences in that we will henceforth assume that
• anything multiplying an instance(for some input size) of the function is a constant
• the recurrence relations are homogeneous -- they don’t contain any term that isn’t a multiple of an instance of the function we are trying to obtain a 'closed' solution for
Hence, we are here trying to solve kth order recurrences of the general form
ak f(n) + ak-1 f(n-1) +… + a0 f(n-k) = 0 where the {ai} are constants.
(Putting everything on the l.h.s. and equating it to zero rather than writing 'f(n) = ...' is a convenience, as will be seen later.)
Try the solution f(n) = α n (see Brassard and Bratley p.65 for a justification of this choice):
n n-1 n-k ak α + ak-1 α +… + a0 α = 0
αn-k can be seen to be a common factor:
n-k k k-1 α (ak α + ak-1 α +… + a0) = 0
characteristic equation
We assume that we are not interested in the trivial solution α=0, as this would imply f(n)=0 -- in an algorithmic context that the work needed for an input of any size n was zero -- only in the non-trivial solutions of the characteristic equation above.
COMP1004: Part 2.2 35
The characteristic equation is a polynomial of degree n and would in general be expected to have k distinct roots (the case that some of these may not be distinct will be considered later).
Assuming that the k roots ρ 1, ρ 2, ... ρ k of the characteristic n equation are all distinct, any linear combination of {ρi }
k n f(n) = ∑c iρi i=1
(Brassard and Bratley p.65) solves the homogeneous recurrence, where the {ci} are constants determined by the k initial conditions that define the base cases of the recurrence.
Example (k=2):
f(0) = 0 f(1) = 1 f(n) = 3f(n-1) – 2f(n-2), n > 1
Try a solution of the form f(n) = αn:
αn - 3αn-1 + 2αn-2 = 0
Divide by αn-2 to get the characteristic equation
α2 – 3α + 2 = 0 → (α - 1)(α - 2) = 0
The roots ρ1 = 1, ρ2 = 2 give a general solution of the form
n n f(n) = c1.1 + c2 .2
Using the initial conditions: c1 + c2 = 0 (n=0) → c1 = -1, c2 = 1 c1 + 2c2 = 1 (n=1)
→ f(n) = 2n – 1
COMP1004: Part 2.2 36 Example (k=3):
f(0) = 1 f(1) = 5 f(2) = 19 f(n) = 7f(n-1) – 14f(n-2) + 8f(n-3), n > 2
The characteristic equation is
α3 – 7α2 + 14α - 8 = 0 → (α - 1) (α - 2) (α - 4) = 0
Roots ρ1 = 1, ρ2 = 2, ρ3 = 4 give a general solution of the form
n n n f(n) = c1 .1 + c2 .2 + c3 .4
Using the initial conditions: c1 + c2 + c3 = 1 (n=0) → c1 = -1, c2 = 1, c3 = 1 c1 + 2c2 + 4c3 = 5 (n=1) c1 + 4c2 + 16c3 = 19 (n=2)
→ f(n) = 4n + 2n - 1 ∈ O( 4n )
(Note that under exam conditions you would not be expected to solve k > 2 examples. This is because it’s only for k=2 that there is a simple formula for finding the roots of the (then quadratic) characteristic equation; for k > 2 the roots might be considerably more difficult to find.)
COMP1004: Part 2.2 37 If the characteristic equation has a root ρ with multiplicity m, then (Brassard and Bratley p.67)
n n m-1 n xn = ρ , xn = n ρ , …, xn = n ρ are all possible solutions of the recurrence.
In this case the general solution is a linear combination (with k constant coefficients to be determined by the initial conditions) of these terms together with those contributed by the other roots of the characteristic equation.
Example (k=2):
f(0) = 1 f(1) = 5 f(n) = 2f(n-1) – f(n-2), n > 1
The characteristic equation is
α2 – 2α + 1 = 0 → (α - 1)2 = 0
The repeated root ρ = 1 gives a general solution of the form
n n f(n) = c1 .1 + c2 .n.1
Using the initial conditions: c1 = 1 (n=0) → c1 = 1, c2 = 4 c1 + c2 = 5 (n=1)
→ f(n) = 4n + 1
COMP1004: Part 2.2 38 Example (k=3):
f(0) = 2 f(1) = 9 f(2) = 29 f(n) = 5f(n-1) – 8f(n-2) + 4f(n-3), n > 2
The characteristic equation is
α3 – 5α2 + 8α - 4 = 0 → (α - 1) (α - 2)2 = 0
Roots ρ1 = 1 and ρ2 = 2 (m=2) give a general solution of the form
n n n f(n) = c1 .1 + c2 .2 + c3 .n.2
Using the initial conditions: c1 + c2 = 2 (n=0) → c1 = 1, c2 = 1, c3 = 3 c1 + 2c2 + 2c3 = 9 (n=1) c1 + 4c2 + 8c3 = 29 (n=2)
→ f(n) = 3n2n + 2n + 1 ∈ O( n2n )
COMP1004: Part 2.2 39 Example: Fibonacci numbers
The Fibonacci numbers are a famous sequence
0, 1, 1, 2, 3, 5, 8, 13, 21, ...
1 defined by the recurrence
Fib(n) = Fib(n-1) + Fib(n-2) with
Fib(0) = 0 Fib(1) = 1
Patterns incorporating this sequence appear to be widespread in nature, for example in the florets in the head of a sunflower or in the growth of pine cones.
The sequence was first introduced to Europe in a treatise on the growth of rabbit populations by Leonardo 2 Fibonacci in 1202 (though it may have been known in India for much longer).
The size of the Fibonacci numbers grows exponentially -- the 49th number is 7,778,742,049, more than the total number of people in the world.
1 Helianthus flower, L Shyamal 2006 2 Young cones of a Colorado spruce, http://www.noodlesnacks.com 2008
COMP1004: Part 2.2 40 We can show the exponential growth of the Fibonacci numbers by solving the recurrence relation for the series values Fib(n) using the characteristic equation method.
The general-n case is
Fib(n) – Fib(n-1) – Fib(n-2) = 0 with associated characteristic equation
α2 – α – 1 = 0
Unfortunately this doesn't factorise neatly with integer solutions like previous examples and it's necessary to use the formula for the roots of a quadratic equation to give the solutions
1± 5 ! = 1,2 2 and hence a general solution for the Fibonacci series values of the form n n Fib(n) = c1ρ1 + c2ρ2
Using the base case values Fib(0)=0, Fib(1)=1:
1 1 c1 + c2 = 0 (n=0) → c1 = , c2 = ! 5 5
c1 ρ1 + c2 ρ2 = 1 (n=1) and hence ! $n ! $n 1 1+ 5 1 1' 5 1 n !n Fib(n) # & # & = # & ' # & = (( ' ( ) 5 " 2 % 5 " 2 % 5 in which
! ! = (1+ 5) / 2 " 1.61803 , ! = "1/ ! # "0.61803
COMP1004: Part 2.2 41 It can easily be seen that
Fib(n) ! O("n)
! since as | ! | < 1 the contribution from the oscillating second part of the solution falls to zero very rapidly as n increases.
Rabbits really do breed like rabbits!
(Though Fibonacci assumed, among other things, that they were also immortal and hence his model of rabbit population growth would not nowadays be thought very realistic.)
But does this necessarily mean that computing the Fibonacci numbers will take an exponentially growing amount of time? Consider first a naïve procedure based directly on the definition:
ALGORITHM Fib1(n) // Calculates the nth Fibonacci number for integer n ≥ 0 if n ≤ 1 return n else return Fib1(n-1)+Fib1(n-2)
Taking addition here as the elementary operation the cost F(n) in terms of the number of additions can be seen to be
F(0) = 0 F(1) = 0 F(n) = F(n-1) + F(n-2) + 1
Unfortunately this is an inhomogeneous second order recurrence, and the characteristic equation method presented here doesn't extend to these cases.
(In fact it does fall within the slightly expanded range of functional forms considered in Brassard and Bratley (p.68) but we will not cover these more general methods of solution.)
COMP1004: Part 2.2 42 But there's a trick: notice that if we define G(n) = F(n)+1 then the recurrence relation for the number of additions becomes
G(0) = 1 G(1) = 1 G(n) = G(n-1) + G(n-2) which is homogeneous and so soluble using our usual methods.
Even better, it's not actually necessary to solve this recurrence as it can also be observed that G(n) = Fib(n+1) and hence that 1 ! F(n) = G(n) !1 = Fib(n+1) !1 = "n+1 ! "n+1 !1 5 ( )
So it is the case that F(n) ! O("n) -- the amount of work required by the definition-derived algorithm grows exponentially in the same way as the sequence values Fib(n) themselves.
However it's possible to do better.
It might have been predicted that Fib1 was not an efficient algorithm from inspection of its structure. Working through some examples for small n (such as the n=5 case on p.82 of Levitin) will convince you there is massive inefficiency in the recomputation of already-known values -- for example in the case of n=5 Fib1(4) is computed four times over.
There is a simple iterative alternative that takes only linear time as measured by the number of integer additions, storing already- known values in an array A[0..n]:
ALGORITHM Fib2(n) // Calculates the nth Fibonacci number for integer n ≥ 0 A[0] <− 0 A[1] <− 1 for i <− 2 to n do A[i] <− A[i-1]+A[i-2] return A[n]
COMP1004: Part 2.2 43 This clearly does only n-1 additions, so is in O(n).
BUT -- be careful -- though the second algorithm will certainly be preferable, considering additions in either case as having unit cost may not be realistic because the the size of the numbers the Fibonacci sequence generates.
It's the same situation as for the factorial function recurrence, found to be O(n) in terms of integer multiplications but where the integers in question would quickly become very large.
Realistically one would need to think about single bit operations, the size of an instance n then being given as 1 !log n#. +" 2 $
CHANGE OF VARIABLE
Consider the problem of raising a number, a, to some power n. The naïve algorithm for doing this is
ALGORITHM Exp1(a,n) // Computes an for positive integer n if n=1 return a else return a*Exp1(a,n-1)
Exp1 has the same general structure as the factorial function we looked at earlier and by a similar argument can be demonstrated also to be O(n) (counting multiplications at unit cost).
Can we improve on this simple exponentiation algorithm?
Consider the following sequence for computing a32 : a → a2 → a4 → a8 → a16 → a32
Each term is obtained by squaring the previous one, so only 5 multiplications are required, not 31.
COMP1004: Part 2.2 44
It's possible to base an alternate algorithm on this successive squaring idea:
ALGORITHM Exp2(a,n) // Computes an for positive integer n if n=1 return a else if even(n) return [Exp2(a,n/2)]2 else return a*[Exp2(a,(n-1)/2)]2 (divisions in each case give integer result)
Exp2(a,n) gets broken into smaller (Exp2) instances and then squared to give the required result.
Let E(n) be the work required to compute Exp2(a,n) and use multiplication as the unit-cost elementary operation:
E(1) = 0 squaring of smaller-input instance
E(n) = E(n/2) + 1 if n even E((n-1)/2) + 2 if n odd
squaring of smaller-input instance and multiplication of result by a
Suppose for the moment that n is not only even but is a power of 2. Make the change of variable
n → 2k
COMP1004: Part 2.2 45 Then the even-n recurrence takes the form
E(2k) = E(2k/2) + 1
= E(2k-1) + 1
k or more compactly, writing E(2 ) ≡ Ek
Ek – Ek-1 = 1
This can now easily be solved by the ladder method:
Ek – Ek-1 = 1 + Ek-1 – Ek-2 = 1 k equations + … … … + E1 - E0 = 1 Ek - E0 = k
So Ek = E0 + k
0 where E0 ≡ E(2 ) = E(1) = 0
→ E(2k) = k
Since k n = 2 ⇔ k = log2 n
E(n) = log2 n
Thus E(n) ∈ O(log n | n is a power of 2) (remember that bases of logs can be dropped under ‘O’ )
conditional asymptotic notation
COMP1004: Part 2.2 46 It can be shown (Brassard and Bratley, p.46) that f(n) ∈ O( g(n) | n is a power of 2 ) ⇒ f(n) ∈ O(g(n)) if
(a) g is eventually non-decreasing (b) g(2n) ∈ O( g(n) ) -- 'smoothness property' log n is an increasing function for all n, and since log(2n) = log 2 + log n ∈ O(log n) we can say finally that for all n
E(n) ∈ O(log n)
Exp2 is thus significantly more efficient, as n grows, than the naïve algorithm Exp1 -- though it is always advisable to check that the computational overheads in implementing a more sophisticated algorithm don’t outweigh the benefits for the size of input instances you are likely to encounter.
Example : f(1) = 1 f(n) = 4 f(n/2) + n2, n > 1
Change variables n → 2k to obtain the recurrence
k fk = 4fk-1 + 4
Make the further substitution (because of the multiplier ‘4’ in front of fk-1): k fk = 4 gk, f0 = g0
k k-1 k → 4 gk = 4. 4 gk-1 + 4
COMP1004: Part 2.2 47 Dividing by 4k and constructing the ladder
gk - gk-1 = 1 + gk-1 – gk-2 = 1 + … … … + g1 - g0 = 1 gk - g0 = k
k → gk = g0 + k (where k and n are related by n = 2 , k = log2 n)
Multiplying the solution for g by 4k:
k k fk = 4 f0 + 4 . k
2 2 0 k k 2 2 → f(n) = n f(1) + n log2n (f0 = f(2 ) = f(1); 4 = (2 ) = n )
f(n) ∈ O( n2 log n | n is a power of 2)
Since n2 log n is a non-decreasing function for all n > 1 (it has a turning point between 0 and 1), and
(2n)2 log(2n) = 4 n2(log2) + 4n2(log n)
∈ O(n2log n) it can be concluded that for all n
f(n) ∈ O(n2log n)
COMP1004: Part 2.2 48 Example: f(1)=1 f(2)=2 f(n)=4f(n/2) – 4f(n/4), n>2
Change variables n → 2k (noting that as n/2 → 2k-1, n/4 → 2k-2 ) to obtain the recurrence for the new variable k
fk = 4 fk-1 – 4 fk-2
This is a homogeneous 2nd order recurrence, and should be solved using the characteristic equation method.
The characteristic equation for the above recurrence (variable k) is
α2 – 4α + 4 = 0 → (α - 2)2 = 0
Repeated root ρ=2, so the general solution for k is
k k fk = c1 2 + c2 k 2
log2n
In terms of the original variable n, this general solution is
f(n) = c1 n + c2 n log2n
Is this O(n log n)? It depends on the base cases, which determine c1 and c2. f(1) = c1 + c2 log21 = c1 = 1
0 f(2) = 2c1 + 2 c2 log2 2 = 2(c1 + c2) = 2
1 →c1 = 1, c2 = 0
→f(n)=n O(n) not O(n log n)
COMP1004: Part 2.2 49 Strassen’s algorithm for multiplication
We now look for ways to improve the efficiency of multiplication of two n-bit numbers -- the methods we have seen so far are O(n2).
We can write the two n-bit numbers which are to be multiplied like this:
a b c d
n/2-bits n/2-bits × n/2-bits n/2-bits
( a . 2n/2 + b ) × ( c. 2n/2 + d )
= a. c. 2n + (a. d + b. c). 2n/2 + b. d
this is just this is putting putting n zeros n/2 zeros after after the value (a.d + b.c), so is O(n) a.c, so is O(n)
Base 10 (decimal) example for n=6 digits: 12345610 = 12300010 + 45610 3 = 12310×10 + 45610
a b
Let M(n) be the cost of multiplying two n-bit numbers using Strassen’s algorithm.
M(1) = a0 (the cost of multiplying 2 bits) M(n) = 4 M(n/2) + a1n + a2
all other arithmetic operations apart from the 4 n/2-bit multiplications -- note that addition of two n-bit numbers is O(n)
4 pairs of n/2-bit multiplications ( a.c, b.d, a.d and b.c )
COMP1004: Part 2.2 50 This recurrence is similar to the one on p.47 and we proceed similarly:
k k First, set n = 2 , M(2 ) = Mk to get
k Mk = 4 Mk-1 + a1 2 + a2
k Now set Mk = 4 Nk, M0 = N0 (this is just doing both necessary substitutions in one step, a shortened form of the method in the previous example):
k k-1 k 4 Nk = 4. 4 Nk-1 + a1 2 + a2
Divide by 4k and construct ladder:
k k Nk - Nk-1 = a1/2 + a2/4 k-1 k-1 + Nk-1 - Nk-2 = a1/2 + a2/4
+ … … 1 1 + N1 - N0 = a1/2 + a2/4
k i i Nk - N0 = ∑(a1 / 2 + a2 / 4 ) i=1
k Multiplying by 4 (with M0 = M(1) = a0 ):
i i ⎧ k 1 k 1 ⎫ k ⎛ ⎞ ⎛ ⎞ Mk = 4 ⎨a0 + a1∑⎜ ⎟ + a2 ∑⎜ ⎟ ⎬ ⎩ i=1 ⎝ 2⎠ i=1 ⎝ 4⎠ ⎭
COMP1004: Part 2.2 51 From the earlier lecture on geometric series,
i k k ⎛ 1⎞ 1/ 2(1− (1/ 2) ) ∑⎜ ⎟ = i=1 ⎝ 2⎠ 1−1/ 2
1 = 1− < 1 2k
i k k ⎛ 1⎞ 1/ 4(1− (1/ 4) ) ∑⎜ ⎟ = i=1 ⎝ 4 ⎠ 1−1/ 4
1 1 ⎛ ⎞ < 1/3 = ⎜1− k ⎟ 3 ⎝ 4 ⎠ Hence
k k M(2 ) < 4 (a0 + a1 + a2/3)
2 → M(n) < n (a0 + a1 + a2/3)
Since n2 is non-decreasing, and (2n)2 ∈ O(n2), we can say that M(n) ∈ O(n2).
But this is the same time-complexity as shift-and-add multiplication and the à la Russe method), so at first sight Strassen’s algorithm looks no better than our previous ones.
COMP1004: Part 2.2 52 However if we use the following observation, due to Strassen
(a - b). (d - c) = (a. d + b. c) – a. c –b. d we can compute a.c, b.d and (a.d + b.c) in just three multiplications: a.c, b.d and (a-b).(d-c)
(we assume subtractions are of negligible (O(n)) cost compared to multiplications).
Repeating the above analysis with
M(n) = 3M(n/2) + a1n + a2
we end up with i i ⎧ k 2 k 1 ⎫ k k ⎛ ⎞ ⎛ ⎞ M(2 ) = 3 ⎨a0 + a1 ∑⎜ ⎟ + a2 ∑⎜ ⎟ ⎬ ⎩ i=1 ⎝ 3⎠ i=1 ⎝ 3⎠ ⎭ where i k k 2 ⎛ 2 ⎞ ⎛ ⎞ 2⎜1 ⎛ ⎞ ⎟ < 2 ∑⎜ ⎟ = ⎜ − ⎜ ⎟ ⎟ i=1 ⎝ 3⎠ ⎝ ⎝ 3⎠ ⎠ i k k 1 1⎛ 1 ⎞ ⎛ ⎞ ⎜1 ⎛ ⎞ ⎟ < ½ ∑⎜ ⎟ = ⎜ − ⎜ ⎟ ⎟ i=1 ⎝ 3⎠ 2 ⎝ ⎝ 3⎠ ⎠ and hence
k k ⎛ a2 ⎞ M(2 ) < 3 ⎜a0 + 2a1 + ⎟ ⎝ 2 ⎠
log2 3 ⎛ a2 ⎞ M(n) < n ⎜a0 + 2a1 + ⎟ ⎝ 2 ⎠
using 3log2 n ≡ (2log2 3 )log2 n = (2log2 n )log2 3 = nlog2 3
COMP1004: Part 2.2 53 Therefore M(n) ∈ O(nlog2 3 | n is a power of 2 )
So, since nlog2 3 is non-decreasing, and (2n)log2 3 ∈ O(nlog2 3 ) , this result can be generalised to all n, and hence
M(n) ∈ O(nlog2 3 ) = O(n1.59…)
This is a time-complexity intermediate between the O(n2) of the elementary algorithms and the O(n) of the lower bound on the single-bit-operation cost of multiplying two n-bit numbers.
How much further could a Strassen-like method be pushed toward an O(n) performance?
Dividing into 3 parts of n/3 bits, and using a similar -- but more complicated -- trick to reduce the number of multiplications in this case to get just 5 (as opposed to the 9 that would be näively required) leads by similar techniques to
M(n) ∈ O(nlog3 5 )=O(n1.46…)
But relatively speaking this is a much smaller improvement than was gained by going from the elementary algorithms to breaking the numbers into two n/2-bit parts, and the associated O(n) costs (from the 'trick') are greater.
If we try to break the numbers into n/4-bit, n/5-bit...parts and use a Strassen-like method then the relative improvements each time get lessened still further, and the O(n) overheads become still greater, so that such elaborate algorithms would be found better in practice only for such large-n cases that they would not really be of use even to cryptographers. The moral is that the singleminded pursuit of an algorithm with the best possible asymptotic (n→∞) behaviour may not always in practice be a sensible thing.
COMP1004: Part 2.2 54