Amortized Analysis
1. The concepts of amortized analysis
2. Methods of amortized analysis • Aggregate method • Accounting method • Potential method
3. Problem examples by amortized analysis •Stack • Binary counter • Dynamic array
CP412 ADAII, Wilfrid Laurier University Introduction # 1 1. What is Amortized Analysis • Previous analysis focuses on time/space usage of algorithms for arbitrary input of a problem. – The worst analysis considers the worst cost over all possible inputs of given size – The average case considers the average cost over all possible input of given size. The average is a probability expectation. – Rough and pessimistic
• Amortized Analysis is concerned with the overall cost of a sequence of operations – the average performance of each operation in the worst case over all possible sequences of the operations.
CP412 ADAII, Wilfrid Laurier University Introduction # 2 Why concerning a sequence of operations
– data structures typically undergo a sequence of operations. E.g. push or pop operations stack.
– For a sequence of operations, the cost of an operation at different position are different.
E.g. for tree insertion operation, the cost of early insertion and later insertions are different.
– amortized analysis allows a tighter bound that better reflects performance than the worst case analysis
CP412 ADAII, Wilfrid Laurier University Introduction # 3 Amortized Analysis vs other analysis • Amortized analysis vs average case analysis – Average case analysis relies on probabilistic assumptions about the data structures and operations in order to compute an expected running time of an algorithm. – Amortized analysis needs no such assumptions. – Amortized analysis offers a true upper bound on the worst case running time of a sequence of operations. – An average case bound does not provide a true upper bound. That means one may not “unlucky” and encounter an input that requires much more than the expected computation time. • Amortized analysis vs competitive analysis – Amortization is useful for competitive analysis's performance. Sleator and Tarjan (1985a) offer an example of using amortized analysis to perform competitive analysis
CP412 ADAII, Wilfrid Laurier University Introduction # 4 History • In finance, amortization refers to paying off a debt, such as a loan or mortgage, by smaller payments made over time
• Tarjan introduced amortized analysis for algorithms as a general technique in 1985 – Three methods of amortized analysis were recognized: – the banker's method (Brown and Tarjan,1980) – Aggregate method ((Aho et al. 1974) – potential method (Sleator 1985)
CP412 ADAII, Wilfrid Laurier University Introduction # 5 2. Amortized analysis methods • The aggregate method, where the total running time for a sequence of operations is analyzed. T(n) is total cost over a worst-case sequence of n operations T(n) Amortized cost per operation = n
• The accounting (or banker's) method, where we impose an extra charge on inexpensive operations and use it to pay for expensive operations later on.
• The potential (or physicist's) method, in which we derive a potential function characterizing the amount of extra work we can do in each step. This potential either increases or decreases with each successive operation, but cannot be negative.
CP412 ADAII, Wilfrid Laurier University Introduction # 6 Stack with push, pop, MultiPop operations
Consider stack data structure with three operations Push(S, x) -- push x onto stack S, worst-case cost O(1). Pop (S) -- pop the top element of S, worst-case cost O(1). MultiPop(S, k): it pops top k elements of S, worst-case cost O(k).
MultiPop (S, k) while S is not empty ; and k > 0 Pop (S) k = k - 1 end
What is the worst-case complexity of n MultiPop Stack ops? Because any single MultiPop can be (n), the standard worst-case bound is O(n2).
CP412 ADAII, Wilfrid Laurier University Introduction # 7 Aggregate method of stack
Theorem:
A sequence of n ops, on an initially empty stack, has cost O(n).
Proof:
Each element is popped at most once (either by Pop or during a MultiPop). Total cost of Pops, including MultiPops, is ≤ number of Pushes, which is at most n. Total cost of all operations is T(n) ≤ 2n.
Thus, amortized cost per operation for this stack is at most 2.
This bound holds in worst-case, over any sequence!
CP412 ADAII, Wilfrid Laurier University Introduction # 8 Accounting method of Stack
The key is to choose the amortized costs carefully.
Suppose the actual cost of the ith operation is Ci ,
we use some other cost Ai as its amortized cost. Then, we have to make sure that, for any possible nn sequence of n operations, ACii ii11 The total credit accrued by the data structure at any point nn is (ACii ) 0 ii11
CP412 ADAII, Wilfrid Laurier University Introduction # 9 Accounting method of Stack
The actual costs (Ci) of operations in the stack example are: 1 for Push, 1 for Pop, min(k, s) for MultiPop, where s is the stack size when the MultiPop is called.
Let us assign the following amortized costs (Ai) to these ops: 2 for Push 0 for Pop 0 for MultiPop
The amortized cost of MultiPop is a constant (in fact, 0), even though it's real cost if variable. The intuition is that the credit accrued through Push operations will be enough to pay for the actual cost of MultiPop.
CP412 ADAII, Wilfrid Laurier University Introduction # 10 Accounting method of stack
Think of stack as a bank, where each deposit and withdrawal costs $1. When an item is pushed onto the stack, we use $1 for the Push operation, and leave the second dollar (of its amortized cost) with the item in the bank. This second (spare) dollar will be used to pay for the item's Pop.
The item may be popped either through a single Pop or a MultiPop, but since each item in the stack has a spare dollar allocated to it, the Pop and MultiPop can be entirely paid using that credit. The amortized cost of Pop and MultiPop is therefore 0.
Thus, for any sequence of n Push, Pop, MultiPop, the total amortized cost is at most 2n = O(n), and it is an upper bound on the actual cost. The average cost per operation is 2.
CP412 ADAII, Wilfrid Laurier University Introduction # 11 Potential method of stack
Keep track of the potential energy of the data structure.
Start with initial data structure Di , and perform n updates.
Let Ci be the actual cost of ith op;
let Di be data structure state after op i. Function measures potential of data structure. Define amortized cost of operation i as
ACii= + (( D i ) ( D i1 ))
CP412 ADAII, Wilfrid Laurier University Introduction # 12 Potential method of stack
Since ACii = + ( ( D i ) ( D i1 )), by adding them up them up over the sequence, we get nn ACii+ ( D n ) ( D0 ), ii11
Suppose (DDi ) (0 ) 0, for all i , then nn ACii ii11 So, amortized cost is an upper bound on actual total cost.
How to define (Di )
CP412 ADAII, Wilfrid Laurier University Introduction # 13 Potential method of stack
Let = number of items on stack
()0,()DDD00i ()0 Amortized cost of Push:
ACii(( D i ) ( D i1 ))11 2 Amortized cost of Pop:
ACii(( D i ) ( D i1 ))110 Amortized cost of MultiPop (S, k)::
ACii(( D i ) (Dskski 1 )) min( , ) min( , ) 0 n Thus, amortized total cost Ani 2 , i 1 amortized cost per operation is O(1).
CP412 ADAII, Wilfrid Laurier University Introduction # 14 Amortized Analysis of a Binary Counter
Binary counter implemented as an array of bits: A[0...k - 1]. Consider incrementing A starting from 0. Increment (A) i = 0; while i k and A[i] = 1 A[i] = 0; i = i + 1; if i < k then A[i] = 1;
Cost of Increment is number of bits flipped.
CP412 ADAII, Wilfrid Laurier University Introduction # 15 Analysis of a Binary Counter
Binary counter bit sequence:
Value Bits Cost 0 0…000 0 1 0…001 1 2 0…010 2 3 0…011 1
In the worst-case, a single increment can flip k bits.
There are n increments, so by standard worst-case analysis, the cost is O(nk).
CP412 ADAII, Wilfrid Laurier University Introduction # 16 Amortized Analysis of Binary Counter
Theorem: starting with 0, worst-case cost of n increments is O(n). Proof: Bit A[0] flips each increment. Bit A[1] flips every other increment. Bit A[i] flips during every 1/2i -th increment. During n increments, bit A[i] flips n/2i times. Assume n ≤ 2k; otherwise, the counter resets, and restart analysis. Total cost
log n n 1 nn2 ii ii0022
Amortized cost per increment is 2
CP412 ADAII, Wilfrid Laurier University Introduction # 17 Accounting method of binary counter
Let us assign the following amortized costs Ai = 2 for every increment operation
Prove that up to j increments, the actual cost and amortized total coast satisfying:
jj Cjii2 A ii11
CP412 ADAII, Wilfrid Laurier University Introduction # 18 CP412 ADAII, Wilfrid Laurier University Introduction # 19 Potential method of binary counter
Let ii = (D ) be the number of 1's in the counter
after nth operation, then i 0 = 0 for all i.
Suppose ith Increment resets ti bits.
Actual cost C ii = t + 1.
ii1 bb, then iii t + 1 Amortized cost
ACii( ii1 ) (t i + 1) ( b ii t +1 b i ) 2 n Thus, total amortized cost Ani 2 . i 1
CP412 ADAII, Wilfrid Laurier University Introduction # 20 Dynamic Array
Description Think of an array initialized with a fixed number of slots, and supports APPEND and DELETE operations.
When we APPEND too many elements, the array would be full and we need to expand the array (make the size larger).
When we DELETE too many elements, we want to shrink to the array (make the size smaller).
Requirement: the array must be using one contiguous block of memory all the time.
CP412 ADAII, Wilfrid Laurier University Introduction # 21 Append algorithm
CP412 ADAII, Wilfrid Laurier University Introduction # 22 One way to expand
• If the array is full when APPEND is called • Create a new array of twice the size • Copy the all the elements from old array to new array • Append the element
CP412 ADAII, Wilfrid Laurier University Introduction # 23 Amortized analysis of expand
• Now consider a dynamic array initialized with size 1 and a sequence of m APPEND operations on it.
• Analyze the amortized cost per operation
• Assumption: only count array assignments, i.e., append an element and copy an element
CP412 ADAII, Wilfrid Laurier University Introduction # 24 Content template
•…
CP412 ADAII, Wilfrid Laurier University Introduction # 25 Aggregate Analysis
CP412 ADAII, Wilfrid Laurier University Introduction # 26 Accounting method How much money do we need to earn at each operation, so that all future costs can be paid for?
• How much money to earn for each APPEND element ?
$1 ? $2 ? $3 ? $m ?…
CP412 ADAII, Wilfrid Laurier University Introduction # 27 Earn $1 for each appended element
This $1 (the “append-dollar”) is spent when appending the element.
But, when we need to copy this element to a new array (when expanding the array), we don’t have any money to pay for it --
BROKE!
CP412 ADAII, Wilfrid Laurier University Introduction # 28 Earn $2 for each appended element
$1 (the “append-dollar”) will be spent when appending the element
$1 (the “copy-dollar”) will be spent when copying the element to a new array
What if the element is copied for a second time (when expanding the array for a second time)?
BROKE!…
CP412 ADAII, Wilfrid Laurier University Introduction # 29 Earn $3 for each appended element
$1 (the “append-dollar”) will be spent when appending the element
$1 (the “copy-dollar”) will be spent when copying the element to a new array
$1 (the “recharge-dollar”) is used to recharge the old elements that have spent their “copy-dollars”.
NEVER BROKE!… Amortized cost is A(n) = O(3n) Average cost O(1)
CP412 ADAII, Wilfrid Laurier University Introduction # 30 Potential Method
CP412 ADAII, Wilfrid Laurier University Introduction # 31 Potential Method
CP412 ADAII, Wilfrid Laurier University Introduction # 32 Potential Method
CP412 ADAII, Wilfrid Laurier University Introduction # 33 Amortized analysis provides us valuable insights into what is the proper strategy of expanding dynamic arrays.
Shrinking strategy: When the array is ½ full after DELETE, create a new array of half of the size, and copy all the elements.
Consider the following sequence of operations performed on a full array with n element… APPEND, DELETE, APPEND, DELETE, APPEND, …
Ɵ(n) amortized cost per operation since every APPEND or DELETE causes allocation of new array. NO GOOD!
CP412 ADAII, Wilfrid Laurier University Introduction # 34 The right way of shrinking
When the array is ¼ full after DELETE, create a new array of ½ of the size, and copy all the elements.
For any sequence of APPEND or DELETE , earning $3 per APPEND and $3 per DELETE would be enough for paying all the cost. 1 append/delete-dollar 1 copy-dollar 1 recharge-dollar
Therefore the amortized cost per operation of any sequence is upper-bounded by 3, i.e., O(1).
CP412 ADAII, Wilfrid Laurier University Introduction # 35 Expansion and Contraction
Let α be element count / array size. When α drops too low, contract the table. – Allocate a new, smaller one. – Copy all items.
• Still want
– α bounded from below by a constant, – amortized cost per operation = O(1).
CP412 ADAII, Wilfrid Laurier University Introduction # 36 Expansion and Contraction
Double as before: – When inserting with α = 1 – After doubling, α = 1/2.
• Halve size – When deleting with α = 1/4 – After halving, α = 1/2. • Thus, immediately after either expansion or contraction have – α = 1/2. • Always have – 1/4 ≤ α ≤ 1. CP412 ADAII, Wilfrid Laurier University Introduction # 37 Expansion and Contraction
CP412 ADAII, Wilfrid Laurier University Introduction # 38 Expansion and Contraction
CP412 ADAII, Wilfrid Laurier University Introduction # 39 Expansion and Contraction
CP412 ADAII, Wilfrid Laurier University Introduction # 40 Expansion and Contraction
CP412 ADAII, Wilfrid Laurier University Introduction # 41