Beyond Worst Case Analysis Amortized Analysis Worst-case analysis. ■ Analyze running time as function of worst input of a given size. Average case analysis. ■ Analyze average running time over some distribution of inputs. ■ Ex: quicksort. Amortized analysis. ■ Worst-case bound on sequence of operations. ■ Ex: splay trees, union-find. Competitive analysis. ■ Make quantitative statements about online algorithms. ■ Ex: paging, load balancing. Princeton University • COS 423 • Theory of Algorithms • Spring 2001 • Kevin Wayne 2 Amortized Analysis Dynamic Table Amortized analysis. Dynamic tables. ■ Worst-case bound on sequence of operations. ■ Store items in a table (e.g., for open-address hash table, heap). – no probability involved ■ Items are inserted and deleted. ■ Ex: union-find. – too many items inserted ⇒ copy all items to larger table – sequence of m union and find operations starting with n – too many items deleted ⇒ copy all items to smaller table singleton sets takes O((m+n) α(n)) time. – single union or find operation might be expensive, but only α(n) Amortized analysis. on average ■ Any sequence of n insert / delete operations take O(n) time. ■ Space used is proportional to space required. ■ Note: actual cost of a single insert / delete can be proportional to n if it triggers a table expansion or contraction. Bottleneck operation. ■ We count insertions (or re-insertions) and deletions. ■ Overhead of memory management is dominated by (or proportional to) cost of transferring items. 3 4 Dynamic Table: Insert Dynamic Table: Insert Dynamic Table Insert Accounting method. Initialize table size m = 1. ■ Charge each insert operation $3 (amortized cost). – use $1 to perform immediate insert INSERT(x) – store $2 in with new item IF (number of elements in table = m) ■ When table doubles: Generate new table of size 2m. Re-insert m old elements into new table. – $1 re-inserts item m ← 2m – $1 re-inserts another old item Insert x into table. Aggregate method. ■ Sequence of n insert ops takes O(n) time. n log n 2 j th ∑c ≤ n + ∑ 2 ■ Let c = cost of i insert. i i i=1 j=0 = n + (2n −1) i if i - 1 is an exact power of 2 c = < 3n i 1 otherwise 5 6 Dynamic Table: Insert and Delete Dynamic Table: Insert and Delete Insert and delete. Insert and delete. ■ Table overflows ⇒ double table size. ■ Table overflows ⇒ double table size. ■ Table ≤ ½ full ⇒ halve table size. ■ Table ≤ ¼ full ⇒ halve table size. ! Bad idea: can cause thrashing. Dynamic Table Delete Initialize table size m = 1. 1 1 1 1 DELETE(x) 2 2 2 2 IF (number of elements in table ≤ m / 4) Generate new table of size m / 2. 3 3 3 3 m ← m / 2 4 4 4 4 Reinsert old elements into new table. 5 5 Delete x from table. 7 8 Dynamic Table: Insert and Delete Dynamic Table: Delete Accounting analysis. ■ Charge each insert operation $3 (amortized cost). 1 2 3 4 5 6 7 8 – use $1 to perform immediate insert – store $2 with new item 1 2 3 4 5 6 7 ■ When table doubles: – $1 re-inserts item – $1 re-inserts another old item 1 2 3 4 5 6 ■ Charge each delete operation $2 (amortized cost). 1 2 3 4 5 – use $1 to perform delete – store $1 in emptied slot ■ When table halves: 1 2 3 4 – $1 in emptied slot pays to re-insert a remaining item into new half-size table 1 2 3 4 Contract table 9 10 Dynamic Table: Insert and Delete Binary Search Tree Theorem. Sequence of n inserts and deletes takes O(n) time. Binary tree in "sorted" order. ■ Amortized cost of insert = $3. ■ Maintain ordering property for ALL sub-trees. ■ Amortized cost of delete = $2. root (middle value) left subtree right subtree (larger values) (smaller values) 11 12 Binary Search Tree Binary Search Tree Binary tree in "sorted" order. Binary tree in "sorted" order. ■ Maintain ordering property for ALL sub-trees. ■ Maintain ordering property for ALL sub-trees. 51 51 14 72 14 72 06 33 53 97 06 33 53 97 13 25 43 64 84 99 13 25 43 64 84 99 13 14 Binary Search Tree Splay Trees Insert, delete, find (symbol table). Search Splay trees (Sleator-Tarjan, 1983a). Self-adjusting BST. ■ Amount of work proportional to height of tree. ■ Most frequently accessed items are close to root. ■ O(N) in "unbalanced" search tree. ■ Tree automatically reorganizes itself after each operation. Insert ■ O(log N) in "balanced" search tree. – no balance information is explicitly maintained ■ Tree remains "nicely" balanced, but height can potentially be n - 1. Types of BSTs. ■ Sequence of m ops involving n inserts takes O(m log n) time. ■ AVL trees, 2-3-4 trees, red-black trees. ■ Treaps, skip lists, splay trees. Theorem (Sleator-Tarjan, 1983a). Splay trees are as efficient (in amortized sense) as static optimal BST. BST vs. hash tables. Theorem (Sleator-Tarjan, 1983b). Shortest augmenting path algorithm ■ Guaranteed vs. expected performance. for max flow can be implemented in O(mn log n) time. ■ Growing and shrinking. ■ Sequence of mn augmentations takes O(mn log n) time! ■ Augmented data structures: order statistic trees, interval trees. ■ Splay trees used to implement dynamic trees (link-cut trees). 15 16 Splay Splay Find(x, S): Determine whether element x is in splay tree S. Implementing Join(S, S'). Insert(x, S): Insert x into S if it is not already there. ■ Call Splay(+∞, S) so that largest element of S is at root and all Delete(x, S): Delete x from S if it is there. other elements are in left subtree. Join(S, S'): Join S and S' into a single splay tree, assuming that ■ Make S' the right subtree of the root of S. x < y for all x ∈ S, and y ∈ S'. Implementing Delete(x, S). All operations are implemented in terms of basic operation: ■ Call Splay(x, S) to bring x to the root if it is there. ■ Remove x: let S' and S'' be the resulting subtrees. Splay(x, S): Reorganize splay tree S so that element x is at the root if x ∈ S; otherwise the new root is either ■ Call Join(S', S''). max { k ∈ S : k < x} or min { k ∈ S : k > x} . Implementing Insert(x, S). Implementing Find(x, S). ■ Call Splay(x, S) and break tree at root to form S' and S''. ■ Call Splay(x, S). ■ Call Join(Join(S', {x}), S''). ■ If x is root, then return x; otherwise return NO. 17 18 Implementing Splay(x, S) Implementing Splay(x, S) Splay(x, S): do following operations until x is root. Splay(x, S): do following operations until x is root. ■ ZIG: If x has a parent but no grandparent, then rotate(x). ■ ZIG: If x has a parent but no grandparent. ■ ZIG-ZIG: If x has a parent y and a grandparent, and if both x and y ■ ZIG-ZIG: If x has a parent y and a grandparent, and if both x and y are either both left children or both right children. are either both left children or both right children. ■ ZIG-ZAG: If x has a parent y and a grandparent, and if one of x, y ■ ZIG-ZAG: If x has a parent y and a grandparent, and if one of x, y is a left child and the other is a right child. is a left child and the other is a right child. root z x x y y y y x D A C ZIG(x) A x ZIG-ZIG z C B A B ZAG(y) B C A B C D 19 20 Implementing Splay(x, S) Splay Example Splay(x, S): do following operations until x is root. Apply Splay(1, S) to tree S: 10 ■ ZIG: If x has a parent but no grandparent. ■ ZIG-ZIG: If x has a parent y and a grandparent, and if both x and y 9 are either both left children or both right children. 8 ■ ZIG-ZAG: If x has a parent y and a grandparent, and if one of x, y is a left child and the other is a right child. 7 6 ZIG-ZIG z x 5 y ZIG-ZAG z y 4 A 3 x D A B C D 2 1 B C 21 22 Splay Example Splay Example Apply Splay(1, S) to tree S: Apply Splay(1, S) to tree S: 10 10 9 9 8 8 7 7 6 ZIG-ZIG 6 ZIG-ZIG 5 1 4 4 1 2 5 2 3 3 23 24 Splay Example Splay Example Apply Splay(1, S) to tree S: Apply Splay(1, S) to tree S: 10 10 9 1 8 8 1 6 9 6 ZIG-ZIG 4 7 ZIG 4 7 2 5 2 5 3 3 25 26 Splay Example Splay Example Apply Splay(1, S) to tree S: Apply Splay(2, S) to tree S: 1 10 1 2 8 10 1 8 6 9 4 8 10 4 7 6 9 3 6 9 Splay(2) 2 5 7 4 5 7 3 2 5 3 27 28 Splay Tree Analysis Splay Tree Analysis Definitions. Splay invariant: node x always has at least µ(x) credits on deposit. ■ Let S(x) denote subtree of S rooted at x. ≤ µ µ ■ |S| = number of nodes in tree S. Splay lemma: each splay(x, S) operation requires 3( (S) - (x)) + 1 credits to perform the splay operation and maintain the invariant. ■ µ(S) = rank = log |S| . 2 ■ µ(x) = µ (S(x)). Theorem: A sequence of m operations involving n inserts takes S(8) O(m log n) time.