Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Learning outcomes Able to carry out simple asymptotic analysis of COMP108 algorithms Algorithmic Foundations Algorithm efficiency Prudence Wong 2 (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Time Complexity Analysis Time Complexity Analysis How fast is the algorithm? How to measure efficiency? Code the algorithm and run the program, Number of operations usually expressed in then measure the running time terms of input size 1. Depend on the speed of the computer If we doubled/trebled the input size, how much 2. Waste time coding and testing if the algorithm is slow longer would the algorithm take? Identify some important operations/steps and count how many times these operations/steps needed to be executed 3 4 (Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Why efficiency matters? Amount of data handled matches speed of computation by hardware has been speed increase? improved When computation speed vastly increased, can we handle much more data? efficiency still matters Suppose ambition for computer applications grow with • an algorithm takes n2 comparisons to sort n numbers computer power • we need 1 sec to sort 5 numbers (25 comparisons) demand a great increase in speed of computation • computing speed increases by factor of 100 Using 1 sec, we can now perform ?? comparisons, i.e., to sort ?? numbers With 100 times speedup, only sort ?? times more numbers! 5 6 (Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Time/Space Complexity Analysis Look for improvement input n Important operation of input n summation: addition sum = 0 Mathematical formula gives us an for i = 1 to n do sum = n*(n+1)/2 alternative way to find the sum of output sum How many additions this begin sum = sum + i first n numbers: algorithm requires? end output sum 1 + 2 + ... + n = n(n+1)/2 We need n additions (depend on the input size n) We only need 3 operations: We need 3 variables n, sum , & i 1 addition, 1 multiplication, and 1 division ⇒ needs 3 memory space (no matter what the input size n is) In other cases, space complexity may depend on the input size n 7 8 (Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Improve Searching Binary Search We've learnt sequential search and it more efficient way of searching when the sequence takes n comparisons in the worst case. of numbers is pre-sorted Input: a sequence of n sorted numbers a 1, a 2, …, a n in ascending order and a number X If the numbers are pre -sorted, then we can improve the time complexity of Idea of algorithm: searching by binary search . compare X with number in the middle then focus on only the first half or the second half (depend on whether X is smaller or greater than the middle number) reduce the amount of numbers to be searched by half 9 10 (Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Binary Search (2) To find 24 Binary Search (3) To find 30 3 7 11 12 15 19 24 33 41 55 10 nos 3 7 11 12 15 19 24 33 41 55 10 nos 24 X 30 X 19 24 33 41 55 19 24 33 41 55 24 30 19 24 19 24 24 30 24 24 24 found! 30 not found! 11 12 (Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Binary Search – Pseudo Code Binary Search – Pseudo Code first = 1 while first <= last do last = n is the floor function, begin while (first <= last) do truncates the decimal part mid = (first+last)/2 if (X == a[mid]) begin mid = (first+last)/2 report "Found!" & stop if (X == a[mid]) else if (X < a[mid]) // check with no. in middle report "Found!" & stop else last = mid-1 Modify it to if (X < a[mid]) else include stopping end last = mid-1 first = mid+1 conditions in the end while loop report "Not Found!" else first = mid+1 13 14 (Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Number of Comparisons Best case: Worst case: Time complexity - Big O notation … Why? 15 (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Note on Logarithm Which algorithm is the fastest? Consider a problem that can be solved by 5 algorithms Logarithm is the inverse of the power function A1, A 2, A 3, A 4, A 5 using different number of operations x log 2 2 = x (time complexity). log 2 x*y = log 2 x + log 2 y f (n) = 50n + 20 f (n) = 10 n log n + 100 For example, log 2 4*8 = log 2 4 + log 2 8 = 2+3 = 5 1 2 2 2 2 0 log 16*16 = log 16 + log 16 = 8 f3(n) = n - 3n + 6 f4(n) = 2n log 2 1 = log 2 2 = 0 2 2 2 n f5(n) = 2 /8 - n/4 + 2 1 log 2 2 = log 2 2 = 1 log x/y = log x - log y n 1 2 4 8 16 32 64 128 256 512 1024 2048 2 2 2 f1(n) = 50n + 20 70 120 220 420 820 1620 3220 6420 12820 25620 51220 102420 2 f2(n) = 10 n log 2n + 100 100 120 180 340 740 1700 3940 9060 20580 46180 102500 225380 log 32/8 = log 32 - log 8 = 5-3 = 2 2 log 2 4 = log 2 2 = 2 2 2 2 f3(n) = n - 3n + 6 4 4 10 46 214 934 3910 16006 64774 3E+05 1E+06 4E+06 2 f4(n) = 2n 2 8 32 128 512 2048 8192 32768 131072 5E+05 2E+06 8E+06 4 log 2 1/4 = log 2 1 - log 2 4 = 0-2 = -2 n log 2 16 = log 2 2 = 4 f5(n) = 2 /8 - n/4 + 2 2 2 3 32 8190 5E+08 2E+18 8 log 2 256 = log 2 2 = 8 f5(n) f3(n) f1(n) 10 log 2 1024 = log 2 2 = 10 17 Depends on the size of the input! 18 (Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 1000000 What do we observe? 100000 There is huge difference between 2 10000 functions involving powers of n (e.g., n, n , called polynomial functions) and 1000 Time functions involving powering by n (e.g., 2 n, 3 n, f1(n)f1(n) == 50n50n + + 20 20 100 f2(n)f2(n) == 10 10 n n log log2n2n + 100+ 100 called exponential functions) 2 f3(n)f3(n) == n n2- - 3n 3n + +6 6 2 10 f4(n)f4(n) == 2n 2n2 Among polynomial functions, those with same n f5(n)f5(n) == 2 2n/8/8 - - n/4 n/4 + +2 2 order of power are more comparable 1 2 2 1 2 4 8 16 32 64 128 256 512 102420484096 e.g., f3(n) = n - 3n + 6 and f4(n) = 2n n 19 20 (Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Growth of functions Relative growth rate n 2 n2 n log n c n 21 22 (Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Hierarchy of functions Hierarchy of functions (2) 1 log n n n 2 n3 ... n k ... 2n We can define a hierarchy of functions each having a greater order of growth than its predecessor: constant logarithmic polynomial exponential 1 log n n n 2 n3 ... n k ... 2n Note: as we move from left to right, successive functions have greater order of growth than constant logarithmic polynomial exponential the previous ones. As n increases, the values of the later functions We can further refine the hierarchy by inserting increase more rapidly than the earlier ones. n log n between n and n2, ⇒ Relative growth rates increase n2 log n between n2 and n3, and so on. 23 24 (Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Hierarchy of functions (3) Hierarchy of functions (4) (log n) 3 1 log n n n 2 n3 ... n k ... 2n What about log 3 n & n? Which is higher in hierarchy? constant logarithmic polynomial exponential Remember: n = 2 log n So we are comparing (log n) 3 & 2log n Now, when we have a function, we can classify the ∴ log 3 n is lower than n in the hierarchy function to some function in the hierarchy: For example, f(n) = 2n 3 + 5n 2 + 4n + 7 3 Similarly, log k n is lower than n in the hierarchy, The term with the highest power is 2n . 3 for any constant k The growth rate of f(n) is dominated by n . This concept is captured by Big-O notation 25 26 (Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Big-O notation Exercise f(n) = O(g(n)) [read as f(n) is of order g(n)] Determine the order of growth of the Roughly speaking, this means f(n) is at most a following functions. constant times g(n) for all large n 3 2 Examples 1. n + 3n + 3 2 3 2 2n 3 = O(n 3) 2. 4n log n + n + 5n + n 2 2 3n = O(n ) 3. 2n 2 + n 2 log n 2n log n = O(n log n) 4. 6n 2 + 2 n n3 + n 2 = O(n 3) Look for the term highest in the hierarchy 27 28 (Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 More Exercise Big-O notation - formal definition Are the followings correct? f(n) = O(g(n)) 1. n2log n + n 3 + 3n 2 + 3 O(n 2log n)? There exists a constant c and no such that ≤≤≤ f(n) c g(n) for all n > n o 2.