COMP108 Algorithmic Foundations Learning Outcomes Time

Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Learning outcomes

Able to carry out simple asymptotic analysis of COMP108 algorithms Algorithmic Foundations Algorithm efficiency

Prudence Wong

(Efficiency)

Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Time Complexity Analysis Time Complexity Analysis How fast is the algorithm? How to measure efficiency?

Code the algorithm and run the program, Number of operations usually expressed in then measure the running time terms of input size

1. Depend on the speed of the computer If we doubled/trebled the input size, how much 2. Waste time coding and testing if the algorithm is slow longer would the algorithm take?

Identify some important operations/steps and count how many times these operations/steps needed to be executed

3 4

(Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Why efficiency matters? Amount of data handled matches

speed of computation by hardware has been speed increase? improved When computation speed vastly increased, can we handle much more data? efficiency still matters Suppose ambition for computer applications grow with • an algorithm takes n2 comparisons to sort n numbers computer power • we need 1 sec to sort 5 numbers (25 comparisons) demand a great increase in speed of computation • computing speed increases by factor of100 Using 1 sec, we can now perform ?? comparisons, i.e., to sort ??numbers With 100 times speedup, only sort ?? times more numbers!

5 6

(Efficiency) (Efficiency)

Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Time/Space Complexity Analysis Look for improvement

input n Important operation of input n summation: addition sum = 0 Mathematical formula gives us an for i = 1 to n do sum = n*(n+1)/2 alternative way to find the sum of output sum How many additions this begin sum = sum + i first n numbers: algorithm requires? end output sum 1 + 2 + ... + n = n(n+1)/2 We need n additions (depend on the input size n) We only need 3 operations: We need 3 variables n, sum , & i 1 addition, 1 multiplication, and 1 division ⇒ needs 3 memory space (no matter what the input size n is) In other cases, space complexity may depend on the input size n 7 8

(Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Improve Searching Binary Search

We've learnt sequential search and it more efficient way of searching when the sequence takes n comparisons in the worst case. of numbers is pre-sorted

Input: a sequence of n sorted numbers a 1, a 2, …, a n in ascending order and a number X If the numbers are pre -sorted, then we can improve the time complexity of Idea of algorithm: searching by binary search . compare X with number in the middle then focus on only the first half or the second half (depend on whether X is smaller or greater than the middle number) reduce the amount of numbers to be searched by half

9 10

(Efficiency) (Efficiency)

Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Binary Search (2) To find 24 Binary Search (3) To find 30

3 7 11 12 15 19 24 33 41 55 10 nos 3 7 11 12 15 19 24 33 41 55 10 nos 24 X 30 X 19 24 33 41 55 19 24 33 41 55 24 30 19 24 19 24 24 30 24 24 24 found! 30 not found!

11 12

(Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Binary Search – Pseudo Code Binary Search – Pseudo Code first = 1 while first <= last do last = n  is the floor function, begin while (first <= last) do truncates the decimal part mid = (first+last)/2  if (X == a[mid]) begin mid = (first+last)/2  report "Found!" & stop if (X == a[mid]) else if (X < a[mid]) // check with no. in middle report "Found!" & stop else last = mid-1 Modify it to if (X < a[mid]) else include stopping end last = mid-1 first = mid+1 conditions in the end while loop report "Not Found!" else first = mid+1

13 14

(Efficiency) (Efficiency)

Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Number of Comparisons Best case:

Worst case: Time complexity - Big O notation … Why?

(Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Note on Logarithm Which algorithm is the fastest? Consider a problem that can be solved by 5 algorithms Logarithm is the inverse of the power function A1, A 2, A 3, A 4, A 5 using different number of operations x log 2 2 = x (time complexity). log 2 x*y = log 2 x + log 2 y f (n) = 50n + 20 f (n) = 10 n log n + 100 For example, log 2 4*8 = log 2 4 + log 2 8 = 2+3 = 5 1 2 2 2 2 0 log 16*16 = log 16 + log 16 = 8 f3(n) = n - 3n + 6 f4(n) = 2n log 2 1 = log 2 2 = 0 2 2 2 n f5(n) = 2 /8 - n/4 + 2 1 log 2 2 = log 2 2 = 1 log x/y = log x - log y n 1 2 4 8 16 32 64 128 256 512 1024 2048 2 2 2 f1(n) = 50n + 20 70 120 220 420 820 1620 3220 6420 12820 25620 5 1220 102420 2 f2(n) = 10 n log 2n + 100 100 120 180 340 740 1700 3940 9060 20580 46180 102500 2 25380 log 32/8 = log 32 - log 8 = 5-3 = 2 2 log 2 4 = log 2 2 = 2 2 2 2 f3(n) = n - 3n + 6 4 4 10 46 214 934 3910 16006 64774 3E+05 1E+06 4E+06 2 f4(n) = 2n 2 8 32 128 512 2048 8192 32768 131072 5E+05 2E+06 8E+06 4 log 2 1/4 = log 2 1 - log 2 4 = 0-2 = -2 n log 2 16 = log 2 2 = 4 f5(n) = 2 /8 - n/4 + 2 2 2 3 32 8190 5E+08 2E+18 8 log 2 256 = log 2 2 = 8 f5(n) f3(n) f1(n) 10 log 2 1024 = log 2 2 = 10 17 Depends on the size of the input! 18 (Efficiency) (Efficiency)

Algorithmic Foundations Algorithmic Foundations COMP108 COMP108

1000000 What do we observe?

100000 There is huge difference between 2 10000 functions involving powers of n (e.g., n, n , called polynomial functions) and 1000 Time functions involving powering by n (e.g., 2 n, 3 n, f1(n)f1(n) == 50n50n ++ 2020

100 f2(n)f2(n) == 1010 nn loglog2n2n + +100 100 called exponential functions) 2 f3(n)f3(n) == n n2- - 3n 3n + + 6 6 2 10 f4(n)f4(n) == 2n2n2 Among polynomial functions, those with same n f5(n)f5(n) == 2 2n/8/8 - - n/4n/4 + + 2 2 order of power are more comparable 1 2 2 1 2 4 8 16 32 64 128 256 512 102420484096 e.g., f3(n) = n - 3n + 6 and f4(n) = 2n n

19 20

(Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Growth of functions Relative growth rate

n 2 n2

log n c n

21 22

(Efficiency) (Efficiency)

Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Hierarchy of functions Hierarchy of functions (2) 1 log n n n 2 n3 ... n k ... 2n We can define a hierarchy of functions each having a greater order of growth than its predecessor: constant logarithmic polynomial exponential 1 log n n n 2 n3 ... n k ... 2n Note: as we move from left to right, successive functions have greater order of growth than constant logarithmic polynomial exponential the previous ones. As n increases, the values of the later functions We can further refine the hierarchy by inserting increase more rapidly than the earlier ones. n log n between n and n2, ⇒ Relative growth rates increase n2 log n between n2 and n3, and so on.

23 24

(Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Hierarchy of functions (3) Hierarchy of functions (4) (log n) 3 1 log n n n 2 n3 ... n k ... 2n What about log 3 n & n? Which is higher in hierarchy? constant logarithmic polynomial exponential Remember: n = 2 log n So we are comparing (log n) 3 & 2log n Now, when we have a function, we can classify the ∴ log 3 n is lower than n in the hierarchy function to some function in the hierarchy: For example, f(n) = 2n 3 + 5n 2 + 4n + 7 3 Similarly, log k n is lower than n in the hierarchy, The term with the highest power is 2n . 3 for any constant k The growth rate of f(n) is dominated by n . This concept is captured by Big-O notation

25 26

(Efficiency) (Efficiency)

Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Big-O notation Exercise f(n) = O(g(n)) [read as f(n) is of order g(n)] Determine the order of growth of the Roughly speaking, this means f(n) is at most a following functions. constant times g(n) for all large n 3 2 Examples 1. n + 3n + 3 2 3 2 2n 3 = O(n 3) 2. 4n log n + n + 5n + n 2 2 3n = O(n ) 3. 2n 2 + n 2 log n 2n log n = O(n log n) 4. 6n 2 + 2 n n3 + n 2 = O(n 3)

Look for the term highest in the hierarchy

27 28

(Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 More Exercise Big-O notation - formal definition Are the followings correct? f(n) = O(g(n)) 1. n2log n + n 3 + 3n 2 + 3 O(n 2log n)? There exists a constant c and no such that ≤≤≤ f(n) c g(n) for all n > n o 2. n + 1000 O(n)? ∃∃∃ ∃∃∃ ∀∀∀ ≤≤≤ c no n>n o then f(n) c g(n) 3. 6n 20 + 2 n O(n 20 )? 7000

3 2 2 6000 4. n + 5n log n + n O(n log n) ? c g(n) Graphical 5000 Illustration 4000

Time 3000 f(n) 2000

1000

0 29 100 300n0 500 700 900 1100 30 input size (n) (Efficiency) (Efficiency)

Algorithmic Foundations Algorithmic Foundations COMP108 COMP108

Example: n+60 is O(n) ∃∃∃ Which one is the fastest? constants c & n o such ∀∀∀ ≤≤≤ that n>n o, f(n) c g(n) Usually we are only interested in the 400 c=2, n =60 0 2n asymptotictime complexity 350 i.e., when n is large 300 n + 60 250 O(log n) < O(log 2 n) < O( √n) < O(n) < O(n log n) < O(n 2) < O(2 n) 200 Time 150 n

100

0 10 30 50 70 90 110 130 150 170 190 input size (n) n0 31 32

(Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Proof of order of growth Proof of order of growth (2) 2 2 3 2 3 Prove that 2n + 4n is O(n ) Note: plotting a graph Prove that n + 3n + 3 is O(n ) is NOT a proof Since n ≤≤≤ n2 ∀∀∀n≥1 , Since n2 ≤≤≤ n3 and 1 ≤≤≤ n3 ∀∀∀n≥1 , we have we have 2n 2 + 4n ≤≤≤ 2n 2 + 4n 2 n3 + 3n 2 + 3 ≤≤≤ n3 + 3n 3 + 3n 3 = 6n2 ∀∀∀n≥1. = 7n3 ∀∀∀n≥1. Therefore, by definition, 2n 2 + 4n is O(n 2). Therefore, by definition, n3 + 3n 2 + 3 is O(n 3). Alternatively, Alternatively,

Since 4n ≤≤≤ n2 ∀∀∀n≥4 , Since 3n 2 ≤≤≤ n3 ∀∀∀n≥3 , and 3 ≤≤≤ n3 ∀∀∀n≥2 we have we have 2n 2 + 4n ≤≤≤ 2n 2 + n 2 n3 + 3n 2 + 3 ≤≤≤ 3n 3 ∀∀∀n≥3 .

= 3n2 ∀∀∀n≥4 . Therefore, by definition, n3 + 3n 2 + 3 is O(n 3). 33 34 Therefore, by definition, 2n 2 + 4n is O(n 2). (Efficiency) (Efficiency)

Algorithmic Foundations Algorithmic Foundations COMP108 COMP108 Challenges Some algorithms we learnt Prove the order of growth st input n 1. 2n 3 + n 2 + 4n + 4 is O(n 3) Sum of 1 n no. sum = 0 input n for i = 1 to n do sum = n*(n+1)/2 begin output sum sum = sum + i end O(?) output sum O(?)

Min value among n no. 2. 2n 2 + 2 n is O(2 n) loc = 1 for i = 2 to n do if (a[i] < a[loc]) then loc = i output a[loc] O(?)

35 36

(Efficiency) (Efficiency) Algorithmic Foundations Algorithmic Foundations COMP108 suppose n=8 COMP108 (@ end of ) i count Time complexity of this? What about this? iteration for i = 1 to 2n do i = 1 1 0 count = 0 for j = 1 to n do 1 2 1 x = x + 1 O(?) while i < n O(?) begin 2 4 2 i = 2 * i 3 8 3 count = count + 1 suppose n=32 end output count (@ end of ) i count The outer loop iterates for ?? times. iteration 1 0 The inner loop iterates for ? times for each i. 1 2 1 Total: ?? * ? . 2 4 2 3 8 3 4 16 4 37 5 32 5 38 (Efficiency) (Efficiency)