Searching Algorithms – General definition • Locate an element x in a list of distinct elements a , a , …, a , or determine that it is not in the list Searching and Sorting Algorithms, 1 2 n Complexity Analysis – Linear search Algorithm 2: Linear search Input: unsorted sequence a1, a2, …, an position of target value x Output: subscript of entry equal to target value; 0 if not found Initialize: i ← 1 while (i ≤ n and x ≠ ai) i ← i + 1 if i ≤ n then location ← i else location ← 0 © F. Pedone, N. Schiper 2 Searching Algorithms Searching Algorithms – Binary search – Binary search • Requires input sequence to be sorted General Idea: Algorithm 3: Binary search Input: sorted sequence a1, a2, …, an Find 9 in the sequence: <1, 4, 6, 9, 10, 14> position of target value x Output: subscript of entry equal to target value; 0 if not found Initialize: left ← 1; right ← n left mid right left mid right 9 > amid ? while (left < right) Yes (9 > 6) mid ← (left + right) / 2 <1, 4, 6, 9, 10, 14> <1, 4, 6, 9, 10, 14> if x > amid then left ← mid + 1 else right ← mid if x = aleft then location ← left else location ← 0 9 > amid ? left mid right No (9 < 10) … <1, 4, 6, 9, 10, 14> © F. Pedone, N. Schiper 3 © F. Pedone, N. Schiper 4 1 Complexity Analysis Complexity Analysis – Usually time complexity considered Algorithm 2: Linear search – Space complexity can also be considered Initialize: i ← 1 while (i ≤ n and x ≠ a ) – RAM Model i i ← i + 1 • Constant time basic operations (add, sub, load, store…) if i ≤ n then location ← i else location ← 0 – Worst-case complexity measure Worst-case: element is at the end of the sequence or is • Estimates the time required for the most time-consuming input not present n iterations of loop ⋅c1 + c2 => Θ(n) of each size – Average-case complexity measure Avrg-case: element is at the middle of the sequence • Estimates the average time required for input of each size n/2 iterations of loop ⋅c + c => Θ(n) • Not always easy to see what is the average-case 1 2 © F. Pedone, N. Schiper 5 © F. Pedone, N. Schiper 6 Complexity Analysis Sorting Algorithms Algorithm 3: Binary search – General definition Initialize: left ← 1; right ← n • Putting a number of elements into a list in while (left < right) mid ← (left + right) / 2 which the elements are in increasing order if x > a then left mid + 1 else right mid mid ← ← – Input: if x = aleft then location ← left else location ← 0 • A sequence of n numbers <a , a , …, a > n elements 1 2 n … – Output: height = k • A permutation (reordering) <a’1, a’2, …, a’n> of the input sequence such that a’1 ≤ a’2 ≤ … ≤ a’n Average & worst-case analysis: k iteration of while loop ⋅ c1 + c2 => Θ(k) = Θ(log2 n) © F. Pedone, N. Schiper 7 © F. Pedone, N. Schiper 8 2 Sorting Algorithms Sorting Algorithms – Insertion sort – Insertion sort – General idea: Algorithm 3: Insertion sort • Same idea as what you do when cards are Input: unsorted sequence a1, a2, …, an distributed Output: sorted sequence a1, a2, …, an for j ← 2 to n distribute all cards key ← aj • Your left hand is initially empty i ← j - 1 while i > 0 and a > a insert at right position in left hand • Until all cards have been distributed i j a ← a – Take a card with right hand and insert it at the right i+1 i i ← i - 1 position in left hand ai ← key © F. Pedone, N. Schiper 9 © F. Pedone, N. Schiper 10 Insertion Sort - example Correctness proof of insertion sort j <3, 18, 4, 10, 7> <3, 18, 4, 10, 7> – Must prove that: j • The algorithm terminates • The algorithm sorts the input sequence <3, 18, 4, 10, 7> <3, 4, 18, 10, 7> j – Termination: • All operations of the algorithm take a finite amount of time <3, 4, 18, 10, 7> <3, 4, 10, 18, 7> • The algorithm executes a bounded number of loop iterations j <3, 4, 10, 18, 7> <3, 4, 7, 10, 18> © F. Pedone, N. Schiper 11 © F. Pedone, N. Schiper 12 3 Correctness proof of insertion sort Correctness proof of insertion sort – The algorithm sorts the input sequence Algorithm 3: Insertion sort for j ← 2 to n Algorithm 3: Insertion sort key ← aj for j ← 2 to n i ← j - 1 key ← aj while i > 0 and ai > aj i ← j - 1 ai+1 ← ai while i > 0 and a > a i j i ← i - 1 ai+1 ← ai ai ← key i ← i - 1 ai ← key – Prove loop invariant by induction • Prove a loop invariant: • Base step: j = 2 – a1 ... aj sorted at end of iteration j • Induction step: if true for 2 ≤ k < j then true for j © F. Pedone, N. Schiper 13 © F. Pedone, N. Schiper 14 Correctness proof of insertion sort Correctness proof of insertion sort Algorithm 3: Insertion sort Algorithm 3: Insertion sort for j ← 2 to n for j ← 2 to n key ← aj key ← aj i ← j - 1 i ← j - 1 while i > 0 and ai > aj while i > 0 and ai > aj ai+1 ← ai ai+1 ← ai i ← i - 1 i ← i - 1 ai ← key ai ← key – Base step (j = 2): a1 … a2 – Induction step: • Property holds by condition of while loop and last line • Property holds by condition of while loop and last line © F. Pedone, N. Schiper 15 © F. Pedone, N. Schiper 16 4 Correctness proof of insertion sort Sorting Algorithms – We proved that a1 … aj is sorted at end of – Bubble Sort iteration j Algorithm 4: Bubble sort Input: unsorted sequence a1, a2, …, an Output: sorted sequence a1, a2, …, an – The last iteration is when j = n for i ← 1 to n for j ← n to i + 1 if aj < aj-1 then interchange aj and aj-1 – Consequently, when the algorithm terminates, a1 … aj is sorted © F. Pedone, N. Schiper 17 © F. Pedone, N. Schiper 18 Bubble sort - example Complexity Analysis i Algorithm 3: Insertion sort <18, 4, 10, 7> <4, 18, 7, 10> for j ← 2 to n i key ← aj i ← j - 1 < 4, 18, 7, 10 > < 4, 7, 18, 10> while i > 0 and ai > aj ai+1 ← ai i i ← i - 1 ai ← key < 4, 7, 18, 10 > < 4, 7, 10, 18> i Worst-case: input is sorted in reverse order j 1 n " (n "1)2 < 4, 7, 10, 18 > <4, 7, 10, 18> ##c = c + 2c + ...+ (n "1)c = c => Θ(n2) j= 2 i=1 2 © F. Pedone, N. Schiper 19 © F. Pedone, N. Schiper 20 ! 5 Complexity Analysis Complexity Analysis Algorithm 3: Insertion sort Algorithm 4: Bubble sort for i 1 to n for j ← 2 to n ← for j ← n to i+1 key ← aj if a < a then interchange a and a i ← j - 1 j j-1 j j-1 while i > 0 and ai > aj ai+1 ← ai i ← i - 1 Average & worst-case: a key n n 2 i ← (n #1) " "c = (n #1)c + (n # 2)c + ...+ c = c => Θ(n2) i=1 j= i+1 2 Average-case: half of the elements in a1..aj-1 are greater than aj #( j"1)/ 2$ n # n "1$ n +1 n "1 % %c = c + c + 2c + 2c + ...+ & 'c ( ) => Θ(n2) ! j= 2 i=1 & 2 ' 2 2 © F. Pedone, N. Schiper 21 © F. Pedone, N. Schiper 22 ! Summary Complexity of algorithms – Common terminology Algorithm Avrg-case Worst-case Linear search Θ(n) Θ(n) Complexity Terminology O(1) Constant complexity Binary search Θ(log (n)) Θ(log (n)) 2 2 O(log n) Logarithmic complexity Insertion sort Θ(n2) Θ(n2) O(n) Linear complexity O(n log n) n log n complexity Bubble sort Θ(n2) Θ(n2) O(nb) Polynomial complexity O(bn), b > 1 Exponential complexity O(n!) Factorial complexity © F. Pedone, N. Schiper 23 © F. Pedone, N. Schiper 24 6.