Searching and Sorting Algorithms, Complexity Analysis

– General definition • Locate an element x in a list of distinct elements a , a , …, a , or determine that it is not in the list Searching and Sorting Algorithms, 1 2 n Complexity Analysis – Linear search Algorithm 2: Linear search

Input: unsorted sequence a1, a2, …, an position of target value x Output: subscript of entry equal to target value; 0 if not found Initialize: i ← 1

while (i ≤ n and x ≠ ai) i ← i + 1 if i ≤ n then location ← i else location ← 0

Searching Algorithms Searching Algorithms

– Binary search – Binary search • Requires input sequence to be sorted General Idea: Algorithm 3: Binary search Input: sorted sequence a1, a2, …, an Find 9 in the sequence: <1, 4, 6, 9, 10, 14> position of target value x Output: subscript of entry equal to target value; 0 if not found Initialize: left 1; right n left mid right left mid right ← ← 9 > amid ? while (left < right) Yes (9 > 6) mid ← (left + right) / 2 <1, 4, 6, 9, 10, 14> <1, 4, 6, 9, 10, 14> if x > amid then left ← mid + 1 else right ← mid

if x = aleft then location ← left else location ← 0 9 > amid ? left mid right No (9 < 10) … <1, 4, 6, 9, 10, 14>

Complexity Analysis Complexity Analysis

– Usually time complexity considered Algorithm 2: Linear search – Space complexity can also be considered Initialize: i ← 1 while (i ≤ n and x ≠ a ) – RAM Model i i ← i + 1 • Constant time basic operations (add, sub, load, store…) if i ≤ n then location ← i else location ← 0 – Worst-case complexity measure Worst-case: element is at the end of the sequence or is • Estimates the time required for the most time-consuming input not present n iterations of loop ⋅c1 + c2 => Θ(n) of each size – Average-case complexity measure Avrg-case: element is at the middle of the sequence • Estimates the average time required for input of each size n/2 iterations of loop ⋅c + c => Θ(n) • Not always easy to see what is the average-case 1 2

Complexity Analysis Sorting Algorithms

Algorithm 3: Binary search – General definition Initialize: left ← 1; right ← n • Putting a number of elements into a list in while (left < right) mid ← (left + right) / 2 which the elements are in increasing order if x > a then left mid + 1 else right mid mid ← ← – Input: if x = aleft then location ← left else location ← 0 • A sequence of n numbers n elements 1 2 n … – Output: height = k • A permutation (reordering) of the input sequence such that a’1 ≤ a’2 ≤ … ≤ a’n

Average & worst-case analysis:

k iteration of while loop ⋅ c1 + c2 => Θ(k) = Θ(log2 n)

Sorting Algorithms Sorting Algorithms

– Insertion sort – Insertion sort – General idea: Algorithm 3: Insertion sort • Same idea as what you do when cards are Input: unsorted sequence a1, a2, …, an

distributed Output: sorted sequence a1, a2, …, an for j ← 2 to n distribute all cards

key ← aj • Your left hand is initially empty i ← j - 1 while i > 0 and a > a insert at right position in left hand • Until all cards have been distributed i j a ← a – Take a card with right hand and insert it at the right i+1 i i ← i - 1 position in left hand ai ← key

Insertion Sort - example Correctness proof of insertion sort j

 <3, 18, 4, 10, 7> <3, 18, 4, 10, 7> – Must prove that: j • The algorithm terminates • The algorithm sorts the input sequence  <3, 18, 4, 10, 7> <3, 4, 18, 10, 7>

j – Termination: • All operations of the algorithm take a finite amount of time  <3, 4, 18, 10, 7> <3, 4, 10, 18, 7> • The algorithm executes a bounded number of loop iterations j

 <3, 4, 10, 18, 7> <3, 4, 7, 10, 18>

Correctness proof of insertion sort Correctness proof of insertion sort

– The algorithm sorts the input sequence Algorithm 3: Insertion sort for j ← 2 to n Algorithm 3: Insertion sort key ← aj for j ← 2 to n i ← j - 1 key ← aj while i > 0 and ai > aj i ← j - 1 ai+1 ← ai while i > 0 and a > a i j i ← i - 1 ai+1 ← ai ai ← key i ← i - 1

ai ← key – Prove loop invariant by induction • Prove a loop invariant: • Base step: j = 2 – a1 ... aj sorted at end of iteration j • Induction step: if true for 2 ≤ k < j then true for j

Correctness proof of insertion sort Correctness proof of insertion sort

Algorithm 3: Insertion sort Algorithm 3: Insertion sort for j ← 2 to n for j ← 2 to n

key ← aj key ← aj i ← j - 1 i ← j - 1

while i > 0 and ai > aj while i > 0 and ai > aj

ai+1 ← ai ai+1 ← ai i ← i - 1 i ← i - 1

ai ← key ai ← key

– Base step (j = 2): a1 … a2 – Induction step: • Property holds by condition of while loop and last line • Property holds by condition of while loop and last line

Correctness proof of insertion sort Sorting Algorithms

– We proved that a1 … aj is sorted at end of – Bubble Sort iteration j Algorithm 4: Bubble sort

Input: unsorted sequence a1, a2, …, an

Output: sorted sequence a1, a2, …, an – The last iteration is when j = n for i ← 1 to n for j ← n to i + 1

if aj < aj-1 then interchange aj and aj-1 – Consequently, when the algorithm terminates,

a1 … aj is sorted

Bubble sort - example Complexity Analysis i Algorithm 3: Insertion sort

 <18, 4, 10, 7> <4, 18, 7, 10> for j ← 2 to n

i key ← aj i ← j - 1

 < 4, 18, 7, 10 > < 4, 7, 18, 10> while i > 0 and ai > aj

ai+1 ← ai i i ← i - 1

ai ← key  < 4, 7, 18, 10 > < 4, 7, 10, 18> i Worst-case: input is sorted in reverse order j 1 n " (n "1)2  < 4, 7, 10, 18 > <4, 7, 10, 18> ##c = c + 2c + ...+ (n "1)c = c => Θ(n2) j= 2 i=1 2

Complexity Analysis Complexity Analysis

Algorithm 3: Insertion sort Algorithm 4: Bubble sort for i 1 to n for j ← 2 to n ← for j ← n to i+1 key ← aj if a < a then interchange a and a i ← j - 1 j j-1 j j-1

while i > 0 and ai > aj

ai+1 ← ai i ← i - 1 Average & worst-case: a key n n 2 i ← (n #1) " "c = (n #1)c + (n # 2)c + ...+ c = c => Θ(n2) i=1 j= i+1 2 Average-case: half of the elements in a1..aj-1 are greater

than aj #( j"1)/ 2$ n # n "1$ n +1 n "1 % %c = c + c + 2c + 2c + ...+ & 'c ( ) => Θ(n2) ! j= 2 i=1 & 2 ' 2 2

Summary Complexity of algorithms

– Common terminology Algorithm Avrg-case Worst-case

Linear search Θ(n) Θ(n) Complexity Terminology O(1) Constant complexity Binary search Θ(log (n)) Θ(log (n)) 2 2 O(log n) Logarithmic complexity

Insertion sort Θ(n2) Θ(n2) O(n) Linear complexity O(n log n) n log n complexity Bubble sort Θ(n2) Θ(n2) O(nb) Polynomial complexity O(bn), b > 1 Exponential complexity O(n!) Factorial complexity