CSC 308 (ALGORITHMS AND COMPLEXITY ANALYSIS) 1 DISCLAIMER The contents of this document are intended for leaning purposes at the undergraduate level. The materials are from different sources including the internet and the contributor do not in any way claim authorship or ownership of them. 2 Introduction of the course This course applies design and analysis techniques to numeric and nonnumeric algorithms which act on data structures. Design is emphasized so that the student will be able to develop new algorithms. Analysis of algorithms is concerned with the resources an algorithm must use to reach a solution. Only theoretical techniques of analysis are covered. Topics include introduction to algorithm, Basic algorithmic analysis: Asymptotic analysis of Upper and average complexity bounds (Asymptotic complexity, sorting and searching); standard Complexity Classes Time and space tradeoffs in algorithms analysis recursive algorithms. Algorithmic Strategies: Fundamental computing algorithms: Numerical algorithms, sequential and binary search algorithms; sorting algorithms, Binary Search trees, Hash tables, graphs & its representation. 3 Objectives By the end of the semester, the student should be able to: (1) describe algorithm and types of algorithms (2) describe and recognize the use of analysis of algorithms such as asymptotic analysis of upper and average complexity bounds (3) recognize the use of different types of algorithms such as sorting, searching, recursive algorithms, etc, and use these algorithms / methods in solving problems (4) Determine asymptotic growth rates for algorithms (5) apply graphs and its theorem/representation to algorithm analysis 4 Course Overview Basic algorithmic analysis: Asymptotic analysis of Upper and average complexity bounds; standard Complexity Classes Time and space tradeoffs in algorithms analysis recursive algorithms. Algorithmic Strategies: Fundamental computing algorithms: Numerical algorithms, sequential and binary search algorithms; sorting algorithms, Binary Search tress, Hash tables, graphs & its representation. 5 Course Plan Week 1: Introduction, basic algorithm analysis, algorithm complexity, comparing algorithm Week 2: Types of algorithms Week 3: Asymptotic Complexity, Asymptotic analysis of Upper and average complexity bounds, Week 4: Standard Complexity Classes Time and space tradeoffs in algorithms analysis Week 5: Revision and 1st test Week 6: searching (binary search) and sorting algorithms Week 7: Recursive algorithms. Week 8: other fundamental computing algorithms: Numerical algorithms, sequential algorithms, Week 9: Binary Search tress, Hash tables, Week 10: Graphs & its representation. Week 11: revision and 2nd test 6 Lecture One Introduction It is very common for beginning computer science students to compare their programs with one another. You may also have noticed that it is common for computer programs to look very similar, especially the simple ones. An interesting question often arises. When two programs solve the same problem but look different, is one program better than the other? In order to answer this question, we need to remember that there is an important difference between a program and the underlying algorithm that the program is representing. As we stated in Chapter 1, an algorithm is a generic, step-by-step list of instructions for solving a problem. It is a method for solving any instance of the problem such that given a particular input, the algorithm produces the desired result. A program, on the other hand, is an algorithm that has been encoded into some programming language. There may be many programs for the same algorithm, depending on the programmer and the programming language being used. 7 Algorithms What is an algorithm? An algorithm is a finite set of precise instructions for performing a computation or for solving a problem. It can also be defined as step-by-step instructions to accomplish a task or solve a problem Format & Detail level depends on user 8 Algorithms Properties of algorithms: • Input from a specified set, • Output from a specified set (solution), • Definiteness of every step in the computation, • Correctness of output for every possible input, • Finiteness of the number of calculation steps, • Effectiveness of each calculation step and • Generality for a class of problems. 9 Algorithm Examples We will use a pseudocode to specify algorithms, which slightly reminds us of Basic and Pascal. Example: an algorithm that finds the maximum element in a finite sequence procedure max(a1, a2, …, an: integers) max := a1 for i := 2 to n if max < ai then max := ai {max is the largest element} 10 Algorithm Examples Another example: a linear search algorithm, that is, an algorithm that linearly searches a sequence for a particular element. procedure linear_search(x: integer; a1, a2, …, an: integers) i := 1 while (i n and x ai) i := i + 1 if i n then location := i else location := 0 {location is the subscript of the term that equals x, or is zero if x is not found} 11 Analysis of Algorithms In computer science, the analysis of algorithms is the determination of the computational complexity of algorithms, that is the amount of time, storage and/or other resources necessary to execute them. Usually, this involves determining a function that relates the length of an algorithm's input to the number of steps it takes (its time complexity) or the number of storage locations it uses (its space complexity). An algorithm is said to be efficient when this function's values are small. Since different inputs of the same length may cause the algorithm to have different behavior, the function describing its performance is usually an upper bound on the actual performance, determined from the worst case inputs to the algorithm. 12 The term "analysis of algorithms" was coined by Donald Knuth. Algorithm analysis is an important part of a broader computational complexity theory, which provides theoretical estimates for the resources needed by any algorithm which solves a given computational problem. These estimates provide an insight into reasonable directions of search for efficient algorithms. 13 Efficiency of Algorithms Different algorithms for a single task may have different efficiency -Must be able to evaluate efficiency (speed) of computer algorithms 14 Complexity = Speed = Efficiency Complexity is the number of basic operations required by an algorithm Will 2 algorithms with same complexity take the same actual amount of time to run? 15 Complexity Examples Do 10 times Read x, y, z Read x, y, z x = y + z x = y + z Print x Print x 16 Complexity Examples Do 10 times Do n times Read x, y, z Read x, y, z x = y + z x = y + z Print x Print x 17 Complexity In general, we are not so much interested in the time and space complexity for small inputs. For example, while the difference in time complexity between linear and binary search is meaningless for a sequence with n = 10, it is gigantic for n = 230. 18 Complexity For example, let us assume two algorithms A and B that solve the same class of problems. The time complexity of A is 5,000n, the one for B is 1.1n for an input with n elements. For n = 10, A requires 50,000 steps, but B only 3, so B seems to be superior to A. For n = 1000, however, A requires 5,000,000 steps, while B requires 2.51041 steps. 19 Complexity Comparison: time complexity of algorithms A and B Input Size Algorithm A Algorithm B n 5,000n 1.1n 10 50,000 3 100 500,000 13,781 1,000 5,000,000 2.51041 1,000,000 5109 4.81041392 20 Complexity This means that algorithm B cannot be used for large inputs, while algorithm A is still feasible. So what is important is the growth of the complexity functions. The growth of time and space complexity with increasing input size n is a suitable measure for the comparison of algorithms. 21 Comparing Algorithms Want to choose the “best” algorithm for tasks Generally, best = fastest But there are other considerations – Hardware – Size of data set – Data Structure But a standard measurement is needed for this!!! 22 The Need for Analysis Algorithms are often quite different from one another, though the objective of these algorithms are the same. For example, we know that a set of numbers can be sorted using different algorithms. Number of comparisons performed by one algorithm may vary with others for the same input. Hence, time complexity of those algorithms may differ. At the same time, we need to calculate the memory space required by each algorithm. 23 Analysis of algorithm is the process of analyzing the problem-solving capability of the algorithm in terms of the time and size required (the size of memory for storage while implementation). However, the main concern of analysis of algorithms is the required time or performance. Generally, we perform the following types of analysis − • Worst-case − The maximum number of steps taken on any instance of size a. • Best-case − The minimum number of steps taken on any instance of size a. • Average case − An average number of steps taken on any instance of size a. 24 To solve a problem, we need to consider time as well as space complexity as the program may run on a system where memory is limited but adequate space is available or may be vice-versa. In this context, if we compare bubble sort and merge sort. Bubble sort does not require additional memory, but merge sort requires additional space. Though time complexity of bubble sort is higher compared to merge sort, we may need to apply bubble sort if the program needs to run in an environment, where memory is very limited. 25 Bubble Sort def bubbleSort(alist): for passnum in range(len(alist)-1,0,-1): for i in range(passnum): if alist[i]>alist[i+1]: temp = alist[i] alist[i] = alist[i+1] alist[i+1] = temp alist = [54,26,93,17,77,31,44,55,20] bubbleSort(alist) print(alist) 26 Merge Sort Conceptually, a merge sort works as follows: • Divide the unsorted list into n sublists, each containing 1 element (a list of 1 element is considered sorted).