Complexity of Algorithms
Total Page:16
File Type:pdf, Size:1020Kb
Complexity of Algorithms Lecture Notes, Spring 1999 Peter G´acs Boston University and L´aszl´oLov´asz Yale University 1 Contents 0 Introduction and Preliminaries 1 0.1 Thesubjectofcomplexitytheory . ... 1 0.2 Somenotationanddefinitions . .. 2 1 Models of Computation 3 1.1 Introduction................................... 3 1.2 Finiteautomata ................................ 5 1.3 TheTuringmachine .............................. 7 1.3.1 ThenotionofaTuringmachine. 7 1.3.2 UniversalTuringmachines. 9 1.3.3 Moretapesversusonetape . 11 1.4 TheRandomAccessMachine . 17 1.5 BooleanfunctionsandBooleancircuits. ...... 22 2 Algorithmic decidability 30 2.1 Introduction................................... 30 2.2 Recursive and recursively enumerable languages . ......... 31 2.3 Otherundecidableproblems. .. 35 2.3.1 Thetilingproblem .. .. .. .. .. .. .. .. .. .. .. 35 2.3.2 Famous undecidable problems in mathematics . .... 38 2.4 Computabilityinlogic . 40 2.4.1 Godel’sincompletenesstheorem. .. 40 2.4.2 First-orderlogic ............................ 42 2.4.3 A simple theory of arithmetic; Church’s Theorem . ..... 45 3 Computation with resource bounds 48 3.1 Introduction................................... 48 3.2 Timeandspace................................. 48 3.3 Polynomial time I: Algorithms in arithmetic . ....... 50 3.3.1 Arithmeticoperations . 50 3.3.2 Gaussianelimination. 52 3.4 PolynomialtimeII:Graphalgorithms . .... 55 3.4.1 Howisagraphgiven? . .. .. .. .. .. .. .. .. .. .. 55 3.4.2 Searchingagraph ........................... 55 3.4.3 Maximum bipartite matching and alternating paths . ..... 57 3.5 Polynomialspace................................ 60 4 General theorems on space and time complexity 63 4.1 Spaceversustime................................ 69 5 Non-deterministic algorithms 70 5.1 Non-deterministic Turing machines . ..... 70 5.2 Witnesses and the complexity of non-deterministic algorithms. 72 5.3 General results on nondeterministic complexity classes........... 74 2 5.4 ExamplesoflanguagesinNP . 77 5.4.1 Graphs ................................. 77 5.4.2 Arithmeticandalgebra . 79 5.4.3 Systemsoflinearinequalities . .. 81 5.5 NP-completeness ................................ 84 5.6 FurtherNP-completeproblems . .. 88 5.6.1 NP-completeproblemsforgraphs. 88 5.6.2 NP-complete problems in arithmetic and algebra . ..... 93 6 Randomized algorithms 99 6.1 Introduction................................... 99 6.2 Verifyingapolynomialidentity . .... 99 6.3 Primetesting ..................................103 6.4 Randomizedcomplexityclasses . .108 7 Information complexity: the complexity-theoretic notion of randomness112 7.1 Introduction...................................112 7.2 Informationcomplexity . 112 7.3 Thenotionofarandomsequence . 117 7.4 Kolmogorovcomplexity and data compression . .119 8 Pseudo-random numbers 124 8.1 Introduction ..................................124 8.2 Classicalmethods................................ 124 8.3 The notion of a psuedorandom number generator . .126 8.4 One-wayfunctions ...............................129 8.4.1 Candidates for one-way functions . .130 8.5 Random number generators and one-way functions . .130 8.6 Discretesquareroots. .. .. .. .. .. .. .. .. .. .. .. .. 133 9 An application of complexity: cryptography 136 9.1 Aclassicalproblem............................... 136 9.2 Asimplecomplexity-theoreticmodel . .136 9.3 Public-keycryptography. .137 9.4 TheRivest-Shamir-Adlemancode. .139 10 Parallel algorithms 142 10.1 Parallelrandomaccessmachines . .142 10.2TheclassNC ..................................146 11 Decision trees 150 11.1 Algorithmsusingdecisiontrees . .150 11.1.1 Binarysearch. .. .. .. .. .. .. .. .. .. .. .. .. .150 11.1.2 Finding a false coin with a one-armed scale . .150 11.1.3 Finding a false coin with a two-armed scale . .151 11.1.4 Sorting .................................151 3 11.1.5 Convexhull...............................152 11.2 Thenotionofdecisiontrees . .153 11.3 Nondeterministicdecisiontrees . .155 11.4 Lowerbounds on the depth of decision trees . .158 12 Communication complexity 163 12.1Examples ....................................163 12.2 Communication matrix and protocol-tree . .165 12.3 Non-deterministic communication complexity . .........169 12.4 Randomizedprotocols . 172 13 Interactive proofs 174 13.1 Howtosavethelastmoveinchess? . .174 13.2 How to check a password — without knowing it? . .175 13.3 How to use your password — without telling it? . .176 13.4 Howtoprovenon-existence? . .177 13.5 How to handle submissions that keep the main result secret? .......179 13.6 How to referee exponentially long papers?Probabilistically Checkable Proofs180 14 The complexity of algebraic computations 183 15 Circuit complexity 184 15.1Introduction................................... 184 15.2 Lower bound for the Majority Function . .185 15.3Monotonecircuits ............................... 187 0 0 Introduction and Preliminaries 0.1 The subject of complexity theory The need to be able to measure the complexity of a problem, algorithm or structure, and to obtain bounds and quantitive relations for complexity arises in more and more sciences: besides computer science, the traditional branches of mathematics, statistical physics, biology, medicine, social sciences and engineering are also confronted more and more frequently with this problem. In the approach taken by computer science, com- plexity is measured by the quantity of computational resources (time, storage, program, communication) used up by a particualr task. These notes deal with the foundations of this theory. Computation theory can basically be divided into three parts of different character. First, the exact notions of algorithm, time, storage capacity, etc. must be introduced. For this, different mathematical machine models must be defined, and the time and storage needs of the computations performed on these need to be clarified (this is generally measured as a function of the size of input). By limiting the available resources, the range of solvable problems gets narrower; this is how we arrive at different complexity classes. The most fundamental complexity classes provide an important classification of problems arising in practice, but (perhaps more surprisingly) even for those arising in classical areas of mathematics; this classification reflects the practical and theoretical difficulty of problems quite well. The relationship between different machine models also belongs to this first part of computation theory. Second, one must determine the resource need of the most important algorithms in various areas of mathematics, and give efficient algorithms to prove that certain important problems belong to certain complexity classes. In these notes, we do not strive for completeness in the investigation of concrete algorithms and problems; this is the task of the corresponding fields of mathematics (combinatorics, operations research, numerical analysis, number theory). Nevertheless, a large number of concrete algorithms will be described and analyzed to illustrate certain notions and methods, and to establish the complexity of certain problems. Third, one must find methods to prove “negative results”, i.e. for the proof that some problems are actually unsolvable under certain resource restrictions. Often, these questions can be formulated by asking whether certain complexity classes are different or empty. This problem area includes the question whether a problem is algorithmically solvable at all; this question can today be considered classical, and there are many important results concerining it; in particular, the decidability or undecidablity of most concrete problems of interest is known. The majority of algorithmic problems occurring in practice is, however, such that algorithmic solvability itself is not in question, the question is only what resources must be used for the solution. Such investigations, addressed to lower bounds, are very difficult and are still in their infancy. In these notes, we can only give a taste of this sort of results. In particular, we discuss complexity notions like communication complexity or decision tree complexity, where by focusing only on one type of rather special resource, we can give a more complete analysis of basic complexity classes. It is, finally, worth noting that if a problem turns out to be “difficult” to solve, this 1 is not necessarily a negative result. More and more areas (random number generation, communication protocols, cryptography, data protection) need problems and structures that are guaranteed to be complex. These are important areas for the application of complexity theory; from among them, we will deal with random number generation and cryptography, the theory of secret communication. 0.2 Some notation and definitions A finite set of symbols will sometimes be called an alphabet. A finite sequence formed from some elements of an alphabet Σ is called a word. The empty word will also be considered a word, and will be denoted by . The set of words of length n over Σ is denoted by Σn, the set of all words (including∅ the empty word) over Σ is denoted by Σ∗. A subset of Σ∗, i.e. , an arbitrary set of words, is called a language. Note that the empty language is also denoted by but it is different, from the language containing only the empty word. ∅ Let us{∅} define some orderings of the set of words. Suppose that an ordering of the elements of Σ is given. In the lexicographic ordering of the elements of Σ∗, a word α precedes a word β if either α is a prefix (beginning segment) of β or the first letter