Computational Complexity for Physicists

L I M I T S O F C O M P U T A T I O N

The theory of computational complexity has some interesting links to physics, in particular to quantum computing and statistical mechanics. This article contains an informal introduction to this theory and its links to physics.

ompared to the traditionally close rela- mentation as well as the computer on which the tionship between physics and mathe- program is running. matics, an exchange of ideas and meth- The theory of computational complexity pro- ods between physics and computer vides us with a notion of complexity that is Cscience barely exists. However, the few interac- largely independent of implementation details tions that have gone beyond Fortran program- and the computer at hand. Its precise definition ming and the quest for faster computers have been requires a considerable formalism, however. successful and have provided surprising insights in This is not surprising because it is related to a both fields. This is particularly true for the mutual highly nontrivial question that touches the foun- exchange between statistical mechanics and the dation of mathematics: What do we mean when we theory of computational complexity. Here, I discuss say a problem is solvable? Thinking about this this exchange in a manner directed at physicists question leads to Gödel’s incompleteness theo- with little or no knowledge of the theory. rem, Turing machines, and the Church-Turing thesis on computable functions. Here we adopt a more informal, pragmatic The measure of complexity viewpoint. A problem is solvable if a computer The branch of theoretical computer science program written in your favorite programming known as computational complexity is con- language can solve it. Your program’s running cerned with classifying problems according to time or time complexity must then be defined with the computational resources required to solve some care to serve as a meaningful measure of them (for additional information about this field, the problem’s complexity. see the “Related Works” sidebar). What can be measured (or computed) is the time that a par- Time complexity ticular algorithm uses to solve the problem. This In general, running time depends on a prob- time, in turn, depends on the algorithm’s imple- lem’s size and on the specific input data—the instance. Sorting 1,000 numbers takes longer than sorting 10 numbers. Some sorting algorithms 1521-9615/02/$17.00 © 2002 IEEE run faster if the input data is partially sorted already. To minimize the dependency on the spe- STEPHAN MERTENS cific instance, we consider the worst-case time Otto-von-Guericke University, Magdeburg complexity T(n):

MAY/JUNE 2002 31

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. strongly on the algorithm’s implementation de- Related Work tails—smart programmers and optimizing com- For introductions into the ﬁeld of computational complexity, see pilers will try to reduce it. Therefore, rather than

E.L. Lawler et al., eds., The Traveling Salesman Problem, Wiley-Interscience Series consider the precise number T(n) of elementary in Discrete Mathematics and Optimization, John Wiley & Sons, New York, 1985. operations, we only consider the asymptotic be- H.R. Lewis and C.H. Papadimitriou, “The Efficiency of Algorithms,” Scientific havior of T(n) for large values of n as the Lan- American, vol. 109, no. 1, Jan. 1978, pp. 96–109. dau symbols and Θ denote: C.H. Papadimitriou and K. Steiglitz, Combinatorial Optimization, Prentice-Hall, O Englewood Cliffs, N.J., 1982. • We say T(n) is of order at most g(n) and write T(n) = (g(n)) if positive constants c and n ex- For a deep understanding of the field, read the following classic O 0 textbooks: ist such that T(n) ≤ cg(n) for all n ≥ n0. • We say T(n) is of order g(n) and write T(n) M.R. Garey and D.S. Johnson, Computers and Intractability: A Guide to the Theory of NP-Completeness, W.H. Freeman, New York, 1997. =Θ(g(n)) if positive constants c1, c2, and n0 exist C.H. Papadimitriou, Computational Complexity, Addison-Wesley, Reading, such that c1g(n) ≤ T(n) ≤ c2g(n) for all n ≥ n0. Mass., 1994. Multiplying two n × n matrixes requires n3 The complexity of Ising spin systems is discussed in: multiplications, according to the textbook for- B. Hayes, “Computing Science: The World in a Spin,” American Scientist, vol. 88, no. mula. However, this does not mean that the 5, Sept./Oct. 2000, pp. 2384–388; www.amsci.org/amsci/issues/comsci00/ problem of multiplying two n × n matrices has compsci2000-09.html. complexity Θ(n3). The textbook formula is a par- For an entertaining introduction (written by a physicist) to ticular algorithm, and an algorithm’s time com- Gödel’s incompleteness theorem, Turing machines, and the plexity is only an upper bound for a problem’s in- Church-Turing Thesis on computable functions, read herent complexity. In fact, researchers have found faster matrix multiplication algorithms with com- R. Penrose, The Emperor’s New Mind, Oxford Univ. Press, 1989. plexity (nα) and α < 3 during the last decades— O 1 To learn more about quantum parallelism, see www.qubit.org or the current record being α = 2.376. Because the product matrix has n2 entries, α cannot be smaller D. Aharonov, “Quantum Computation,” Ann. Rev. Computational Physics VI, D. Stauffer, ed., World Scientific, 1998. than 2; it is an open question whether an algorithm can achieve this lower bound. Phase transitions in computational complexity are discussed in: A problem where the upper bound from algo-

O. Dubois et al., eds., “Phase Transitions in Combinatorial Problems,” special rithmic complexity meets an inherent lower issue of Theor. Comp. Sci., vol. 265, nos. 1–2, 2001. bound is sorting n items. Under the general as- B. Hayes, “Computing Science: The Easiest Hard Problem,”American Scientist, sumption that comparisons between pairs of vol. 90, no. 2, Mar./Apr. 2002, pp. 113–117; www.amsci.org/amsci/issues/ items are the only source of information about comsci02/compsci2002-03.html. them, Θ(n log n) is a lower bound for the num- B. Hayes, “Computing Science: Can’t Get No Satisfaction,” American Scientist, vol. 85, no. 2, Mar./Apr. 1997, pp. 108–112; www.amsci.org/amsci/issues/ ber of comparisons to sort n items in the worst 2 Comsci97/compsci9703.html. case. This bound is met by algorithms such as “heapsort” or “mergesort.”

Problem size T(n) = max t(x), (1) Our measure of time complexity still depends x = n on the somewhat ambiguous notion of problem where t(x) is the algorithm’s running time for in- size. In the matrix multiplication example, we put data x (in arbitrary units), and the maximum tacitly took the number n of rows of one input is taken over all problem instances of size n. The matrix as the “natural” measure of size. Using worst-case time is an upper bound for the ob- the number of elements m = n2 instead speeds up servable running time. the (n3) algorithm to (m3/2) without chang- O O A measure of time complexity should be based ing a single line of program code. An unam- on a unit of time that is independent of the spe- biguous deﬁnition of problem size is required to ciﬁc CPU’s clock rate. Such a unit is provided by compare algorithms. the time it takes to perform an elementary op- In computational complexity, all problems eration such as adding two integer numbers. solvable by a polynomial algorithm—that is, an Measuring the time in this unit means counting algorithm with time complexity Θ(nk) for some the number of elementary operations your algo- k—are lumped together and called tractable. rithm executes. This number in turn depends Problems that can only solvable by algorithms

32 COMPUTING IN SCIENCE & ENGINEERING

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. n with nonpolynomial running time, such as Θ(2 ) 8 7 or Θ(n!), are also lumped together and called in- 4 9 tractable. There are practical as well as theoreti- 2 cal reasons for this rather coarse classification. 14 11 4 One of the theoretical advantages is that it does 7 6 not distinguish between the (n3) and (m3/2) 8 10 O O algorithm example from above; hence, we can 2 afford some sloppiness and stick with our am- 1 biguous natural measure of problem size. Figure 1. A weighted graph and its minimum spanning tree (shaded edges).

Tractable and intractable problems The terms tractable and intractable for problems with polynomial and exponential algo- a spanning subgraph) and whose edges have rithms refer to the fact that an exponential al- minimum total weight. Your subgraph should gorithm means a hard limit for the accessible not contain cycles because you can always re- problem sizes. Suppose that, with your current move an edge from a cycle, keeping all nodes equipment, you can solve a problem of size n connected and reducing the cost. A graph just within your schedule. If your algorithm has without cycles is a tree, so what you are look- complexity Θ(2n), a problem of size n + 1 needs ing for is a minimum spanning tree (MST) in a twice the time, pushing you off schedule. The weighted graph (see Figure 1). So, given a increase in time caused by an Θ(n) or Θ(n2) al- weighted graph G = (V, E) with nonnegative gorithm, on the other hand, is far less dramatic weights, find a spanning tree T ⊆ G with min- and can easily be compensated for by upgrad- imum total weight. ing your hardware. How do you find an MST? A naive approach You might argue that a Θ(n100) algorithm out- is to generate all possible trees with n vertices performs a Θ(2n) algorithm only for problem and keep the one with minimal weight. The enu- sizes that will never occur in your application. A meration of all trees can be done using Prüfer polynomial algorithm for a problem usually goes codes,3 but Cayley’s formula tells us that there hand in hand with a mathematical insight into that are nn–2 different trees with n vertices. Already problem, which lets you find a polynomial algo- for n = 100 there are more trees than atoms in rithm with a small degree, typically Θ(nk), k = 1, the observable universe. Hence, exhaustive enu- 2, or 3. Polynomial algorithms with k > 10 are meration is prohibitive for all but the smallest rare and arise in rather esoteric problems. It is trees. The mathematical insight that turns MST this mathematical insight (or rather the lack of into a tractable problem is this: it) that turns intractable problems into a challenge for algorithm designers and computer sci- Lemma: Let U ⊂ V be any subset of the ver- entists. Let’s look at a few examples of tractable tices of G = (V, E), and let e be the edge with the and intractable problems to learn more about smallest weight of all edges connecting U and V – what separates one from the other. U. Then e is part of the MST. Proof (by contradiction): Suppose T is an MST Tractable trees not containing e. Let e = (u, v), with u ∈ U and v Consider the following problem from network ∈ V – U. Then, because T is a spanning tree, it design. You have a business with several offices, contains a unique path from u to v that together and you want to lease phone lines to connect with e forms a cycle in G. This path must include them. The phone company charges different another edge f connecting U and V – U. Now T + amounts to connect different pairs of cities, and e – f is another spanning tree with less total your task is to select a set of lines that connects weight than T. So T was not an MST. all your offices with a minimum total cost. In mathematical terms, the cities and the lines The lemma lets an MST grow edge by edge— between them form the vertices V and edges E using Prim’s algorithm, for example: of a weighted graph G = (V, E), the weight of an edge being the corresponding phone line’s leas- Prim(G) ing costs. Your task is to find a subgraph that Input: weighted graph G(V, E) connects all vertices in the graph (for example, Output: minimum spanning tree T ⊆ G

MAY/JUNE 2002 33

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. n c n (π) = ∑ diπ (i). (2) i=1

The TSP is probably the most famous optimization problem (see www.keck.caam.rice.edu/tsp). Finding good solutions, even to large problems, is not difficult, but how can we find the best solution for a given instance? There are (n (a) (b) – 1)! cyclic permutations, and calculating the length of a single tour Figure 2. Same instance, different problems: A valid configuration of (a) the Traveling takes (n). So, an exhaustive O Salesman Problem and (b) the Assignment Problem. Whereas the Assignment search has complexity (n!). O Problem can be solved in polynomial time, the TSP is intractable. Again this approach is limited to very small instances. begin Is there a mathematical insight that provides Let T be a single vertex v from G a shortcut to the optimum solution, such as for while T has less than n vertices an MST? Nobody knows. Despite many efforts, find the minimum edge researchers have not found a polynomial algo- connecting T to G – T rithm for the TSP. There are some smart and ef- add it to T ficient (that is, polynomial) algorithms that find end good solutions but do not guarantee yielding the end optimum solution.4 According to this definition, the TSP is intractable. The precise time complexity of Prim’s algorithm Why is the TSP intractable? Again, nobody depends on the data structure used to organize the knows. There is no proof that excludes the exis- edges, but in any case, (n2log n) is an upper bound tence of a polynomial algorithm for the TSP, so O (faster algorithms are discussed elsewhere3). maybe someday somebody will come up with a Equipped with such a polynomial algorithm, you polynomial algorithm and the corresponding can find MSTs with thousands of nodes within sec- mathematical insight. This is very unlikely, how- onds on a personal computer. Compare this to an ever, as we will see soon. exhaustive search. According to our definition, The TSP’s intractability astonishes all the finding an MST is a tractable problem. more considering the tractability of a very similar, almost identical problem, the Assignment Intractable itineraries problem: Encouraged by the efficient algorithm for an MST, let us now investigate a similar problem. Assignment: Given an n × n cost matrix with el- Your task is to plan an itinerary for a traveling ements dij ≥ 0, find a permutation π : [1...n] → salesman who must visit n cities. You are given a [1...n] that minimizes map with all cities and the distances between n them. In what order should the salesman visit c (π) = d . the cities to minimize the total distance traveled? n ∑ iπ (i) (3) i=1 You number the cities arbitrarily and write down the distance matrix (dij), where dij denotes the The only difference between the TSP and As- distance between city i and city j. A tour is given signment is that the latter allows all permuta- by a cyclic permutation π: [1...n]→[1...n], where tions on n items, not just the cyclic ones. If dij π(i) denotes the successor of city i. Your prob- denotes distances between cities, Assignment lem can be defined as corresponds to total tour length minimization for a variable number of salesmen, each travel- The Traveling Salesman Problem (TSP): Given ing his own subtour (see Figure 2). an n × n distance matrix with elements dij ≥ 0, The classical application of Assignment is the find a cyclic permutation π : [1...n] → [1...n] that assignment of n tasks to n workers, subject to the minimizes constraint that each worker is assigned exactly

34 COMPUTING IN SCIENCE & ENGINEERING

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. C Figure 3. The seven bridges c d C of Königsberg, (a) as drawn e c g in Leonhard Euler’s paper d from 17366 and (b) e represented as a graph. In A D A D the graph, the riverbanks b and islands are condensed to f points (vertices), and each of a the bridges is drawn as a line c f (edge). B b (a) (b) B one task. Let dij denote the cost of worker I per- We can solve MST(D) in polynomial time by forming task j, and π(i) denote the task assigned simply solving the optimization variant MST to worker i. Assignment is the problem of mini- and comparing the result to the parameter B. mizing the total cost. For the TSP(D), this approach does not help. In There are n! possible assignments of n tasks to fact, we see later that a polynomial algorithm ex- n workers—hence, exhaustive enumeration is ists for the TSP(D) if and only if there exists a again prohibitive. In contrast to the TSP, how- polynomial algorithm for the TSP. It seems as if ever, we can solve Assignment in polynomial we cannot learn more about the gap between time—for example, using the (n3) Hungarian tractable and intractable problems by consider- O algorithm.5 Compared to an MST, the algorithm ing decision variants of optimization problems. and the underlying mathematical insight are a So let’s look at other decision problems not de- bit more involved and not discussed here. rived from optimization problems.

Eulerian circuits Decision problems Our ﬁrst genuine decision problem dates back So far, we have discussed optimization prob- to the 18th century, where in the city of Königs- lems: solving MST, TSP, or Assignment implies berg (now Kaliningrad), seven bridges crossed that we compare an exponential number of feasi- the river Pregel and its two arms (see Figure 3). ble solutions with each other and pick the opti- A popular puzzle of the time asked if it were pos- mum solution. Exhaustive search does this sible to walk through the city crossing each of explicitly; polynomial shortcuts implicitly. By in- the bridges exactly once and returning home. vestigating simpler problems whose solutions are Leonhard Euler solved this puzzle in 1736.6 recognizable without explicit or implicit compar- First, he recognized that to solve the problem, ison to all feasible solutions, we might learn more the only thing that matters is the pattern of in- about the barrier that separates tractable from in- terconnections of the banks and islands—a graph tractable problems. So, let’s consider decision prob- G = (V, E) in modern terms. The graph corre- lems, whose solution is either “yes” or “no.” sponding to the puzzle of the Königsberg We can turn any optimization problem into a bridges has four vertices for the two banks and decision problem by adding a bound B to the in- two islands and seven edges for the bridges (see stance. For example, Figure 3). Euler’s paper on the Königsberg bridges is the birth of graph theory. MST (decision): Given a weighted graph G = To generalize the Königsberg bridges prob- (V, E) with nonnegative weights and a number lem, we need some terminology from graph the- B ≥ 0, does G contain a spanning tree T with to- ory.3 A walk in a graph G = (V, E) is an alternat- tal weight ≤ B? ing sequence of vertices v ∈ V and edges (v, v′) ∈ TSP (decision): Given an n × n distance matrix E: v1, (v1, v2), v2, (v2, v3), ..., (vl–1, vl), vl. Note that with elements dij ≥ 0 and a number B ≥ 0, is there the sequence begins and ends with a vertex, and a tour π with length ≤ B? each edge is incident with the vertices immediately preceding and succeeding it. A walk is In a decision problem, the feasible solutions are termed closed if vl = v1; otherwise, it is open. A not evaluated relative to each other but to an ab- walk is called a trail if all its edges are distinct, solute criterion: a tour in the TSP either has and a closed trail is called a circuit. What the length ≤ B or it doesn’t. strollers in Königsberg tried to ﬁnd was a circuit

MAY/JUNE 2002 35

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. Theorem: A connected graph G = (V, E) contains an Eulerian circuit if and only if the degree of every vertex v ∈ V is even.

Euler’s theorem lets us devise a polynomial algorithm for the Eulerian Circuit: Check the degree of every vertex in the graph. If one vertex has an odd degree, return “no.” If all vertices have an even degree, return “yes.” This algorithm’s running time depends on the graph’s en- coding. If G = (V, E) is encoded as a |V| × |V| adjacency matrix with entries aij = number of (a) (b) edges connecting vi and vj, the running time is (|V|2). O Figure 4. Sir Hamilton’s Icosian game: (a) Find a route along the Thanks to Euler, the Eulerian Circuit is a edges of the dodecahedron, passing each corner exactly once and tractable problem. The burghers of Königsberg, returning to the starting corner. (b) A solution is indicated (shaded on the other hand, had to learn from Euler that edges) in the planar graph that is isomorphic to the dodecahedron. they would never find a walk through their hometown crossing each of the seven bridges ex- that contains all edges. In honor of Euler, such a actly once. circuit is called an Eulerian circuit. We can now define the generalization of the Hamiltonian cycles Königsberg bridges problem: Another decision problem is associated with the mathematician and Astronomer Royal of Ire- Eulerian Circuit: Given a graph G = (V, E), land, Sir William Rowan Hamilton. In 1859, does G contain an Eulerian circuit? Hamilton put on the market a new puzzle called the Icosian game (see Figure 4). Obviously, this is a decision problem. The an- Generalizing the Icosian game calls for some swer is either “yes” or “no,” and we can check to more definitions from graph theory: A closed see whether a circuit is Eulerian without resort- walk in a graph is called a cycle if all its vertices (ex- ing to all possible circuits. cept the first and the last) are distinct. A Hamil- Once again, an exhaustive search would solve tonian cycle is one that contains all vertices of a this problem, but Euler noticed the intractability graph. The generalization of the Icosian game of this approach. More than 200 years before the then reads advent of computers, he wrote, Hamiltonian Cycle: Given a graph G = (V, E), The particular problem of the seven bridges of does G contain a Hamiltonian cycle? Königsberg could be solved by carefully tabulat- ing all possible paths, thereby ascertaining by in- There is a certain similarity between the Euler- spection which of them, if any, met the require- ian Circuit and Hamiltonian Cycle. In the for- ment. This method of solution, however, is too mer, we must pass each edge once—in the lat- tedious and too difficult because of the large ter, each vertex once. Despite this resemblance, number of possible combinations, and in other the two problems represent entirely different de- problems where many more bridges are involved grees of difficulty. The available mathematical it could not be used at all.7 insights into the Hamiltonian Cycle provide us neither with a polynomial algorithm nor with a He solved the Königsberg bridges problem not proof that such an algorithm is impossible. The by listing all possible trails but by using mathe- Hamiltonian Cycle is intractable, and nobody matical insight. He noticed that in a circuit, you knows why. must leave each vertex via an edge different from the edge that has taken you there. In other Coloring words, the vertex’s degrees (that is, the number Imagine we wish to arrange talks in a congress of edges adjacent to it) must be even. This is ob- in such a way that no participant will be forced viously a necessary condition, but Euler proved to miss a talk he or she wants to hear. Assuming a that it is also sufficient: good supply of lecture rooms that lets us hold as

36 COMPUTING IN SCIENCE & ENGINEERING

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. many parallel talks as we like, can we finish the F (x , x ) = (x ∨ x ) ∧ x ∧ x (5) program within k time slots? We can formulate 2 1 2 1 2 2 1 this problem in terms of graphs: Let G be a graph is not. whose vertices are the talks and in which two We can write every Boolean formula in con- talks are adjacent (joined by an edge) if and only junctive normal form (CNF)—that is, as a set of if there is a participant wishing to attend both. clauses Ck combined exclusively with the AND Your task is to assign one of the k time slots to operator each vertex in such a way that adjacent vertices F = C ∧ C ∧ ... ∧ C , (6) have different slots. The common formulation of 1 2 m this problem uses colors instead of time slots: where the literals in each clause are combined exclusively with the OR operator. The examples k-Coloring: Given a graph G = (V, E), is there a F1 and F2 are both written in CNF. Each clause coloring of the vertices of G using at most k dif- can be considered a constraint on the variables, ferent colors such that no two adjacent vertices and satisfying a formula means satisfying a set of have the same color? (possibly conflicting) constraints simultaneously. Hence, consider the following as a prototype of When k = 2, this problem is tractable—construct- a constraint satisfaction problem:8 ing a polynomial algorithm is an easy exercise. For k = 3, however, things change considerably: 3-Col- Satisfiability (SAT): Given disjunctive clauses oring is intractable. Note that for larger k, the C1, C2, ..., Cm of literals, where a literal is a vari- problem gets easier again: a planar graph is always able or negated variable from the set {x1, x2, ..., colorable with four colors. This is the famous 4- xn}, is there an assignment of variables that satis- Color Theorem. 3-Coloring remains intractable fies all clauses simultaneously? even when restricted to planar graphs. Fixing the number of literals in each clause leads to Satisfiability I close this section with a decision problem k-SAT: Given disjunctive clauses C1, C2, ..., Cm that is not from graph theory but from Boolean of k literals each, where a literal is a variable or logic. A Boolean variable x can take on the value negated variable from the set {x1, x2, ..., xn}, is 0 (false) or 1 (true). Boolean variables can be there an assignment of variables that satisfies all combined in clauses using the Boolean operators clauses simultaneously?

• NOT ⋅ (negation): the clause x is true Polynomial algorithms are known for 1-SAT ( x = 1) if and only if x is false (x = 0). and 2-SAT.9 No polynomial algorithm is known • AND ∧ (conjunction): the clause x1 ∧ x2 is for general SAT and k-SAT if k > 2. true (x1 ∧ x2 = 1) if and only if both variables are true: x1 = 1 and x2 = 1. Complexity classes • OR ∨ (disjunction): the clause x1 ∨ x2 is true (x1 ∨ x2 = 1) if and only if at least one of the Now we have seen enough examples to intro- variables is true: x1 = 1 or x2 = 1. duce two important complexity classes for decision problems and to discuss how these classes A variable x or its negation x is called a literal. relate to other kinds of problems. Different clauses can combine to yield complex Boolean formulas such as Tractable problems Defining the class of tractable decision problems F1(x1, x2, x3) = (x1 ∨ x2 ∨ x3) ∧ is easy: it consists of those problems for which a polynomial algorithm is known. The corre- (x2 ∨ x3) ∧ (x1 ∨ x2) ∧ (x1 ∨ x3) . (4) sponding class is named for “polynomial”: P A Boolean formula evaluates to either 1 or 0, de- pending on the assignment of the Boolean vari- Definition: A decision problem P is element of ables. For example, F = 1 for x = 1, x = 1, x = the class if and only if a polynomial time algo- 1 1 2 3 P 0, and F1 = 0 for x1 = x2 = x3 = 1. A formula F is rithm can solve it. called satisfiable if there is at least one assignment of the variables such that the formula is true. F1 Eulerian Circuit, 2-Coloring, MST(D), and so is satisfiable, forth are in . P

MAY/JUNE 2002 37

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. returns “yes.” The answer is “no” if none of the branches reports “yes.” We say that a nondeterministic algorithm solves a decision problem in polynomial time if the number of steps used by the first of the branches to report “yes” is bounded by a polynomial in the problem’s size. Elapsed time We require polynomial solubility only for a decision problem’s “yes” instances. This asymmetry between “yes” and “no” reflects the asymmetry between the “there is” and “for all” quantifiers in no decision problems. A graph G is a “yes” instance of the Hamiltonian Cycle if there is at least one no no no Hamiltonian cycle in G. For a “no” instance, all yes cycles in G must be non-Hamiltonian. no no Conventional (deterministic) algorithms are no no no no special cases of nondeterministic algorithms no yes (those nondeterministic algorithms that do not use the goto both instruction). It follows imme- Figure 5. Example of a nondeterministic algorithm’s execution history. diately that ⊆ . P NP All decision problems discussed thus far have been members of . Here’s a nondeterminis- NP Nondeterministic algorithms tic polynomial algorithm for SAT: The definition of the second complexity class involves the concept of a nondeterministic algo- Satisfiability (F) rithm, which is like an ordinary algorithm, ex- Input: Boolean formula F(x1, ..., xn) cept that it might use one additional, very pow- Output: ‘yes’ if F is satisfiable, ‘no’ otherwise erful instruction:10 begin for i = 1 to n goto both label 1, label 2 goto both label 1, label 2 label 1: xi = true; continue This instruction splits the computation into two label 2: xi = false; continue parallel processes, one continuing from each of end the instructions, indicated by “label 1” and “label if F(x1, ..., xn) = true then return ‘yes’ 2.” By encountering more such instructions, the else return ‘no’ computation branches like a tree into several end parallel computations that potentially can grow as an exponential function of the time elapsed The for loop branches at each iteration: in one (see Figure 5). A nondeterministic algorithm can branch, xi = true; in the other branch, xi = false perform an exponential number of computations (the continue instruction starts the loop’s next in polynomial time. iteration). After executing the for loop, we have In the world of conventional computers, non- 2n branches of computation—one for each of the deterministic algorithms are a theoretical con- possible assignments of n Boolean variables. cept only, but this could change in quantum The power of nondeterministic algorithms is computing. We need the concept of nondeter- that they allow the exhaustive enumeration of an minism to define the class of “nondeter- exponentially large number of candidate solu- NP ministic polynomial” problems: tions in polynomial time. If the evaluation of each candidate solution (calculating F(x1, ..., xn) Definition: A decision problem P is in the class in the SAT example) in turn can be done in poly- if and only if a nondeterministic algorithm nomial time, the total nondeterministic algo- NP can solve it in polynomial time. rithm is polynomial. For a problem from the class , the sole source of intractability is the NP Solubility by a nondeterministic algorithm means exponential size of the search space. this: All branches of the computation will stop, returning either “yes” or “no.” We say that the over- Succinct certificates all algorithm returns “yes” if any of its branches There is a second, equivalent definition of

38 COMPUTING IN SCIENCE & ENGINEERING

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. Hamiltonian Cycle TSP(D) Hamiltonian Cycle TSP(D) SAT Primality SAT NP complete NP 3-Coloring 3-Coloring 3-SAT 3-SAT

Compositeness Graph-Isomorphism Primality Graph-Isomorphism NP Compositeness

Eulerian Circuit Eulerian Circuit 2-Coloring 2-Coloring 2-SAT P 2-SAT P Assignment(D) Assignment(D) MST(D) MST(D)

(a) (b)

Figure 6. (a) A ﬁrst map of complexity; (b) the map of complexity revisited. All problems indicated are deﬁned within the text. Problems with a (D) are decision variants of optimization problems.

based on the notion of a succinct certificate. be verified in quadratic time by multiplication. NP A certificate is a proof. If you claim that a graph So, Compositeness ∈ . NP G has a Hamiltonian Cycle, you can proof your Most decision problems ask for the existence of claim by providing a Hamiltonian Cycle. Cer- an object with a given property, such as a cycle, tificates for the Eulerian Circuit and k-Coloring which is Hamiltonian or a factorization with inte- are an Eulerian Circuit and a valid coloring, re- ger factors. In these cases, the desired object might spectively. A certificate is succinct if its size is serve as a succinct certificate. For some problems, bounded by a polynomial in the problem size. this does not work, however. For example, The second definition then reads Primality: Given a positive integer N, is N prime? Definition: A decision problem P is element of the class if and only if for every “yes” in- Primality is the negation or complement of NP stance of P there exists a succinct certificate that Compositeness: the “yes” instances of the for- can be verified in polynomial time. mer are the “no” instances of the latter and vice versa. A succinct certificate for Primality is by The equivalence between both definitions can no means obvious. In fact, for many decision be proven easily.10 The idea is that a succinct problems in , no succinct certificate is NP certificate can deterministically select the branch known for the complement—that is, whether the in a nondeterministic algorithm that leads to a complement is also in is not known. For NP “yes” output. Primality, however, there is a succinct certificate The definition based on nondeterministic al- based on Fermat’s Theorem.11 Hence, Primal- gorithms reveals the key feature of the class ity ∈ . NP NP more clearly, but the second definition is more useful for proving that a decision problem is in A first map of complexity . For example, consider Figure 6a summarizes what we have achieved NP so far. The class consists of all decision NP Compositeness: Given a positive integer N, are problems whose sole source of difficulty is the there integer numbers p > 1 and q > 1 such that N size of the search space, which grows exponen- = pq? tially with the problem size. These problems are intractable unless some mathematical insight A certificate of a “yes” instance N of Compos- provides us with a polynomial shortcut to avoid iteness is a factorization N = pq. It is succinct be- an exhaustive search. Such an insight promotes a cause the number of bits in p and q is less than problem into the class of polynomially solu- P or equal to the number of bits in N, and it can ble problems.

MAY/JUNE 2002 39

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. The class not only contains many prob- Hamiltonian Cycle in G is a valid tour in the NP lems with important applications but also repre- corresponding TSP with all intercity distances sents a real challenge: all problems in still having length 1—that is, with total length n. NP have a chance to be in . A proof of nonexis- Conversely, suppose that the TSP has a tour of P tence of a polynomial algorithm for a single length n. Because the intercity distances are ei- problem from would establish that ≠ ther 1 or 2, and a tour sums up n such distances, NP P . As long as such a proof is missing, a total length n implies that each pair of succes- NP sively visited cities must have distance 1—that ? is, the tour follows existing edges in G and cor- P = NP (7) responds to a Hamiltonian Cycle. So, the call to represents the most famous open conjecture in a subroutine that solves TSP(D) (line 7) yields a theoretical computer science. solution to the Hamiltonian Cycle. How does this algorithm relate the computa- completeness tional complexity of the Hamiltonian Cycle to NP So far, all the intractable problems seem to be that of the TSP(D)? This is a polynomial algo- isolated islands in the map of complexity. In fact, rithm if the call to the TSP(D) solver is consid- they are tightly connected by a device called poly- ered an elementary operation. If someone comes nomal reduction, which lets us map the computa- up with a polynomial algorithm for TSP(D), we tional complexity of one problem to the com- will instantly have a polynomial algorithm for plexity of another. This finally leads to the the Hamiltonian Cycle. We say that the Hamil- surprising result that there are some intractable tonian Cycle is polynomially reducible to TSP(D) problems that are equivalent to all other prob- and write lems. Each of these so called -complete NP Hamiltonian Cycle ≤ TSP(D). (8) problems embodies the secret of intractability, since a polynomial time algorithm for one of In many books, polynomial reducibility is de- them would immediately imply the tractability noted by ∝ instead of ≤. We use ≤ because this of all problems in . notation stresses an important consequence of NP polynomial reducibility:12 the existence of a Polynomial reductions. The computational com- polynomial reduction from P1 to P2 excludes the plexities of two problems, P1 and P2, can be re- possibility that we can solve P2 in polynomial lated to each other by constructing an algorithm time but not P1. Hence, we can read P1 ≤ P2 as for P1 that uses an algorithm for P2 as a “sub- P1 cannot be harder than P2. Here is the (informal) routine.” Consider the following algorithm that deﬁnition: relates Hamiltonian Cycle to TSP(D): Deﬁnition: We say a problem P1 is polynomi- Hamiltonian Cycle (G) ally reducible to a problem P2 and write P1 ≤ P2 Input: Graph G = (V, E) if a polynomial algorithm exists for P1 provided Output: ‘yes’ if G contains a Hamiltonian there is a polynomial algorithm for P2. cycle, ‘no’ otherwise (1) begin NP-complete problems. Here are some other poly- (2) n : = |V| nomial reductions that we can verify, similar to (3) for i = 1 to n Equation 8 (the corresponding reduction algo- (4) for j = 1 to n rithms appear elsewhere13): (5) if (v , v ) ∈ E then d := 1 i j ij SAT ≤ 3-SAT (6) else d := 2 ij 3-SAT ≤ 3-Coloring (7) if TSP-decision (d, B := n) = ‘yes’ 3-Coloring ≤ Hamiltonian Cycle then return ‘yes’ (9) (8) else return ‘no’ (9) end Polynomial reducibility is transitive: P1 ≤ P2 and P2 ≤ P3 imply P1 ≤ P3. From transitivity and Equa- This algorithm solves the Hamiltonian Cycle tions 8 and 9, it follows that each SAT, 3-SAT, 3- by solving an appropriate instance of the Coloring, and Hamiltonian Cycle reduces to TSP(D). In the for loops (lines 3 through 5), a TSP(D)—that is, a polynomial algorithm for distance matrix d is set up with entries dij = 1 if TSP(D) implies a polynomial algorithm for all there is an edge (vi, vj) in G—otherwise, dij = 2. A these problems. This is amazing, but it’s only the

40 COMPUTING IN SCIENCE & ENGINEERING

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. beginning. Stephen Cook14 re- vealed polynomial reducibility’s true scope in 1971 when he NP complete NP complete proved that NP Theorem: All problems in P = NP are polynomially reducible NP to SAT, P P ∀ P ∈ : P ≤ SAT. (10) NP This theorem means that (a) (b) (c)

• No problem in is harder Figure 7. Three tentative maps of . We can rule out map B; map A is likely (but not NP NP than SAT, or SAT is among surely) the correct map. The discovery of a polynomial algorithm for any of the - NP the hardest problems in . complete problems would turn C into the correct map. NP • A polynomial algorithm for SAT would imply a polynomial algorithm for every problem in — and G´ isomorphic—that is, is there a permuta- NP that is, it would imply = . tion π of V such that G´ = π (G), where π(G) de- P NP notes the graph (V, {[π(u), π(v)] : [u, v] ∈ E})? It seems as if SAT is very special, but according to transitivity and Equations 8 and 9, 3-SAT, 3- There are more problems in that resist NP Coloring, the Hamiltonian Cycle, or the classification in or , but none of these P NP TSP(D) can replace it. These problems form a problems has been proven not to belong to or P new complexity class: . What has been proven is NP Definition: A problem P is called complete Theorem: If ≠ , then problems exist NP P NP NP if P ∈ and Q ≤ P for all Q ∈ . that are neither in nor complete. NP NP P NP The class of -complete problems collects the This theorem15 rules out one of three tentative NP hardest problems in . If any of them has an maps of (see Figure 7). NP NP efficient algorithm, then every problem in NP can be solved efficiently—thus, = . This is Beyond P NP NP extremely unlikely, however, considered the fu- The notions and complete strictly NP NP tile efforts of many brilliant people to find poly- apply only to decision problems (“Is there a so- nomial algorithms for problems such as the lution?”). The ideas of this approach can be gen- Hamiltonian Cycle or TSP(D). eralized to optimization problems (“What is the best solution?”) and counting problems (“How The map of . Since Cook developed his the- many solutions are there?”). NP orem, many problems have been shown to be complete (see www.nada.kth.se/~viggo/ Optimization problems. How does the classifica- NP wwwcompendium for a comprehensive, up-to- tion of decision problems relate to optimization date list of hundreds of -complete prob- problems? The general instance of an optimiza- NP lems). Thus, the map of presented in Fig- tion problem is a pair (F, c), where F is the set of NP ure 6a needs some redesign (see Figure 6b). It feasible solutions and c is a cost function c: F → turns out that all the intractable problems we ޒ. Here I consider only combinatorial optimiza- have discussed so far are complete—except tion where the set F is countable. A combinator- NP Compositeness and Primality. For both prob- ial optimization problem P comes in three dif- lems, neither a polynomial algorithm is known ferent flavors: nor a polynomial reduction that classifies them as complete. Another problem that re- 1.The optimization problem P(O): Find the NP NP sists classification in either or is feasible solution f *∈ F that minimizes the cost P NP function. Graph Isomorphism: Given two graphs G = (V, 2.The evaluation problem P(E): Find the cost E) and G´(V, E´) on the same set of nodes, are G c* = c(f *) of the minimum solution.

MAY/JUNE 2002 41

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. 3.The decision problem P(D): Given a bound ber of different Hamiltonian Cycles in a given B ∈ ޒ, is there a feasible solution f ∈ F such graph, #TSP for the number of different tours that c( f ) ≤ B? with length ≤ B, and so on.

Under the assumption that we can evaluate the Definition: A counting problem #P is a pair (F, cost function c in polynomial time, it is straight- d), where F is the set of all feasible solutions, and forward to write down polynomial reductions d is a decision function d: F → {0,1}. The output that establish of #P is the number of f ∈ F with d(f) = 1. The class # (pronounced “number P”) consists of P(D) ≤ P(E) ≤ P(O). (11) P all counting problems associated with a decision If an optimization problem’s decision variant is function d that can be evaluated in polynomial complete, there is no efficient algorithm for time. NP the optimization problem at all—unless = P . An optimization problem such as the TSP, Like the class , # collects all problems NP NP P whose decision variant is complete, is de- whose sole source of intractability is the number NP noted hard. of feasible solutions. A polynomial algorithm for NP Does a polynomial algorithm for a decision a counting problem #P implies a polynomial al- problem imply a polynomial algorithm for the gorithm for the associated decision problem P: optimization or evaluation variant? For that, we P ≤ #P. Hence, it is unlikely that #SAT can be need to prove the reversal of Equation 11: solved efficiently. In fact, we can define polynomial reducibility for counting problems and P(O) ≤ P(E) ≤ P(D). (12) prove that all problems in # reduce polynomi- P P(E) ≤ P(D) can be shown to hold if the opti- ally to #SAT:6 mum solution’s cost is an integer with logarithm bounded by a polynomial in the input’s size. Theorem: #SAT is # complete. P The corresponding polynomial reduction evaluates the optimal cost c* by asking, “Is c* ≤ B?” As you might have guessed, #Hamiltonian Cy- for a sequence of values B that approaches c* cle and #TSP are also # complete. Despite the P (similar to the bisection method for finding a similarity between and # , counting prob- NP P function’s zeroes). lems are inherently harder than decision prob- There is no general method to prove P(O) ≤ lems. This is documented by those # -complete P P(E), but a strategy that often works for the TSP is problems for which the corresponding decision this: Let c* be the known solution of TSP(E). Re- problem can be solved in polynomial time—the place an arbitrary entry dij of the distance matrix classical example being the problem of calculat- with a value c > c* and solve TSP(E) with this mod- ing a matrix’s permanent.16 ified distance matrix. If this modification doesn’t affect the tour’s optimum length, the link ij does not belong to the optimal tour. Repeating this pro- Computational complexity and physics cedure for different links, we can reconstruct the The relationship between computation com- optimum tour with a polynomial number of calls plexity and physics should offer new insights, to a TSP(E) solver; hence, TSP(O) ≤ TSP(E). some of which are discussed here. For example, some knowledge of computational complexity Counting problems. So far, we have studied two helps us understand the promises and limitations related styles of problems: Decision problems of quantum computers. In addition, we can ask whether a desired solution exists, and opti- seemingly transfer the notion of tractable and mization problems require that a solution be intractable problems to the problem of analytical produced. A third important and fundamentally solubility of models from statistical physics, ex- different kind of problem asks how many solu- plaining to some extent why, for example, the tions exist. The counting variant of SAT reads Ising model is exactly soluble in 2D but not 3D. Another interesting link between computa- #SAT: Given a Boolean expression, compute tion complexity and physics is that statistical the number of different truth assignments that mechanics offer means for the general proba- satisfy it. bilistic analysis of computational problems. In statistical mechanics, we typically formulate an Similarly, #Hamiltonian Cycle asks for the num- optimization problem as a spin glass and ana-

42 COMPUTING IN SCIENCE & ENGINEERING

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. lyze the latter’s low temperature properties. To gain the advantage of exponential paral- This “physical” approach often yields results lelism, we must combine it with another quantum that go beyond the results obtained by tradi- feature known as interference. The goal is to tional methods. Another surprising observation arrange the computation such that constructive is the fact that random instances of intractable interference amplifies desired result and destruc- problems can be solved in polynomial time. To tive interference cancels the rest. Because of the observe the exponential running time, the pa- importance of interference phenomena, it is not rameters of the random ensemble must be ad- surprising that calculating the Fourier transform justed carefully. This scenario corresponds to a was the first problem that underwent an exponen- phase transition in physical systems and is there- tial speedup: from (n log n) on a classical to O fore best studied within the framework of sta- (log2n) on a quantum computer. This speedup O tistical mechanics. was the seed for the most important quantum algorithm known today: Shor’s algorithm to factor Quantum parallelism an integer in polynomial time.18 There is some theoretical evidence that com- Although Shor’s algorithm has some conse- puters using quantum systems as computational quences for public key cryptography, it does not devices are more powerful than computers based shatter the world of : remember that Com- NP on classical devices. The hope is that problems positeness is in , but it is not complete. NP NP that are intractable on a classical computer be- Hence, the holy grail of quantum computing has come tractable when put on a quantum com- yet to be discovered—a polynomial time quan- puter. Results such as Peter Shor’s celebrated tum algorithm for an -complete problem. NP quantum algorithm for factorization nurture this hope, but a real breakthrough is still missing. Analytical solubility of Ising models In a seminal paper, Richard Feynman pointed Some problems in statistical physics have been out that a system of n quantum particles is expo- exactly solved, but the majority of problems have nentially hard to simulate on a classical com- withstood the efforts of generations of mathe- puter.17 The idea of quantum computing is to re- maticians and physicists. Why are some prob- verse this situation and simulate a classically hard lems analytically solvable, whereas other, often (for example, exponential) problem in polynomial similar, problems are not? Relating this question time on a computer made of quantum devices. to the algorithmic complexity of evaluating the A quantum computer processes qubits—quan- partition function gives us no final answer but tum two-state systems |0〉, |1〉. A quantum com- helps clarify the borderline that separates ana- puter’s key feature is that its registers can hold and lytically tractable from intractable problems. process linear superpositions of all 2n product For example, consider the Ising spin glass on a 19 states of n qubits, such as general graph G. Let σ = (σ1, ..., σN) be an as- 1 signment of Ising spins σi = ±1 (up or down). 1 The energy of a configuration σ is |i1i2...in . (13) n ∑ 2 i1,i2,...,in =0 E(σ) = − Ji, jσ iσ j − H σ i , (14) Using this feature, constructing a quantum com- ∑ ∑ i, j i puter capable of computing any function f(x1, ..., xn) of n Boolean variables simultaneously for all where H is the external magnetic field, Ji,j are the 2n possible input values is not difficult—in the- coupling constants, and the first summation is ory, at least. This quantum parallelism resembles over all edges in G. The fundamental problem a nondeterministic algorithm with its goto both in statistical mechanics is to determine the par- instruction and its exponentially branching exe- tition function cution tree. Is quantum parallelism the key to −βE (σ ) exponential computing power? The problem is ZN (G) = ∑ e . (15) how to extract the exponential information out σ of a quantum computer. When we defined non- Evaluating the sum directly requires (2N) op- O deterministic solubility, we didn’t care about how erations. The notion of analytic solution has no to spot the single “yes” among the 2n “no” an- precise definition, but as a minimum require- swers. This works fine for a theoretical concept, ment, we want to reduce this number from being but for a practical computer, reading the output exponential to being polynomial in N. really matters. Consider the well-known transfer matrix so-

MAY/JUNE 2002 43

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. structing Gˆ from G is easy—just adjoin an additional vertex (spin) σ0 to G and let the additional edges σ0σi have constant interaction energy J = H. The partition function of the J 0,i J J 2 N 1 augmented system reads

σ ˆ −β [− ∑ Ji, jσ iσ j ] −βH ∑σ i βH ∑σ i 3 Z(G) = e e + e , σ σ σ2 ∑ ( ) N 1 σ Figure 8. One-dimensional Ising spin glass with periodic boundary (19) conditions. We can calculate this system’s partition sum in where the additional factor comes from the new polynomial time. spin σ0 = ±1. From this expression, it is easy to see that Z( Gˆ ) equals two times the partition function Z(G) in field H. lution of the 1D Ising glass with periodic bound- But where are the soluble Ising models? It has ary conditions and coupling Jk between spins σk been proven that we can evaluate the partition and σk+1 (see Figure 8): function of Ising systems on planar crystal lattices in polynomial time.22 This includes the cele- N eβ ( Jk +H ) e−βJk  brated Onsager solution of the square ferro- ZN (ring) = Tr  −βJ β ( J −H )  , (16) 23 ∏ e k e k  magnet as well as the 1D example just pre- k=1 sented. It turns out that we can calculate the which can be evaluated in (N) elementary op- Ising model’s partition sum in polynomial time O erations. Because any exact solution must in- for all graphs of fixed genus g.24,25 A graph has clude all numbers Jk, this is the best we can ex- genus g if it can be drawn on an orientable sur- pect. In the homogeneous case Jk ≡ J, where we face of genus g (a sphere with g “handles” at- can calculate the product of transfer matrices, tached to it) without crossing the edges. Planar graphs have genus 0, toroidal graphs have genus N N ZN = λ+ + λ− with λ± = 1, and so on. For the crystal lattices in d > 2, the eβJ cosh(βH) ± cosh2(βH) − 2e−2βH sinh(2βH) , genus increases monotonically with the number   of spins—that is, it is not fixed.24 (17) The mechanism for proving tractability or in- the evaluation complexity drops to (log N) us- tractability is the same in statistical mechanics as O ing fast exponentiation. it is in computational complexity: polynomial re- Writing the partition sum as duction. Francisco Barahona, for example, ap- plies a reduction of the -hard problem Max −βEk NP Z(G) = ∑ n( Ek )e , (18) Cut to the Ising spin glass in 3d to proof the lat- 22 Ek ter’s hardness. A reduction of the planar NP where the sum is over all possible energy val- Ising model to Minimum Weight Matching, on ues, it becomes obvious that calculating Z is the other hand, proofs the tractability of the for- closely related to the # problem of determin- mer because we can solve Minimum Weight P ing n(Ek). For general finite graphs G, this Matching in polynomial time. problem has proven to be # complete,20 so In our classification of spin systems, the na- P there is no hope of finding an analytical solu- ture of the couplings is not significant. A frus- tion (even in the weak sense above). This situ- trated, glassy system with random couplings Ji,j ation hardly improves if we restrict the graphs of both signs is in the same class as the homo- to the more “physical” crystal lattices: com- geneous ferromagnet with Ji,j ≡ J > 0 as long as puting the partition function for a finite sub- the underlying graph G is the same. In the 1D lattice of a nonplanar crystal lattice is # com- example, we did not discriminate these cases: P plete.21 This includes every crystal lattice in d > they are both polynomial. This situation 2, the d = 2 model with next-nearest neighbor changes, of course, if we consider the ground interactions, two coupled d = 2 models, and so states rather than the complete partition func- forth. It also includes all models with d ≥ 2 and tion. Here, the nature of the couplings matters a external field H, because these can be trans- lot: finding the ground states in pure ferromag- formed into zero-field models on an aug- netic systems is trivial on all lattices, whereas it mented graph Gˆ (which is nonplanar unless is hard for glassy systems with positive and NP the underlying lattice graph G is 1D). Con- negative couplings on nonplanar lattices.22

44 COMPUTING IN SCIENCE & ENGINEERING

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. We can classify many other problems arising Some years after the replica solution, Parisi in statistical physics according to the computa- recognized that for exponentially distributed tional complexity to evaluate their partition matrix elements (ρ(a) = e–a), the average optima function.26 We can also evaluate all the prob- for N = 1 and N = 2 are lems known to have an exact solution27 in poly- 1 nomial time. Problems that are # complete, E * = 1 E * = 1 + . (22) P 1 2 2 however, are unlikely to be exactly solvable. 2 Anyone looking for an exact solution of such a From this and the fact that we can write the problem should keep in mind that he or she is replica result for N→∞ as simultaneously attacking TSP, Hamiltonian Cy- 2 ∞ cle, SAT, and all the other -hard problems. π 1 NP = (23) In statistical mechanics, the focus is on results 6 ∑ 2 k=1 k for the thermodynamic limit N → ∞ rather than for finite systems, however. It is not clear how he conjectured35 that the average optimum for much of the “hardness” survives in this limit. ﬁnite systems is

N Probabilistic analysis of combinatorial problems 1 E * = . (24) We can formally consider problems from N ∑ 2 k=1 k combinatorial optimization as models in statistical mechanics. The cost function is renamed Parisi’s conjecture is supported by numerical Hamiltonian, random instances are samples of simulations, but no formal proof has been found quenched disorder, and the formal model’s despite some efforts.36 ground states correspond to the solutions of the Equations 22 and 24 only hold for ρ(a) = e–a, optimization problems. In this way, methods de- whereas Equation 21 is valid for all densities veloped in the framework of statistical mechan- with ρ(a → 0) = 1. For the uniform density on ics of disordered systems become powerful tools [0,1], the first terms are for the probabilistic analysis of combinatorial 28 * 1 * 23 problems. E1 = E2 = . (25) Researchers have applied statistical mechan- 2 30 ics methods, for example, to the TSP,29,30 Graph (If you can you guess the expression for general, Partitioning,31 and k-SAT.32 A particularly nice finite N in this case, please send me an email.) example of this approach is the analysis of As- Sometimes a statistical mechanics analysis not signment (also called Bipartite Matching): Given only yields exact analytical results but also re- an N × N matrix with nonnegative entries ai,j ≥ veals features that are important to design and 0, find understand algorithms. A recent example is the N analysis of the Davis-Putnam algorithm for * 37 EN = min ai,σ (i) , (20) SAT. Another example is given by the number σ ∑ i=1 partitioning problem (NPP), an -hard opti- NP where the minimum is taken over all permuta- mization problem.38 “Physical” reasoning has led tions σ of (1, ..., N). to the conjecture that for this problem, no A probabilistic analysis aims at calculating av- heuristic algorithm can outperform simple ran- erage properties for an ensemble of random in- dom search.39 stances, the canonical ensemble being random numbers ai,j drawn independently from a com- Phase transitions in computational complexity mon probability density ρ(a). Using the replica The theory of computational complexity is method from statistical physics, Marc Mézard based entirely on worst-case analysis. An algo- and Giorgio Parisi33 found (among other things) rithm could require exponential time on patho- 2 logical instances only. A famous example is the * π lim EN = , (21) simplex method for linear programming. De- N → ∞ 6 spite its exponential worst-case complexity, it is where 〈•〉 denotes averaging over the ai,j. David used in many applications to solve really large Aldous recently presented34 a rigorous proof of problems. Apparently the instances that trigger Equation 21, which represents one of the rare exponential running time do not appear under cases where rigorous methods have confirmed a practical conditions. replica result. Linear programming is in thanks to the El- P

MAY/JUNE 2002 45

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. lipsoid algorithm,5 but similar scenarios are ob- 3. B. Bollobás, Modern Graph Theory, Graduate Texts in Mathe- matics, vol. 184, Springer-Verlag, Berlin, 1998. served for -hard problems. Take 3-SAT, for NP 4. G. Reinelt, “The Traveling Salesman,” Computational Solutions example. When generating random instances for TSP Applications, Lecture Notes in Computer Science, vol. 840, with N variables and M clauses and feeding them Springer-Verlag, Berlin, 1994. to a smart but exhaustive algorithm, we observe 5. C.H. Papadimitriou and K. Steiglitz, Combinatorial Optimization, polynomial running time unless the ratio M/N Prentice-Hall, Englewood Cliffs, N.J., 1982. is carefully adjusted. If M/N is too low, the prob- 6. L. Euler, “Solutio Problematis Ad Geometrian Situs Pertinentis,” Commetarii Academiae Scientiarum Imperialis Petropolitanae, vol. lem is underconstrained—that is, it has many sat- 8, 1736, pp. 128–140. isfying solutions, and a clever algorithm will find 7. H.R. Lewis and C.H. Papadimitriou, “The Efficiency of Algo- one of these quickly. If M/N is too large, the rithms,” Scientific American, vol. 109, no. 1, Jan. 1978, pp. problem is overconstrained—that is, it has many 96–109. contradictory constraints, which, again, a clever 8. V. Kumar, “Algorithms for Constraint Satisfaction Problems: A algorithm will discover quickly.40 The real hard Survey,” AI Magazine, vol. 13, no. 1, 1992, pp. 32–44; ftp:// ftp.cs.umn.edu/dept/users/kumar/csp-aimagazine.ps. instances are generated for ratios α = M/N close 41 9. B. Aspvall, M.F. Plass, and R.E. Tarjan, “A Linear-Time Algorithm to a critical value αc. for Testing the Truth of Certain Quantified Boolean Formulas,” The transition from underconstrained to over- Information Processing Letters, vol. 8, no. 3, 1979, pp. 121–123. constrained formulas in 3-SAT bears the hall- 10. D.S. Johnson and C.H. Papadimitriou, “Computational Com- marks of a phase transition in physical systems. plexity,” The Traveling Salesman Problem, Lawler et al., eds., 1985, pp. 37–85. The control parameter is α; the order parame- 11. V.R. Pratt, “Every Prime has a Succinct Certificate,” SIAM J. Com- ter is the probability of the formula being satis- puting, vol. 4, no. 3, 1975, pp. 214–220. fiable. Similar phase transitions do occur in var- 12. G. Ausiello et al., Complexity and Approximation, Springer-Verlag, ious other decision or optimization problems, Berlin, 1999. and mathematical methods from statistical me- 13. R.M. Karp, “Complexity of Computer Computations,” Reducibil- chanics have successfully been used for their ity Among Combinatorial Problems, R.E. Miller and J.W. Thatcher, eds., Plenum Press, New York, 1972, pp. 85–103. analysis. 14. S. Cook, “The Complexity of Theorem Proving Procedures,” Proc. 3rd Ann. ACM Symp. Theory of Computing, ACM Press, New York, 1971, pp. 151–158. eading up to this point hopefully has 15. R.E. Ladner, “On the Structure of Polynomial Time Reducibility,” convinced you that there are some in- J. ACM, vol. 22, no. 1, 1975, pp. 155–171. teresting links between physics and the 16. L.G. Valiant, “The Complexity of Computing the Permanent,” theory of computational complexity. Theoretical Computer Science, vol. 8, 1979, pp. 189–201. RIn fact, mathematicians, computer scientists, and 17. R. Feynman, “Simulating Physics with Computers,” Int’l J. Theo- retical Physics, vol. 21, nos. 6 and 7, 1982, pp. 467–488. physicists have just started interdisciplinary work 18. P.W. Shor, “Polynomial-Time Algorithms for Prime Factorization in this field. Tools and notions from statistical and Discrete Logarithms on a Quantum Computer,” SIAM J. mechanics might shed more light on the typical Computing, vol. 26, no. 5, 1997, pp. 1484–1509. case complexity of problems and might help to 19. B. Hayes, “Computing Science: The World in a Spin,” American improve heuristic algorithms. Quantum com- Scientist, vol. 88, no. 5, Sept./Oct. 2000, pp. 2384–388; www. amsci.org/amsci/issues/comsci00/compsci2000-09.html. puting may even turn intractable problems into 20. F. Jaeger, D.L. Vertigan, and D.J.A. Welsh, “On the Computa- tractable ones some day. A polynomial time al- tional Complexity of the Jones and Tutte Polynomials,” Mathe- gorithm for an -complete problem would be matical Proc. Cambridge Philosophical Soc., vol. 108, 1990, pp. NP a real breakthrough—a theoretical breakthrough 35–53. first of all, but in the far future, quantum com- 21. S. Istrail, “Statistical Mechanics, Three-Dimensionality and NP- Completeness,” Proc. 31st ACM Ann. Symp. Theory of Computing puters might be available as hardware. If this re- (STOC 2000), ACM Press, New York, 2000, pp. 87–96. ally happens, computational complexity and 22. F. Barahona, “On the Computational Complexity of Ising Spin physics will no longer be considered separate Glass Models,” J. Physics A: Mathematical and General, vol. 15, fields. vol. 10, 1982, pp. 3241–3253. 23. L. Onsager, “Crystal Statistics I: A Two-Dimensional Model with an Order-Disorder Transition,” Physical Rev., vol. 65, 1944, pp. 117–149. 24. T. Regge and R. Zecchina, “Combinatorial and Topological Ap- References proach to the 3D Ising Model,” J. Phys. A, vol. 33, 2000, pp. 1. D. Coppersmith and S. Winograd, “Matrix Multiplication via 741–761. Arithmetic Progressions,” J. Symbolic Computation, vol. 9, no. 3, 25. A. Galluccio, Martin Loebl, and Jan Vondrák, “New Algorithm for 1990, pp. 251–280. the Ising Problem: Partition Function for Finite Lattice Graphs,” 2. T.H. Cormen, C.E. Leiserson, and R.L. Rivest, Introduction to Al- Physical Rev. Letters, vol. 84, no. 26, 2000, pp. 5924–5927. gorithms: The MIT Electrical Engineering and Computer Science Se- 26. D.J.A. Welsh, “The Computational Complexity of Some Classi- ries, MIT Press, Cambridge, Mass., 1990. cal Problems from Statistical Physics,” Disorder in Physical Sys-

46 COMPUTING IN SCIENCE & ENGINEERING

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply. tems, G.R. Grimmett and D.J.A. Welsh, eds., Clarendon Press, 2001, pp. 505–531. Oxford, 1990, pp. 307–321. 38. B. Hayes, “Computing Science: The Easiest Hard Problem,”Amer- 27. R.J. Baxter, Exactly Solved Models in Statistical Mechanics, Acade- ican Scientist, vol. 90, no. 2, Mar./Apr. 2002, pp. 113–117; mic Press, San Diego, 1982. www.amsci.org/amsci/issues/comsci02/compsci2002-03.html. 28. M. Mézard, G. Parisi, and M.A. Virasoro, Spin Glass Theory and 39. S. Mertens, “Random Costs in Combinatorial Optimization,” Beyond, World Scientific, Singapore, 1987. Physical Rev. Letters, vol. 84, no. 7, Feb. 2000, pp. 1347–1350. 29. M. Mézard and G. Parisi, “Mean-Field Equations for the Match- 40. B. Hayes, “Computing Science: Can’t Get No Satisfaction,” Amer- ing and the Traveling Salesman Problems,” Europhys. Letters, vol. ican Scientist, vol. 85, no. 2, Mar./Apr. 1997, pp. 108–112; 2, no. 12, Dec. 1986, pp. 913–918. www.amsci.org/amsci/issues/Comsci97/compsci9703.html. 30. W. Krauth and M. Mézard, “The Cavity Method and the Travel- 41. R. Monasson et al., “Determining Computational Complexity ing-Salesman Problem,” Europhys. Lett., vol. 8, no. 3, 1989, pp. from Characteristic ‘Phase Transitions,’” Nature, vol. 400, July 213–218. 1999, pp. 133–137. 31. Y. Fu and P.W. Anderson, “Application of Statistical Mechanics to NP-Complete Problems in Combinatorial Optimization,” J. Physics A: Mathematical and General, vol. 19, 1986, pp. Stephan Mertens is a teaching and research assistant at 1605–1620. the Institute for Theoretical Physics, University of Magde- 32. R. Monasson and Riccardo Zecchina, “Statistical Mechanics of burg. His fields of research are statistical mechanics of the Random k-Satisfiability Model,” Phys. Rev. E, vol. 56, no. 2 hard optimization problems, phase transitions in com- Aug. 1997, pp. 1357–1370. putational complexity, and parallel algorithms. He runs 33. M. Mézard and G. Parisi, “Replicas and Optimization,” J. Physique Letters, vol. 46, 1985, pp. L771–L778. the Institute’s computing machinery, including a 156- 34. D. J. Aldous, “The ζ(2) Limit in the Random Assignment Prob- node Beowulf cluster. He received his PhD in theoretical lem,” Random Structures & Algorithms, vol. 18, no. 4, July 2001, physics from Göttingen University. Contact him at Otto- pp. 381–418. von-Guericke Univ., Inst. f. Theor. Physik, Otto-von-Gu- 35. G. Parisi, A Conjecture on Random Bipartite Matching, 1998; http://xxx.lanl.gov/PS_cache/cond-mat/pdf/9801/9801176.pdf. ericke Univ., PF 4120, D-39016 Magdeburg, Germany; 36. D. Coppersmith and G. Sorkin, “Constructive Bounds and Exact [email protected]. Expectations for the Random Assignment Problem,” Random Structures and Algorithms, vol. 15, no. 2, 1999, pp. 113–144. For more information on this or any other computing 37. S. Cocco and R. Monasson, “Statistical Physics Analysis of the Computational Complexity of Solving Random Satisfiability Prob- topic, please visit our Digital Library at http://computer. lems using Backtrack Algorithms,” European Physics J. B, vol. 22, org/publications/dlib.

2001 June May/ What is Computing in Science & Engineering magazine? CiSE is a peer-reviewed, joint ysics te of Ph n Institu merica d the A iety an ter Soc Compu e IEEE on of th publication of the IEEE Computer Society and blicati oint pu g is a j ineerin & Eng Science ting in Compu the American Institute of Physics. It represents the merger of two ﬁrst-rate scientiﬁc publications—AIP’s Computers in Physics and IEEE Computational Science & Engineering.

Now electrical engineers, physicists, chemists, and others have a magazine that covers a broad range of topics, emphasizing the common techniques and practical lessons that are

F O R neers portable from one area of CSE to another. CiSE al Engi Electric sts Physici is the leading interdisciplinary forum for those ts Chemis hers... and ot who call themselves computational scientists

. ® or engineers, or who have an interest in the rg uter.o /comp http:/ g aip.or www. subject. Specialists from many areas ﬁnd it highly readable and accessible.

Visit www.computer.org/cise or http://ojps.aip.org/cise SUBSCRIBE TODAY!

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply.