L IMITS OF C OMPUTATION Copyright (c) 2002 Institute of Electrical and Electronics Engineers. Reprinted, with permission, from Computing in Science & Engineering. This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way imply IEEE endorsement of any of the discussed products or services. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by sending a blank email message to [email protected]. By choosing to view this document, you agree to all provisions of the copyright laws protecting it. COMPUTATIONALCOMPLEXITY FOR PHYSICISTS The theory of computational complexity has some interesting links to physics, in particular to quantum computing and statistical mechanics. This article contains an informal introduction to this theory and its links to physics. ompared to the traditionally close rela- mentation as well as the computer on which the tionship between physics and mathe- program is running. matics, an exchange of ideas and meth- The theory of computational complexity pro- ods between physics and computer vides us with a notion of complexity that is Cscience barely exists. However, the few interac- largely independent of implementation details tions that have gone beyond Fortran program- and the computer at hand. Its precise definition ming and the quest for faster computers have been requires a considerable formalism, however. successful and have provided surprising insights in This is not surprising because it is related to a both fields. This is particularly true for the mutual highly nontrivial question that touches the foun- exchange between statistical mechanics and the dation of mathematics: What do we mean when we theory of computational complexity. Here, I discuss say a problem is solvable? Thinking about this this exchange in a manner directed at physicists question leads to Gödel’s incompleteness theo- with little or no knowledge of the theory. rem, Turing machines, and the Church-Turing thesis on computable functions. Here we adopt a more informal, pragmatic The measure of complexity viewpoint. A problem is solvable if a computer The branch of theoretical computer science program written in your favorite programming known as computational complexity is con- language can solve it. Your program’s running cerned with classifying problems according to time or time complexity must then be defined with the computational resources required to solve some care to serve as a meaningful measure of them (for additional information about this field, the problem’s complexity. see the “Related Works” sidebar). What can be measured (or computed) is the time that a par- Time complexity ticular algorithm uses to solve the problem. This In general, running time depends on a prob- time, in turn, depends on the algorithm’s imple- lem’s size and on the specific input data—the in- stance. Sorting 1,000 numbers takes longer than sorting 10 numbers. Some sorting algorithms 1521-9615/02/$17.00 © 2002 IEEE run faster if the input data is partially sorted al- ready. To minimize the dependency on the spe- STEPHAN MERTENS cific instance, we consider the worst-case time Otto-von-Guericke University, Magdeburg complexity T(n): MAY/JUNE 2002 31 strongly on the algorithm’s implementation de- Related Work tails—smart programmers and optimizing com- For introductions into the field of computational complexity, see pilers will try to reduce it. Therefore, rather than E.L. Lawler et al., eds., The Traveling Salesman Problem, Wiley-Interscience Series consider the precise number T(n) of elementary in Discrete Mathematics and Optimization, John Wiley & Sons, New York, 1985. operations, we only consider the asymptotic be- H.R. Lewis and C.H. Papadimitriou, “The Efficiency of Algorithms,” Scientific havior of T(n) for large values of n as the Lan- American, vol. 109, no. 1, Jan. 1978, pp. 96–109. dau symbols O and Θ denote: C.H. Papadimitriou and K. Steiglitz, Combinatorial Optimization, Prentice-Hall, Englewood Cliffs, N.J., 1982. • We say T(n) is of order at most g(n) and write O For a deep understanding of the field, read the following classic T(n) = (g(n)) if positive constants c and n0 ex- ≤ ≥ textbooks: ist such that T(n) cg(n) for all n n0. • We say T(n) is of order g(n) and write T(n) M.R. Garey and D.S. Johnson, Computers and Intractability: A Guide to the Theory Θ of NP-Completeness, W.H. Freeman, New York, 1997. = (g(n)) if positive constants c1, c2, and n0 exist ≤ ≤ ≥ C.H. Papadimitriou, Computational Complexity, Addison-Wesley, Reading, such that c1g(n) T(n) c2g(n) for all n n0. Mass., 1994. Multiplying two n × n matrixes requires n3 The complexity of Ising spin systems is discussed in: multiplications, according to the textbook for- B. Hayes, “Computing Science: The World in a Spin,” American Scientist, vol. 88, no. mula. However, this does not mean that the 5, Sept./Oct. 2000, pp. 2384–388; www.amsci.org/amsci/issues/comsci00/ problem of multiplying two n × n matrices has compsci2000-09.html. complexity Θ(n3). The textbook formula is a par- For an entertaining introduction (written by a physicist) to ticular algorithm, and an algorithm’s time com- Gödel’s incompleteness theorem, Turing machines, and the plexity is only an upper bound for a problem’s in- Church-Turing Thesis on computable functions, read herent complexity. In fact, researchers have found faster matrix multiplication algorithms with com- R. Penrose, The Emperor’s New Mind, Oxford Univ. Press, 1989. α plexity O (n ) and α < 3 during the last decades— α 1 To learn more about quantum parallelism, see www.qubit.org or the current record being = 2.376. Because the product matrix has n2 entries, α cannot be smaller D. Aharonov, “Quantum Computation,” Ann. Rev. Computational Physics VI, D. Stauffer, ed., World Scientific, 1998. than 2; it is an open question whether an algo- rithm can achieve this lower bound. Phase transitions in computational complexity are discussed in: A problem where the upper bound from algo- O. Dubois et al., eds., “Phase Transitions in Combinatorial Problems,” special rithmic complexity meets an inherent lower issue of Theor. Comp. Sci., vol. 265, nos. 1–2, 2001. bound is sorting n items. Under the general as- B. Hayes, “Computing Science: The Easiest Hard Problem,”American Scientist, sumption that comparisons between pairs of vol. 90, no. 2, Mar./Apr. 2002, pp. 113–117; www.amsci.org/amsci/issues/ items are the only source of information about comsci02/compsci2002-03.html. them, Θ(n log n) is a lower bound for the num- B. Hayes, “Computing Science: Can’t Get No Satisfaction,” American Scientist, vol. 85, no. 2, Mar./Apr. 1997, pp. 108–112; www.amsci.org/amsci/issues/ ber of comparisons to sort n items in the worst 2 Comsci97/compsci9703.html. case. This bound is met by algorithms such as “heapsort” or “mergesort.” Problem size Tn()= max(), tx (1) Our measure of time complexity still depends = xn on the somewhat ambiguous notion of problem where t(x) is the algorithm’s running time for in- size. In the matrix multiplication example, we put data x (in arbitrary units), and the maximum tacitly took the number n of rows of one input is taken over all problem instances of size n. The matrix as the “natural” measure of size. Using worst-case time is an upper bound for the ob- the number of elements m = n2 instead speeds up servable running time. the O (n3) algorithm to O (m3/2) without chang- A measure of time complexity should be based ing a single line of program code. An unam- on a unit of time that is independent of the spe- biguous definition of problem size is required to cific CPU’s clock rate. Such a unit is provided by compare algorithms. the time it takes to perform an elementary op- In computational complexity, all problems eration such as adding two integer numbers. solvable by a polynomial algorithm—that is, an Measuring the time in this unit means counting algorithm with time complexity Θ(nk) for some the number of elementary operations your algo- k—are lumped together and called tractable. rithm executes. This number in turn depends Problems that can only solvable by algorithms 32 COMPUTING IN SCIENCE & ENGINEERING n with nonpolynomial running time, such as Θ(2 ) 8 7 or Θ(n!), are also lumped together and called in- 4 9 tractable. There are practical as well as theoreti- 2 cal reasons for this rather coarse classification. 14 11 4 One of the theoretical advantages is that it does 7 6 not distinguish between the O (n3) and O (m3/2) 8 10 algorithm example from above; hence, we can 2 afford some sloppiness and stick with our am- 1 biguous natural measure of problem size. Figure 1. A weighted graph and its minimum spanning tree (shaded edges). Tractable and intractable problems The terms tractable and intractable for prob- lems with polynomial and exponential algo- a spanning subgraph) and whose edges have rithms refer to the fact that an exponential al- minimum total weight. Your subgraph should gorithm means a hard limit for the accessible not contain cycles because you can always re- problem sizes. Suppose that, with your current move an edge from a cycle, keeping all nodes equipment, you can solve a problem of size n connected and reducing the cost. A graph just within your schedule. If your algorithm has without cycles is a tree, so what you are look- complexity Θ(2n), a problem of size n + 1 needs ing for is a minimum spanning tree (MST) in a twice the time, pushing you off schedule.