arXiv:cond-mat/9901353v2 [cond-mat.stat-mech] 14 Jun 1999 eae oteproaintasto ftegah.Near graphs. the is of point transition critical com- percolation This less the to a a connectivities. related with as lower point at graphs, critical phase of a plex connectivity has the GPP of the function spin that disordered find to We similarities systems. with partitioning [8] opti- graph problem combinatorial mization the NP-hard the standard use a As will (GPP), problem we physics. EO comparison non-equilibrium SA, for from to basis ideas contrast In on SA. based with is comparison Optimization in [4], Extremal (EO) called method, general-purpose land- [7]. the drastically change behavior SA’s may and that scape parameters context, various themselves particular possess problems a optimization in combinatorial performance typical SA’s determine of crucially quality that the standard parameters mul- the adjustable a to of from titude limited Aside when problems. hard even optimization is combinatorial general, SA of in performance assess to the applicable to But is problems. it Thus, of variety input. a as function cost re- of only the formalism principle quires in the and applies mechanics SA statistical equilibrium [5,6]. “simu- (SA) as lead annealing” such have lated methods materials, optimization disordered general-purpose of to annealing the ticular solutions near-optimal find problems. such to develop for methods to importance reliable great and unrea- of fast is usually it solution Thus, optimal costly. exact, sonably makes the which for barriers search sizable the by unrelated separated may of extrema function number local increasing cost configu- exponentially the all an size, of exhibit system space exhibits growing the For often over function rations. [3] cost “landscape” a sys- complex such the a [2], of frustrated components where is cases individual tem In between [1]. relation beyond en- the and frequently physics a in is task function cost countered some to respect with dom nti ae ewl xlr h rpriso new a of properties the explore will we paper this In par- in processes, physical certain of observation The free- of degrees many with systems of optimization The xrmlOtmzto fGahPriinn ttePercola the at Partitioning Graph of Optimization Extremal hsciia on ut well. k quite O reproduces point Extremal systems, critical hand, critical this other self-organized the of On fo is dynamics runtime. graphs equalized large at for perc mization Annealing the Simulated near gr of connectivities the error average relative of with optimization graphs the sparse problem, for (NP-hard) complex a ally piiain scmae oSmltdAnaigi extens in performance Annealing The Simulated to problem. compared is partitioning Optimization, graph the on strated h eet farcnl rpsdmto oapoiaehar approximate to method proposed recently a of benefits The 2 etrfrNnierSuis o lmsNtoa Laborato National Alamos Los Studies, Nonlinear for Center .INTRODUCTION I. 1 hsc eatet mr nvriy tat,Goga30 Georgia Atlanta, University, Emory Department, Physics tfnBoettcher Stefan My3,2018) 31, (May 1 euni hi ecinweeawaee neighbor weakened a where reaction chain to equilibrium a bound punctuated is in activity as return co-evolutionary to the referred because merely [11,13], species is these what But threshold. possess certain a above ness [9]. (SOC) criticality correlated self-organized highly up- as a such known into of state itself number random organizes certain system new a the with After dates, replaced [12]. simply well as are numbers step time number the smallest that the model, at neighboring Bak-Sneppen sites all the on of values In fitness fitness the species. impacts species: species interrelated one each an in of fitness that of change with The fitness for systems’ in- constraints the obvious provide balancing But species 1. between become terdependencies eventually would system the ihanwnme rw rmauiomdistribution uniform a from drawn [0 number on new replaced and a discarded is with worst update the each (representing at sites number species) the smallest adapted on The located lattice. “fitness,” a their of indicates between that number 1 a by and represented 0 model are Bak-Sneppen species the There, is [11]. EO of development the spired [10]. of evolution freedom the of by degrees dominated atypical is extremely system systems a often of when SOC emerges dynamics [9]. (SOC) non-equilibrium criticality self-organized the exhibiting on based [4], bandi u ueia oprsno AadEO, and V. SA results Sec. of the in and comparison conclusions algorithms by numerical problem, followed the our partitioning present in we graph in obtained IV the method, introduce Sec. EO we and the III behind philosophy Sec. the describe we errors. de- small markedly only SA produces of EO performance while the teriorates point, critical this nteSCsae lotalseishv ece fit- a reached have species all almost state, SOC the In ipeeapeo uhadnmclsse hc in- which system dynamical a such of example simple A optimization to approach new entirely an provides EO hsppri raie sflos ntenx section next the In follows: as organized is paper This 1 , onrslsaototmlpriin at partitions optimal about results nown n odvrerltv oEtea Opti- Extremal to relative diverge to und , ] ihu n neatos l h ubr in numbers the all interactions, any Without 1]. lto hehl.A hstrsod the threshold, this At threshold. olation 2 v ueia iuain.Wiegener- While simulations. numerical ive p attosi atclrydifficult particularly is partitions aph I XRMLOPTIMIZATION EXTREMAL II. fti e ehd aldExtremal called method, new this of piiainpolm r demon- are problems optimization d y o lms M855 USA 87545, NM Alamos, Los ry, tmzto,bsdo h extremal the on based ptimization, 2,USA 322, inThreshold tion can undermine one’s own fitness. Fluctuations that rear- recurrent approaches to many near-optimal configura- range the fitness of many species abound and can rise to tions. Especially in exploring low temperature proper- the size of the system itself, making any possible config- ties of disordered spin systems, those qualities may help uration accessible. Hence, such non-equilibrium systems to avoid the extremely slow relaxation behavior faced by provide a high degree of adaptation for most entities in heat bath based approaches [16]. In that, EO provides an the system without limiting the scale of change towards approach alternative – and apparently equally capable [4] even better states. – to Genetic Algorithms, which are often the only means to illuminate those important properties [17]. The parti- tioning of sparse graphs as discussed here is particularly pertinent in preparation for similar studies on actual spin glasses. It has been observed that many optimization problems exhibit critical points that separate off phases with simple cases of a generally hard problem [18]. Near such a criti- cal point, finding solutions becomes particularly difficult for local search methods which proceed by exploring for an existing solution some neighborhood in configuration space. There, near-optimal solutions become widely sep- arated with diverging barrier heights between them. It is not surprising that search methods based on heat-bath techniques like SA are not particularly successful in this highly correlated state [16]. In contrast, the driven dy- namics of EO does not possess any temperature control parameters to successively limit the scale of its fluctua- tions. Our numerical results in Sec. IV show that EO’s performance does not diminish near such a critical point. A non-equilibrium approach like EO may thus provide a general-purpose optimization method that is comple- mentary to SA: While SA has the advantage far from this critical point, EO appears to work well “where the really hard problems are” [18].
III. GRAPH PARTITIONING FIG. 1. Two random geometric graphs, N = 500, with α = 4 (top) and α = 8 (bottom) in an optimized configu- To illustrate the properties of EO and its differences ration found by EO. At α = 4 the graph barely percolates, with SA, we focus in this paper on the well-studied graph with merely one “bad” edge (between points of opposite sets, partitioning problem (GPP). In particular, we will con- masked by diamonds) connecting a set of 250 round points sider the GPP near a phase transition where the opti- m with a set of 250 square points, thus opt = 1. For the denser mization problem becomes especially difficult and pos- m graph on the bottom, EO reduced the cutsize to opt = 13. sesses many similarities with physical systems. EO attempts to utilize this phenomenology to obtain near-optimal solutions for optimization problems [14]. A. Formulation of the Problem For instance, in a spin glass system [1] we may consider as fitness for each spin its contribution to the total energy The graph (bi-)partitioning problem is easy to formu- of the system. EO would search for ground state configu- late: Take N points where N is an even number, let any rations by perturbing preferentially spins with large con- pair of two points be connected by an edge with a certain tributions. Like in the Bak-Sneppen model, such pertur- probability, divide the points into two sets of equal size bations would be local, random rearrangements of those N/2 such that the number of edges connecting both sets, poorly adapted spins, allowing for better as well as for the “cutsize” m, is minimal: m = mopt. The global con- worse outcomes at each update. In the same way as straint of an equal division of the points between the sets systems exhibiting SOC get driven recurrently towards places this problem generally among the hardest prob- a small subset of attractor states through a sequence of lems in combinatorial optimization, requiring a compu- “avalanches” [15,9], EO can fluctuate widely to escape tational effort that would grow faster than any power of local optima while the extremal selection process ensures N to determine the exact solution with certainty [8]. The
2 two physically motivated optimization methods, SA and threshold that goes beyond the percolating properties of EO, which we focus on here, usually obtain approximate these graphs. solutions in polynomial time. We note, in accordance with Ref. [24], that the criti- For random graphs, the GPP depends on the proba- cal point separates between hard cases and easy-to-solve bility p with which any two points in the system are con- cases of the GPP. The transition is related to the corre- nected. Thus, p determines the total number of edges in sponding percolation problem for the graphs in the fol- an instance, L = pN(N − 1)/2 on average, and its mean lowing manner: If the mean connectivity α is very small, connectivity per point, α = p(N −1) on average. Alterna- the graph of N points consists mainly of disconnected, tively, we can formulate a “geometric” GPP by specifying small clusters or isolated points which can be enumer- N randomly distributed points in the 2-dimensional unit ated and sorted into two equal partitions in polynomial square which are connected with each other if they are time with no edges between them (mopt = 0). If α is large located within a distance d of one another. Then, the and the probability that any two points are connected is average expected connectivity α of such a graph is given p = O(1), almost all points are connected into one giant 2 2 by α = Nπd . This form of the GPP has the advantage cluster with mopt = O(N ), and almost any partition of a simple graphical representation, shown in Fig. 1. leads to an acceptable solution. But when p = O(1/N), It is known that geometric graphs are harder to opti- i. e. α = O(1), the distribution of cluster sizes is broad, mize than random graphs [19]. The characteristics of the and the partitioning problem becomes nontrivial. Obvi- GPP for random and geometric graphs at low connec- ously, as soon as a cluster of size > N/2 appears, mopt tivity appear to be very different due to the dominance must be positive. In this sense, we observe for N → ∞ of long loops and short loops, resp., and we present re- a sharp, percolation-like transition at an αcrit with the sults for both types of graphs here. In fact, in the case cutsize mopt as the order parameter. of random graphs the structure is locally tree-like which For random graphs it is known that a cluster of size N allows for a mean-field treatment that yields exact re- exists for α> 1 [20], but only for α > αc = 2 ln 2 ≈ 1.386 sults [20–22]. In turn, the geometric case corresponds do we find a cluster of size > N/2 [21]. Geometric graphs to continuum percolation of “soft” (overlapping) circles in D = 2 are known to percolate at about α = 4.5 [23], for which precise numerical results exist [23]. Finally, we and we would expect αc for the GPP to be slightly larger also try to determine the average ground state energy than that. Also, the dilute ferro-magnet should exhibit of a dilute ferro-magnetic system on a cubic lattice at a non-trivial energy when the fraction of occupied bonds fixed (zero) magnetization, which amounts to the equal reaches slightly beyond the critical point pc ≈ 0.2488 for partitioning of “up” and “down” spins while minimiz- bond percolation on a cubic (D = 3) lattice [25], i. e. for ing the interface between both types [21]. Here, each connectivities α> 2Dpc ≈ 1.493. vertex of the lattice holds a ±-spin, and any two nearest- neighbor spins either possess a ferromagnetic coupling of unit strength or are unconnected. The probability that a IV. NUMERICAL EXPERIMENTS coupling exists is fixed such that the average connectivity of the system is α. A. Simulated Annealing Algorithm
In SA [5], we try to minimize a global cost function 2 B. Graph Partitioning and Percolation given by f = m + µ(P1 − P2) , where P1 and P2 are the number of points in the respective sets. Allowing the Like many other optimization problems, the GPP ex- size of the sets to fluctuate is required to improve SA’s hibits a critical point as a function of its parameters [18]. performance in outcome and computational time at the In case of the GPP we observe this critical point as a cost of an arbitrary parameter µ to be determined. Then, function of the connectivity α of graphs, with the cutsize starting at a “temperature” T0, the annealing schedule mopt as the order parameter. In fact, the critical point of proceeds with lN trial Monte-Carlo steps on f by tenta- partitioning is closely linked to the percolation threshold tively moving a randomly chosen point from one set to of graphs. In our numerical simulations we proceed by the other (which changes m) to equilibrate the system. averaging over many instances of a class of graphs and This move is accepted, if f improves or if the Boltzmann try to reproduce well-known results from the correspond- factor exp[(fold − fnew)/T ] is larger than a randomly ing percolation problem. Of course, using stochastic op- drawn number between 0 and 1. Otherwise the move timization methods (instead of cluster enumeration) is is rejected and the process continues with another ran- neither an efficient nor a precise means to determine per- domly chosen point. After that, we set Ti = Ti−1(1 − ǫ), colation thresholds. But in turn we obtain also some equilibrate again for lN trials, and so on, until the MC valuable information about the scaling behavior of the acceptance rate drops below Astop for K consecutive tem- average cost < mopt > for optimal partitions near the perature levels. At this point the optimization process can be considered “frozen” and the configuration should
3 be near-optimal, m ≈ mopt (and balanced, P 1 = P 2). value of τ should exist: If τ is too small, points would be While SA is intuitive, controlled, and of very general picked purely at random with no gradient towards a good applicability, its performance in practice is strongly de- partition, while if τ is too large, only a small number of pendent on the multitude of parameters which have to points with particularly bad fitness would be chosen over be arduously tuned. For us it is thus expedient (and and over again, confining the system to a poor local op- most unbiased!) to rely on an extensive study of SA timum. It is a surprising numerical result that this value for graph partitioning [19] which determined µ = 0.05, of τ appears to be rather universal, independent of N, α, T0 = 2.5, ǫ = 0.04, Astop = 2%, and K = 5. Ref. [19] and the type of graph considered. set l = 16, but performance improved noticeably for our choice, l = 64. C. Testbed of Graphs
B. Extremal Optimization Algorithm In our numerical simulations we have generated ran- dom and 2D geometric graphs of varying connectivity by In EO [4], each point i obtains a “fitness” λi = gi/(gi + choosing p or d, resp. For any instance of a graph labeled bi) where gi and bi are the number of “good” and “bad” by a “connectivity α”, the actual connectivity not only edges that connect that point within its set and across varies from point to point, but also the mean connectivity the partition, resp. (We fix λi = 1 for isolated points.) of such graphs follows a normal distribution. (In particu- Of course, point i has an individual connectivity of αi = lar for geometric graphs it is shifted to lower values due to gi + bi while the overall mean connectivity of a graph is the loss of connectivity at the boundaries.) For N = 500, given by α = Pi αi/N. The current cutsize is given by 1000, 2000, 4000, 8000, and 16000, we varied the connec- m = Pi bi/2. At all times, an ordered list λ1 ≤ λ2 ≤ tivity between α = 1.25 and α = 5 for random graphs, ... ≤ λN is maintained where λn is the fitness of the and α = 4 and α = 10 for geometric graphs. Then, point with the n-th rank in the list. for each α we generated 16 different instances of graphs, At each update we draw two numbers, 1 ≤ n1,n2 ≤ N, identical for SA and EO. On each instance, we performed from a probability distribution 8 (32) optimization runs for random (geometric) graphs, both for EO and SA. Each run, we used a new random −τ P (n) ∼ n . (1) seed to establish an initial partition of the points. SA’s runs terminate when the system freezes. We terminated Then we pick the points which are elements n1 and n2 of EO-runs after 200N updates, leading to a comparabile the rank-ordered list of fitnesses. (We repeat a drawing runtime between both methods. of n2 until we obtain a point that is from the opposite 100 set than n1.) These two points swap sets no matter what the resulting new cutsize m may be, in notable distinc- tion to the (temperature-) scale-dependent Monte Carlo update in SA. Then, these two points, and all points they are connected to (2α on average), reevaluate their fitness λ. Finally, the ranked list of λ’s is reordered using a 10 “heap” at a computational cost ∝ α ln N, and the pro- cess is started again. We repeat this process for a number CPU−time in sec of update steps per run that rises linearly with system size, and we store the best result generated along the 1 way. Note that no scales are introduced into the process, 1.5 2 2.5 3 3.5 4 since the selection follows a scale-free power-law distri- Connectivity α bution P (n) and since – unlike in a heat bath – all moves are accepted, allowing for fluctuations on all scales. In- FIG. 2. Typical runtime comparison of SA and EO for the ferromagnet on a 200MHz Pentium. Runtimes generally rise stead of a global cost function, the rank-ordered list of ∝ N, fall with the connectivity α for SA (filled symbols), fitnesses provides the information about optimal config- but rise ∝ α for EO (unfilled symbols). Circles refer to an urations. This information emerges in a self-organized average runtime of graphs with N = 512 = 83 points, squares manner merely by selecting with a bias against badly to N = 1728 = 123, diamonds to N = 4096 = 163, and adapted points, instead of “breeding” better ones [11]. triangles to N = 8000 = 203. There is merely one parameter, the exponent τ in the probability distribution in Eq. (1), that controls the se- For the dilute ferro-magnet, we fixed the number of lection process and optimizes the performance of EO. In couplings to obtain a specific average connectivity α. initial studies, we determined τ = 1.4 as the optimal Those couplings were then placed on random links be- value for all graphs. It is intuitive that such an optimal tween nearest-neighbor spins to generate an instance. We
4 used 16 instances, and 16 runs for each, at connectivities 1. Comparison of EO and SA 1.6 ≤ α ≤ 4. Here, we only used 100N updates for EO, and the temperature length of 16N recommended We evaluate the performance of SA and EO separately. in Ref. [19] but with a higher starting temperature for For each method, we only take its best result for each SA to optimize performance at a comparable runtime for instance and average those best results at any given con- both methods, as shown in Fig. 2. nectivity α to obtain the mean cutsize for that method
102 as a function of α and N. To compare EO and SA, we N=1000 determine the relative error of SA with respect to the N=2000 N=4000 best result found by either method for α ≥ αc. Figs. 3a- N=8000 1 c show how the error of SA diverges with increasing N 10 N=16000 near to αc for each class of graphs. Depending on the type of graph under consideration, the quality of the SA results may vary. The data for 0
Rel. Error in % 10 random graphs in Fig. 3a only shows a relatively weak deficit in SA’s performance relative to EO. Near αc = 2 ln 2 = 1.386, SA’s relative error remains modest, and
10−1 only grows very weakly with increasing N. For large con- 1.5 2 2.5 3 3.5 4 nectivities α, SA quickly becomes the superior method α Connectivity for random graphs, which may be due to their increas- ingly homogeneous structure (i. e. low barriers between 4 N=500 10 N=1000 optima) that does not favor EO’s large fluctuations. On N=2000 the other hand, the averages obtained by EO appear to N=4000 be very smooth (see the scaling in Fig. 4a) whereas the 103 N=8000 N=16000 apparent noise in Fig. 3a indicates large variations be- tween instances for the SA results. 102 The very rugged structure of geometric graphs near Rel. Error in % the percolation threshold (see Fig. 1a), αc ≈ 4.5, is most
1 problematic for SA, leading to huge errors which appear 10 to increase linearly with N. Barriers between optima are
4 5 6 7 8 9 10 high within each graph, now favoring EO’s propensity Connectivity α for large fluctuations. On the scale of Fig. 3b, error bars attached to the data (which we have generally omitted) 103 N=512 would hardly be significant. But experience shows that N=1728 both methods exhibit large variations in results between N=4096
2 N=8000 instances which is in large part due to actual variations 10 in the structure between geometric graphs. The results for the dilute ferromagnet exhibit a mix 101 of the two previous cases. Since the points are arranged
Rel. Error in % on a D = 3-lattice, the structure of these graphs is def- initely geometrical, but local connectivities are limited 0 10 to the 2D = 6 nearest neighbors that each point pos- sesses. Again, SA’s error is huge and appears to diverge 1.5 2 2.5 3 3.5 4 about linearly near the threshold, αc ≈ 1.5. But due to Connectivity α the limited rage of connectivities, graphs soon become FIG. 3. Plot of the error of SA relative to the best re- rather homogeneous for increasing α which in turn ap- sult found on (a) random graphs (top), (b) geometric graphs pears to favor SA away from the transition, especially (middle), and (c) the dilute ferromagnet (bottom) as func- for larger graphs. (For larger N, any local structure gets tion of the mean connectivity α. While the error near αc only quickly averaged out due to the local limits on the con- increases slowly for random graphs, it appears to increase lin- early with N for the ferromagnet and for geometric graphs. nectivity, whereas an unlimited range of local structures At fixed but large connectivities, SA increasingly gains on EO can emerge in the geometric graphs above.) for rising N.
2. Scaling of EO-Data near the Transition D. Evaluation of Results For the data obtained with EO, we make an Ansatz
5 ν β hmopti∼ N (α − αc) (2) we can not tune ourselves arbitrarily close to a critical point. Furthermore, the problem is non-trivial only when to scale the data for all N onto a single curve, shown in α ≥ 3. These graphs have the property that at a given α Figs. 4a-c. From the scaling Ansatz we can extract an and N the optimal cutsizes between instances vary little, estimate for αc to compare with percolation results as a and only few instances are needed to determine hmopti measure of the accuracy of the data obtained with EO. with good accuracy. Furthermore, we also obtain a numerical estimates for 0.3 the exponents ν and β which characterize the transition. N=500 The exponent ν, describing the finite-size scaling behav- N=1000 N=2000 ior, could be infered from general, global properties of a N=4000 class of graphs. For instance, ν = 1 for random graphs 0.2 N=8000 because any global property of these graphs is extensive N=16000
>/N Fit
[21]. On the other hand, the exponent β, describing the opt 0.6 0.4 of β, which would ignore any error received through the Fit >/N limited number of instances averaged over, or any bias opt For geometric graphs in Fig. 4b, we found the best scal- 0 ing for ν =0.6. Since we used only 16 different instances 4 5 6 7 8 9 10 Connectivity α to average over at each N and α, the data gets very noisy for larger connectivities due to large fluctuations in the N=512 optimal cutsizes between those instances and/or EO’s in- 0.6 N=1728 ability to find good approximations. We chose to fit only N=4096 N=8000 points up to α = 7 and obtained β = 1.4 and αc ≈ 4.1, Fit even smaller than the critical value for percolation, 4.5. 3/4 0.4 Obviously, the obtained values are very poor, but at least >/N opt indicate EO’s ability to approximate the optimal cutsizes 6 normalized average energy [13] Gould, S. J. & Eldridge, N. Paleobiology 3, 115-151 (1977). 4hmopti E = −1+ (3) [14] S. Boettcher and A. G. Percus, in GECCO-99: Proceed- αN ings of the Genetic and Evolutionary Computation Con- of −0.840, presumably correct to the digits given. We ference (Morgan Kaufmann, San Francisco, 1999), to ap- found by averaging over 32 instances, using 8 EO runs pear. on each, for N = 1024, 2048, and 4096 that E = [15] D. Dhar, Phys. Rev. Lett. 64, 1613 (1990). −0.844±0.001. But this result is still significantly higher [16] For those issues see, for instance, Applications of the than some theoretical predictions [26,21], and we will in- Monte Carlo Method in Statistical Physics, Ed. K. Binder vestigate whether longer runtimes may further reduce the (Springer, Berlin, 1987), especially Sec. 1.1.5 and related references. cutsizes for these graphs [28]. [17] J. Houdayer and O. C. Martin, Ising Spin Glasses in a Magnetic Field, available at cond-mat/9811419. [18] P. Cheeseman, B. Kanefsky, and W. M. Taylor, in Proc. V. CONCLUSIONS of IJCAI-91, edited by J. Mylopoulos and R. Rediter (Morgan Kaufmann, San Mateo, CA, 1991), pp. 331–337. In this paper we have demonstrated that Extremal [19] D. S. Johnson, C. R. Aragon, L. A. McGeoch, and C. Optimization (EO), a new optimization method derived Schevon, Operations Research 37, 865 (1989). from non-equilibrium physics, may provide excellent [20] P. Erd¨os and A. R´enyi, in The Art of Counting, ed. J. results exactly where Simulated Annealing (SA) fails. Spencer (MIT, Cambridge, 1973). While further studies will be necessary to understand [21] M. Mezard and G. Parisi, Europhys. Lett. 3, 1067 (1987); (and possibly, predict) the behavior of EO, we have used K. Y. M. Wong and D. Sherrington, J. Phys. A: Math. it here to analyze the phase transition in the NP-hard Gen 20, L793 (1987); [22] S. Janson, D. E. Knuth, T. Luczak, and B. Pittel, Ran- graph partitioning problem. The results illustrate con- dom Structures & Algorithms 4, 233-358 (1993). vincingly the advantages of EO and produce a new set [23] I. Balberg, Phys. Rev. B 31, R4053 (1985). of scaling exponents for this transition for a variety of [24] Y. T. Fu and P. W. Anderson, J. Phys. A: Math. Gen different graphs. 19, 1605 (1986). I thank A. Percus, P. Cheeseman, D. S. Johnson, D. [25] D. Stauffer and A. Aharony, Percolation Theory, (Taylor Sherrington, and K. Y. M. Wong for very helpful discus- & Francis, London, 1992). sions. [26] K. Y. M. Wong and D. Sherrington, J. Phys. A: Math. Gen 20, L793 (1987), and K. Y. M. Wong, D. Sherring- ton, P. Mottishaw, R. Dewar, and C. De Dominicis, J. Phys. A: Math. Gen 21, L99 (1988). [27] J. R. Banavar, D Sherrington, and N. Sourlas, J. Phys. A: Math. Gen 20, L1 (1987). [28] S. Boettcher and A. G. Percus (in preparation). [1] M. Mezard, G. Parisi, and M. A. Virasoro, Spin Glass Theory and Beyond (World Scientific, Singapore, 1987). [2] G. Toulouse, Comm. Phys. 2, 115 (1977). [3] See Landscape Paradigms in Physics and Biology, eds. H. Frauenfelder et. al. (Elsevier, Amsterdam, 1997). [4] S. Boettcher and A. G. Percus, Artificial Intelligence (to appear), available at cond-mat/9901351. [5] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, Science 220, 671 (1983). [6] V. Cerny,ˇ J. Optimization Theory Appl. 45, 41 (1985). [7] G. B. Sorkin, Algorithmica 6, 367-418 (1991). [8] M. R. Garey and D. S. Johnson, Computers and In- tractability, A Guide to the Theory of NP-Completeness (W. H. Freeman, New York, 1979). [9] P. Bak, C. Tang, and K. Wiesenfeld, Phys. Rev. Lett. 59, 381 (1987). [10] M. Paczuski, S. Maslov, and P. Bak, Phys. Rev. E 53, 414 (1996). [11] P. Bak and K. Sneppen, Phys. Rev. Lett. 71, 4083 (1993). [12] In the Bak-Sneppen model, the relation between species is not specified in detail. In reality, some fixed (“quenched”) structure may exist between any two species that determines how the adaptive change in one effects the other. 7