Complex Networks the Challenge of Interaction Topology

Complex Networks The Challenge of Interaction Topology

Zoltán Toroczkai

Networks have recently become a paradigmatic way of representing complex systems in which the pattern of interactions between a system’s constituent parts is itself complex and is evolving together with the system’s dynamics. Transport is the main function of these dynamic networks. It is therefore crucial that we understand the coupling between the network structure and the efficiency and robustness of the transport processes on the structure. Such understanding will have a huge impact, allowing us to control signaling processes in the cell and to design robust information and energy-transmission infrastructures, such as the Internet or the power grid. However, achieving this type of understanding is rather challenging, because of the discrete and random nature of network topology. This article reports on some of our results that connect network dynamics and transport efficiency. It also illustrates the power behind the ability to control the topology of the interactions in the design of scalable computer networks.

The roadways of Portland, Oregon.

94 Los Alamos Science Number 29 2005

Complex Networks

Macroscopic snapshot of Internet connectivity (skitter data) with selected backbone Internet service providers. (This photo is courtesy of the Lumeta Corporation.)

The metabolic pathway. (Gerhard Michal: Biochemical Pathways,1999 © Elsevier GmbH, Spektrum Akademischer Verlag, Heidelberg.)

ystems of many interacting par- airlines, and others) social interactions of connections among them. For the ticles typically exhibit complex (acquaintance networks, scientific col- last decade, the science of complex Sbehavior. In most well-known laboration networks, terrorist net- networks has focused on describing complex systems, the topology of the works, sex webs, and others), commu- the structural complexity of real- interactions between particles can be nications networks (the World Wide world network topologies. By looking described by simple structures, such Web, the Internet, microwave back- at the three images on these opening as regular crystalline lattices or a con- bone, and telephone networks), bio- pages, one can easily surmise, that tinuum, and the complex behavior logical networks (metabolic networks, statistical and probabilistic methods arises from nonlinearity and nonlocal- gene regulatory networks, protein are essential to that description. ity, which describe the nature of the interaction networks), and networks in Today, the focus has expanded interactions themselves. There is, ecology (food webs). Although these beyond network structure to an however, a large class of systems systems have been known for a while, understanding of the relationship called complex networks, in which the their complexity has been explored between structure and dynamics and interactions are mediated not by a only recently because the large data- the implications of that relationship continuum (or a simple regular struc- bases and the immense computational for network design. The first half of ture) but by a complex graph, whose power required to analyze network this article traces the main ideas in structure may evolve as part of the data were almost nonexistent two graph theory over the past two cen- dynamics of the interactions. decades ago. turies, which are at the basis of the Familiar systems in almost every Even a cursory “look” at the struc- mathematical approach to networks, area of life form such complex net- ture of real-world networks creates a and the second half is devoted to works. Here are a few examples: breathtaking impression: These are some very recent developments: transport and transportation infrastruc- large objects containing thousands, or computer network design and the tures (electric power grids, water- sometimes, even hundreds of mil- connection between network ways, natural gas pipelines, roadways, lions, of nodes with an intricate mesh dynamics and structure.

Number 29 2005 Los Alamos Science 95

Complex Networks

The Problem of the Königsberg Bridges

Network images can be quite strik- ing. But one might question whether thinking about complex systems in terms of networks leads to more than pretty pictures. Ironically, the funda- mentals of the theory of network structures were introduced by a blind mathematician. It all began with the puzzle of seven bridges, an entertaining brainteaser for people who strolled through Königsberg, the Prussian city at the Baltic Sea, in the 18th century. The river Pregel divides the city into four land areas connected by seven bridges. The burghers of Königsberg wondered if one could visit all the four areas by crossing each bridge exactly once (see Figure 1). Figure 1. The Königsberg Puzzle The puzzle was solved in 1736 by This woodcut shows the ancient Prussian city of Königsberg (now known as Leonhard Euler, who at the time, was Kaliningrad) with its seven bridges across the river Pregel. The possibility of a mathematics professor in St. strolling across the city by crossing each bridge once only became the object of a Petersburg. The power of Euler’s famous brainteaser. solution lies not in the answer itself (which is negative) but in the way it was derived. Euler’s revolutionary idea was to represent the pieces of land separated by bridges as the nodes (dots) A, B, C, and D and to represent the bridges as the edges (line seg- ments) a, b, c, d, e, f, and g, connect- ing the nodes (see Figure 2). The structure formed by the set of nodes and edges, called a graph, is a simpli- fied representation of the puzzle, encoding the relationships between the pieces of land and the paths of access between them (see Figure 2 inset). In this representation, the problem translates into the following one: Find a path that visits all nodes but passes through all edges exactly once. Obviously, the intermediate nodes must have an even number of incident edges (if one visits an intermediate node, one must also leave it). Figure 2. Euler’s Solution to the Königsberg Puzzle Because the Königsberg puzzle has A simple representation of the problem by a graph helps realize that there is no such path that visits all nodes and passes through all seven edges exactly once. 4 (>2) nodes, all with an odd number of edges, there can be no such path.

96 Los Alamos Science Number 29 2005

Complex Networks

Euler’s representation of the relationships between a discrete set of entities as a graph led to the development of a particular type of mathematical nomenclature and ultimately to a new field of discrete mathematics called graph theory.

A Hard Problem: The Ramsey Numbers

For nearly 200 years, graph theory Figure 3. The Party Problem for k = 3 was concerned with topological Six is the minimum number of people that always contains a group of three, all of and/or geometrical properties of small whom either know each other (red links) or do not know each other (blue links). (a) structures, or regular structures (such A complete graph for six people is shown. Note that it contains the complete sub- as a lattice). Then, the 1951 seminal graph CDE. (b) This figure demonstrates that there is no way to draw a complete paper by Ron Solomonoff and Anatol graph without constructing a complete single-color three-node subgraph within it. Rapoport (1951) and the 1959–1960 Suppose that blue links indicate that A does not know C, D, or E. In that case, CD, series of papers by Pál Erdo″s and DE, and EC must be red links so that a complete blue subgraph should not be Alfréd Rényi caused the rebirth of formed. But then, as shown in (b), those three red links form a complete subgraph, which means that C, D, and E know each other. graph theory. These papers introduced the notion of a random graph and, more important, that of graph ensem- If there are n = 5 people present, one bles, which are sets of graphs that can easily color a complete graph share a given property Q. To under- with no such triangle present, ruling stand this notion, let us look briefly at out n = 5 by inspection. the famous Party Problem and the For n = 6, however, there is always Ramsey numbers. This problem, at least one such triangle. To prove inspired from social interactions, is this statement, let us assume the oppo- stated very simply: site, namely, that there can be no such What is the minimum number of triangles. Since for every node there guests, R, one should invite to a party are n – 1 = 5 incident edges but only that would surely have k people who two colors, there must be 3 edges of all know each other or k who do not the same color incident on the node. know each other (at all)? Leonhard Euler For example, consider the edges AC, For k = 3, it is easy to prove that Leonhard Euler, the most prolific mathe- AD, and AE in Figure 3(b) to be the R(3) = 6. We will use Euler’s method: matician of all times, was born in same color, for example, blue. Since Let us denote the six people by the Switzerland in 1707 and spent his life in the triangles ACE, ACD, and ADE nodes A, B, C, D, E, and F. Let us Berlin and St. Petersburg. Opera Omnia cannot have all three of their edges of represent the fact that two people is an incomplete collection of his works the same color, CE, CD, and DE must know each other by drawing a red that has 73 volumes, each over 600 be red. Then CDE is a triangle all pages in length. link (or edge) between them and use a with the same color edges (red), a blue edge to link two people who do contradiction. Hence, R(3) = 6. not know each other. Since pairs of version of the Party Problem is thus to For k = 4, the answer is R(4) = 18, people either know each other or do determine the minimum number of which is hard to prove. For k = 5 and not, the graph obtained is complete, nodes n, such that a complete graph higher, the answers are not known; which means that all possible edges with n nodes and with edges of two only some bounds exist. Although we are drawn—see Figure 3(a). color always has at least one complete have no proof for k = 5, one might Specifically, a complete graph with n subgraph of k nodes with all edges of think that we would surely be able to nodes, denoted here by Kn, always has the same color. For k = 3, a complete use today’s supercomputers to find the n(n – 1)/2 edges. The graph theoretic subgraph is a monochromatic triangle. value of R(5). However, as Bollobás,

Number 29 2005 Los Alamos Science 97

Complex Networks

an eminent graph theoretician, has and ownership and his habit of living stated (1998) “…a head-on attack by out of a suitcase and visiting one computers for R(5) is doomed to fail- mathematician friend after the next ure . . . .” This failure is largely due to were symptoms, perhaps, of his total the combinatorial explosion in the dedication to mathematics. Erdo″s is number of ways we can draw a com- considered by many to be the second plete graph with n nodes using two most productive mathematician of all colors for the edges: On the face of it, times, after Euler. Possibly the great- a computer would have to search a est contribution of Erdo″s is his intro- total of 2n(n–1)/2 such graphs for com- duction of the probabilistic method plete subgraphs with k nodes. For k = in discrete mathematics. For graph 3, when n = R(3) = 6, there are 215 = theory, this means that, instead of 32,768 complete graphs, for k = 4, the asking for detailed properties of all analytic solution gives n = 18, which graphs in an ensemble, we are asking means that there are 2153, or approxi- for average properties, or the proba- mately 1.46 × 1046 graphs. For k = 5, bility that a graph from an ensemble the best known bounds are 43 ≤ R(5) has the property Q. The probabilistic ″ ≤ 49, which would mean approxi- Pál Erdos (1913–1996) method was definitely not new when Pál Erdo″s, who introduced the notion of mately 2903 to 21176 graphs (or on the Erdo″s introduced it to discrete math- random graphs, is probably the second 301 order of 10 graphs). For k = 5 and most prolific mathematician of all times, ematics: By the end of the 19th cen- n = 43, the “ultimate laptop” of Seth having produced over 1500 publications tury, Boltzmann, Gibbs, and others Lloyd (2000), which operates at the with 507 coauthors. had laid down the foundations of physical limit of computation (as equilibrium statistical mechanics, determined by the speed of light, the which is based on applying the prob- Planck constant, and the gravitational the form of partition calculus, combi- abilistic method to ensembles of constant), performing f = 5.4258 × natorics, ergodic theory, logic, analy- microstates and characterizing 1050 operations per second, would sis, algebra, geometry and computer macroscopic properties of the system have to work for at least 2.693 × science. by the properties of the “typical” 10213 years, a mighty long time (the Ultimately, the Party Problem sug- microstates. This natural connection age of the universe is estimated to gests that, if we partition a set into a between statistical mechanics and be between 1.1 × 109 and 2 × 109 fixed number of classes, order must graph theory is currently being years). emerge for large enough sets. This exploited by some research groups So, can we hope ever to solve the principle is also illustrated by van der worldwide, including the Statistical Party Problem? The key idea is to Waerden’s theorem (Bollobás 1998), Physics of Infrastructure Networks understand how different colorings of which states that, for a given k and p, team at Los Alamos. Besides the Kn relate to one another via transfor- if we partition the first w integers into combinatorial explosion in the num- mations, which would allow us to par- k classes, we will always find a class ber of possible graphs (or states), tition the set of two colorings of Kn that contains an arithmetic progres- there is a second strong reason that into a smaller number of classes and, sion with p terms for large enough w. calls for the use of the probabilistic in the absence of a full mathematical Problems like the Party Problem lead method: incomplete information. theory, to program the computer to to a simple conclusion: In order to Real-world networks, as we will see search for the monochromatic com- understand properties of graphs, one from the following sections, are in plete subgraphs on the set of classes has to think in terms of ensembles of many cases very dynamic, with new instead of the full set. Although still graphs that share a certain property, Q. edges and nodes appearing and old unsolved, intense activity in this area ones disappearing as a result of sto- led to a number of generalizations of chastic processes. In addition, in this problem and to the development A Revolutionary Idea some cases, it is hard, or even impos- of a huge branch of mathematics, the sible, to identify precisely the graph Ramsey theory (Graham et al. 1990). The Hungarian mathematician Pál structure at a given moment. Again, a That theory has a number of very Erdo″s was one of the main pioneers of good example is supplied by a prob- deep results that go well beyond the ensemble approach. His complete lem related to social networks, name- graph theory, affecting set theory in disregard for the notion of possession ly, the Gossip Problem:

98 Los Alamos Science Number 29 2005

Complex Networks

Suppose that person A in a set of N tains most of the nodes, and the prob- the binomial distribution in Equation people has a very interesting piece of ability for a node not to belong to this (1). In the limit of N →∞and p → 0 information or gossip. On average, cluster decreases exponentially fast for such that λ = pN = constant, the bino- how many acquaintances must every p > pc. Physicists call this phenome- mial goes into the Poisson distribu- person in N have such that the gossip non percolation. Passing through pc tion: becomes known to all? (by the process of increasing the aver- Since we do not know who knows age number of incident edges), the whom, we determine the answer by network suffers a drastic change, (2) considering all graphs of N people which is called a phase transition in with the nodes representing the indi- the language of physics. viduals and the edges representing the We now introduce one of the most Figure 4 shows a comparison acquaintanceships, or social links, for important characteristics of random between the formula in (2) and the transmitting gossip. In other words, if graphs, namely, their degree distribu- measured degree distribution for a persons A and B are linked and one of tion. The degree of a node x is the binomial graph of N = 20,000 nodes them knows the gossip, we can number k(x) of incident edges on that and a link probability p = 20/N = assume the other knows it too. The node. The degree distribution of the 0.001. It shows that, indeed, the answer to the Gossip Problem must binomial random graph G(N,p) is the approximation is good. The Poisson be probabilistic in nature: It is the probability that the number of nodes distribution P(k) has a “bell curve” class of graphs having nodes with a Xk with degree k is y. In a G(N,p), the shape, with a peak at λ = pN, and fast certain average number of links and probability of a node being connected decaying tails. The degree of a node characterized by the property that to k specific other nodes and not con- characterizes how a node “sees” its everyone knows the gossip in the nected to the rest of N – 1 – k nodes is immediate neighborhood in the net- end. Erdo″s and Rényi came up with a pk(1 – p)N–1–k. Because the number of work. According to the formula in (2), rather surprising solution: Once a ways to connect those k nodes is if we keep λ = pN a constant, while node has on average one link, the equal to the binomial coefficient increasing the size of the network, gossip becomes known to all! In the the distribution of edges in the jargon of social scientists, the set of immediate neighborhood of a node people represented by that graph becomes independent of N for large N. forms a society. The class of graphs However, keeping λ = pN a constant, that Erdo″s and Rényi introduced and means scaling the link probability p that helped give the answer is called the probability of a node having with N – 1. The average node degree is random graphs, a subject with a huge exactly k incident edges in G(N,p) mathematical literature. For a review, becomes see the book by Bollobás (2001).

The Binomial Random Graph. (1) and the standard deviation of the dis- Because we need it for later discussion, tribution around the average is we introduce the binomial random graph G(N,p) and present some of its Note that, as edges are drawn inci- properties. G(N,p) is a class of graphs dent to a node, that node will influ- with N vertices, whose edges are ence the number of edges around the This result shows that the binomial drawn at random and independently, other nodes, and thus, in principle, the graph has a characteristic scale according to a uniform distribution distribution of Xk will not be exactly defined by λ. with probability p. Therefore, the the same as if all the nodes were inde- average number of links incident on a pendent, and the calculation of the node is λ = p(N – 1), or for large exact form of the degree distribution Real-World Networks graphs, it is approximately λ = pN. becomes a hard task. It was Bollobás Thus, according to the answer for the (2001) who showed that, for large The latest revolution in networks Gossip Problem, when λ = 1, or the enough N, the nodes can be treated as science happened toward the end of probability for a node to have an edge if they were independent, and thus, the 1990s, when powerful computers is p = pc = 1/N, a giant cluster, or with good approximation, the degree made it possible to gather and analyze giant component, emerges that con- distribution of G(N,p) is described by data for systems containing a large

Number 29 2005 Los Alamos Science 99

Complex Networks

phone call networks. Why do these real-world networks have similar degree distributions? Is there a universal mechanism that generates these structures? The first crucial observation is that, in most cases, these structures result from dynamic processes with a strong stochastic component, just like the random graph model of Erdo″s and Rényi. However, to deviate from the random graph model, the network evolution process must include stochastic dependency and bias. The question is then, “What stochastic processes will generate scale- free networks?” Figure 4. The Degree Distribution of the Binomial Random Graph Most current models generating The red circles show the degree distribution, or Xk/N, the fraction of nodes with k scale-free networks identify a mecha- links vs k, for a single instance of the binomial random graph G(N,p) with the num- nism for network growth and evolu- 4 ber of nodes N = 2 × 10 and the probability for a link to exist between two nodes tion. Among the notable ones are the being p = 10–3. The continuous black line is a plot of the Poisson distribution P(k)in preferential attachment model of formula (2) with λ = pN = 20. Note the similarity between the two distributions. Albert and Barabási (2002), in which a newly arriving node connects to a number of components: from the in these systems. The most useful node in the existing network with World Wide Web and the Internet, properties for this purpose are degree probability proportional to the current phone call networks, networks of distributions, clustering, assortativity, degree of the node in the network; the movie actors, large-company boards of and shortest paths. fitness-based network growth model directors, scientific collaboration net- Instead of listing the classes of these of Caldarelli et al. (2002); the Chung- works, language networks, crime networks and enumerating their prop- Lu model of power-law random webs, epidemic networks, and the sex erties, we will discuss one ubiquitous graphs (2002); the model of the World web to biological networks such as the class, the so-called scale-free Wide Web by Menczer (2002); the metabolic network, protein interaction networks, originally introduced by initial attractiveness model of networks, cell-signaling, and food Albert-László Barabási. These net- Dorogovtsev et al. (2000); and others. webs. The first important observation works have power-law degree distri- Although these models produce is that most of these networks are very butions (see Figure 5), as opposed to graphs that have power-law degree different from the random graphs of the Gaussian or Poisson degree distri- distributions, they either have been Erdo″s and Rényi. In hindsight, this butions of random graphs (for exam- built for a specific type of network departure is not unexpected: In the ple, Figure 4). These real-world, (for example, the model by Menczer) random graphs of Erdo″s and Rényi, scale-free networks include the net- or are mathematical abstractions in the edges are assumed to exist com- work of movie actors, scientific col- which the stochastic network-growth pletely independently from each other, laboration networks, the sex web, the process has little to do with the actual, whereas in real-world networks, the metabolic network in the cell (on all often quite complicated, evolution existence of edges is typically condi- three levels of life—archaea, bacteria, mechanism of the real-world network. tioned by nonindependent processes, and eukaryotes), the protein interac- The stochastic dynamics for the or constraints, such as spatial embed- tions network, the language network appearance and disappearance of ding and interaction range depend- defined by synonyms (in which case, Internet routers, which has many ency. The real surprise is that, in spite the nodes are the words, and the edges unknown factors, is another example. of their diversity, real-world networks connect the synonyms), and virtually Most real-world networks are also can be classified into a small number all large-scale information networks: strongly coupled to other networks or of different classes of graphs, each the Internet (router and also other large-scale complex systems, characterized by certain structural autonomous domain level), the World and thus, in order to identify the net- properties of the interaction topology Wide Web, some e-mail networks and work evolution mechanism, one can-

100 Los Alamos Science Number 29 2005

Complex Networks

Figure 5. Degree Distribution of Various Scale-Free Networks (a) Shown here is the cumulative degree distribution for citation networks (after Redner 1998); (b) the sex web (after Liljeros et al. 2001); (c) the Internet at the router level (after Faloutsos et al. 1999); (d) the in-link and (e) out-link degree distributions for the World Wide Web (after Albert et al. 1999); and (f) the metabolic networks for three species (after Jeong et al. 2000). [Plot (a) is courtesy of the European Physical Journal B. Plot (c) is courtesy of Computer Communication Review, ACM Publications 1999. Plots (b), (d), (e), (f), (g), and (h) are courtesy of Nature.] not study these networks in isolation. structures and the coupling of the tions, a topic heavily studied at Los To add to the complexity of the prob- flow to the structural evolution. Alamos in the past decade. The usual, lem, the evolution of the network classic approach to epidemics imposes structure can depend on the dynamics The Problem of Epidemics. a number of assumptions that make or flow on the network. Most studies Before presenting some recent results analytic and numerical treatment rela- of complex networks have been static that take into account the coupling tively straightforward; however, at and structural as they try to identify between structure and dynamics, I least in some cases, that approach their graph-theoretic properties. It has will briefly mention an interesting and may cause a departure from reality. become clear, however, that, to solve important real-world problem that is One such assumption is uniform mix- even this problem, we must look at complex in the sense mentioned ing, whereby the individuals of a pop- the full dynamics of the complex net- above. I am referring to epidemics, or ulation are assumed to come in con- work, that is, at the flow on these disease propagation in living popula- tact with equal probability, independ-

Number 29 2005 Los Alamos Science 101

Complex Networks

(to be specified below), there is no universal evolutionary mechanism. Instead, the network structure evolves according to a selection principle that promotes the global efficiency of transport and flow processing through these structures (Toroczkai and Bassler 2004). In other words, regard- less of the specific evolutionary mechanism, that mechanism works within the constraints of the selection principle. And the operation of the selection principle on evolution often results in scale-free networks. Most real-world networks (except those that are defined by artificial associations) serve as transport sub- Figure 6. The Roadways of Portland, Oregon strates for various entities such as The roadway network of Portland forms the substrate for a coupled complex dynam- information, energy, material, and ic network to simulate movements and disease transmission in this highly populat- forces. Some networks have evolved ed urban area. (This image is courtesy of the TRANSIMS Project at Los Alamos.) spontaneously (without global design), and it makes sense to enquire ent of their locations. In order to relax son and a location if that person vis- whether their dynamics obey a selec- this assumption, we observe that con- ited that location during the day. The tion principle toward some kind of tact processes, such as disease trans- edge has a weight associated with it, optimal or efficient behavior. Such a mission, are well localized in space called “timestamp,” which is the principle would be analogous to the and require that the two or more indi- union of time intervals during which one of natural selection that shapes viduals be no farther apart than some the person was at that location. What both the biological networks at the typical distance characteristic of the can we learn, analyzing such a net- cellular level and the food web. disease transmission process. In heav- work, pertinent to disease outbreaks Looking at the Internet, we note ily populated urban areas, disease is in a city? How can knowledge about that, if a router receives too much usually transmitted within such loca- this network be exploited to design traffic and causes constant congestion tions as buildings and mass transit effective vaccination and quarantine of the packets, engineers will fix the areas (waiting areas and mass transit strategies? Conclusions for the spe- problem locally by bringing up more cars). Using census data and mobility cific case of Portland, Oregon, with routers or modifying the routing algo- diaries that specify the times of approximately 1.6 million people and rithm. Similarly, in social networks, if entrance and exit to and from a loca- 181,000 locations can be found in an acquaintance does not satisfy our tion for all locations that a specific Eubank et al. (2004). expectations about some set of social person visited during the day, one norms, that link will “naturally” be obtains a graph that has the desired dropped from our own social network. detailed resolution for contact patterns Scale-Free Networks: To explore this trend toward effi- between people moving around in an Coincidence or Universality? ciency more formally, we first need to urban area. This movement is largely define a flow process on the network. constrained by the roadway network This section presents a different Among the most ubiquitous flow and the traffic on it. Since the road- approach to understanding the emer- processes in Nature are those generat- way network is itself a complex net- gence of the scale-free property for ed by local variations, or gradients, of work, the disease transmission prob- real-world networks. As mentioned scalar quantities. Particle concentra- lem is that of coupled complex previously, so far no one has found an tion, temperature, electric or gravita- dynamic networks (see Figure 6). obvious universal mechanism leading tional potential, and pressure are just a In this network, there are two to power-law degree distributions for few examples. The gradient-induced types of nodes: people and locations, real-world networks. We actually sug- flow processes include granular flow, and an edge is drawn between a per- gest that, for a large class of networks fluid flow, electric current, diffusion

102 Los Alamos Science Number 29 2005

Complex Networks

processes, heat flow, and so on. will be time dependent. If we further- Naturally, the same local-gradient more assume that all links have the mechanism will generate flows in same conductance, or transport prop- complex networks. Two less obvious erties, the gradient network will local-gradient examples are diffusive describe the instantaneous substruc- load balancing schemes used in dis- ture carrying the maximum flow. tributed computation (Rabani et al. Consequently, we can hope to use 1998), which are also employed in gradient networks as a tool to analyze packet routing on the Internet, and the the flow efficiency or susceptibility to reinforcement learning mechanism in jamming on the corresponding sub- social networks with competitive strate networks. dynamics (Anghel et al. 2004). In the Note that, if there are two or more first example, a computer (or a router) nodes in the network neighborhood will ask its neighbors on the network of a node i that share the maximum for their current job load (or packet value, the gradient in i is called load), and the router will balance its degenerate. If each neighborhood load with the neighbor that has the has only one maximum, it is called minimum number of jobs to run (or nondegenerate, and is easier to ana- Figure 7. Definition of a Gradient packets to route). In this case, the Edge lyze. In the discussions below, we scalar is the negative of the number of The gradient edge is a directed link will restrict ourselves to the nonde- jobs, or packets, at nodes, and the from node i to that neighbor on the sub- generate condition, which is easily flow will be along the direction of the strate graph that has the largest value realized if, for example, the scalars gradient of this scalar in the node’s of the scalar in the neighborhood of i.If are continuous random variables. network neighborhood. In the second i has the largest value, then the gradient Since every node has exactly one example, a number of agents/players edge is a self-loop. gradient direction from it (even f it in the social network play an interac- is a self-loop), G has exactly N tive competition game with limited nodes and N edges (and there is at information. At every step of the model of a transport process, we least one self-loop, corresponding to game, each agent has to decide whose assume that there are N nodes and maxi{hi}). A simple but very impor- advice to follow before taking an that the transport takes place on a tant property of nondegenerate gra- action (such as placing a bet), in its fixed substrate network S(V, E), dient networks is that they form circle of acquaintances (network where V is the node set and E is the forests, that is, each gradient net- neighborhood). Typically, an agent edge set that describes the intercon- work is a collection of tree graphs will try to follow the advice of that nections of the nodes. Associated containing no loops (except for self- neighbor who in the past proved to be with each node i, there is a scalar hi loops). We can therefore hope to the most successful in predicting the that describes the “potential” of the analyze network flow processes game. That neighbor is recognized by node. Then a gradient network G can using the techniques of statistical using a scoring mechanism, which is be constructed as the collection of mechanics that have been well the simplest form of reinforcement directed links that point from each developed for treelike structures. learning: Every agent has a success node to the nearest neighbor on the score that changes in time, coupled to substrate network S that has the high- Gradient Networks on Random the game’s evolution. An agent will est potential (see Figure 7). Thus, vs Scale-free Networks. Let us first follow the advice of that agent who only one directed link points away consider a gradient network for a ran- has the highest score in its network from each node in G, and G consists dom graph substrate S. In particular, neighborhood at that moment (Anghel of N directed links. Note that, if the we choose for S the binomial random et al. 2004). In this case, the scalar is potential of a node is higher than the graph, G(N,p) consisting of N nodes, the past success score of the agents, potential of all its nearest neighbor each pair of nodes being linked with and an agent will act based on the nodes, the gradient link of that node probability p. We next assume that the information received along the link is a loop that points back to itself scalar potentials of the various nodes that is in the direction of the gradient (“self-loop”). In general, the potential are independent random variables of this scalar. for each point can evolve in time, and identically distributed according to a To construct a simple and general as a result, the gradient network G distribution η(h). The distribution of

Number 29 2005 Los Alamos Science 103

Complex Networks

the number of links l pointing to each node, the so-called in-degree distribution R(l) of the gradient network G, can be exactly calculated, and it yields the following expression:

(3)

Thus, this in-degree distribution is independent of the particular form of the distribution for the scalar potentials η(h). It is possible to show that in the limit N →∞and p → 0 such that Np = λ = constant >> 1, the expression in Equation (3) becomes the power law R(l) ≈ 1/(λl), with a finite-size cutoff at lc = z; refer to Figure 8(a). Therefore, in this limit, gradient networks are scale-free graphs (up to their cutoff)! This power-law degree distribution for the gradient network is a rather surprising result because, in the same limit, the substrate graph S is a binomial random graph having a Poisson degree distribution with a well-defined average degree λ (setting the scale of the substrate graph), as well as rapidly decaying tails.1 If, instead, the substrate network S is a scale-free graph, the gradient graph will still have a power-law degree distribution. Figure 8(b) com- pares degree distributions P(k) for

Figure 8. Gradient-Graph Degree Distributions for Random and 1 Scale-Free Substrate Networks A similar finding was reported by Lakhina et al. (2003), who repeated on (a) The in-degree distribution is shown for the substrate binomial random graph binomial random graphs the trace-route G(N,p), where N = 1000, and p = 0.1 (z = 100). The numerical values are obtained measurements used to sample the struc- after averaging over 104 sample runs. (b) The in-degree and degree P(k) distribu- ture of the Internet. Lakhina and col- tions are for the substrate Barabási-Albert scale-free graph with parameter m (m = leagues found that the spanning trees 1, 3). In this case N = 105, and the average is performed over 103 samples. obtained in this way have a degree distribution that obeys the 1/k law. Later, Clauset and Moore (2003) have presented an analytical approach to derive the 1/k law. This approach suggests the possibility of mapping between graphs generated by trace-route sampling and gradient networks. Although it is not an exact mapping, a close connection can indeed be made by interpreting trace-route trees as suitably constructed gradient networks.

104 Los Alamos Science Number 29 2005

Complex Networks

scale-free substrate networks, which where 〈〉n means averaging over the constant less than unity for large we generated by the Barabási-Albert disorder in the network structure and networks. In other words, scale-free network with parameter m (minimum 〈〉h means averaging over the ran- networks are not prone to maximal degree, see Albert and Barabási 2002) domness in the scalar field. The value congestion. (This is true for all with the in-degree distributions for the of J is always between 0 and 1, with J power-law networks for which the corresponding gradient networks. One = 1 corresponding to maximal conges- average degree does not grow with immediate conclusion is that the gra- tion and J = 0 corresponding to no N.) Figure 9 shows the congestion dient network is the same type of congestion. Note that J is a congestion factors as a function of network size structure as the substrate. In this case, pressure characteristic generated by for random and scale-free substrate it is a scale-free (power-law) graph gradients rather than an actual networks. Many real-world networks with the same exponent. throughput characteristic. For a bino- evolve more or less spontaneously mial random substrate network (for example, the Internet or the Flow Properties on Random vs G(N,p), we use Equations (3) and (4) World Wide Web), and they can reach Scale-Free Networks. Using the to obtain the corresponding jamming sizes of about 108 nodes. At such properties of gradient networks, we factor: large N, the scaling limit studied can define a transport characteristic above applies, and random networks related to congestion or jamming in have maximal congestion. Thus, such the substrate network. In particular, substrates are very inefficient for we compare the average number of (5) flow processing. Scale-free networks, nodes with in-links with the average on the other hand, have congestion number of nodes with out-links. If that stays bounded away from unity (in) Nl denotes the number of nodes as the number of nodes grows very with l in-links, the total number of In the scaling limit N → ∞ and p = large, and they are therefore much nodes receiving gradient flow will be constant, the jamming factor assumes more efficient substrates for transport the asymptotic behavior and flow processing. Thus, it appears that the scale-free property of many real-world networks is not accidental. Topology may develop quite naturally from a selection rule that tends to The total number of gradient out-links maximize the global efficiency of the is simply Nsend = N because every flow along the network. node has exactly one out-link. Naturally, the ratio Nreceive/Nsend will be related to the instantaneous global Small-World Magic: congestion in the network. The small- That is, the random graph becomes Synchronized Computing er the number of nodes receiving the maximally congested. It is easy to Networks flow (given the same number of show that, in the other limit, when senders), the more congestion is in the z = Np >> 1 is kept constant while We have seen that many real-world substrate network at that instant. If the N →∞, interactions are mediated across com- flow received by a node requires a plex network topologies and that the nonzero processing time (such as structure and dynamics of those com- routing of a packet by the router), a plex networks are becoming better small ratio of Nreceive/Nsend translates understood. It is therefore natural to into large delay times and thus ineffi- wonder whether network concepts can cient flow processing. Let us define Once again, the random graph asymp- be put to practical use. For example, the congestion (or jamming) factor as tomatically becomes maximally con- can those concepts help us design follows: gested, or jammed. systems that exhibit certain desired For scale-free networks, however, properties? In this last section of the the conclusion about jamming is article, I will show how complex net- entirely different. We find that the work concepts were used to solve a (4) jamming coefficient J becomes inde- problem in distributed, or parallel, pendent of N, and it is always a computation.

Number 29 2005 Los Alamos Science 105

Complex Networks

tens of thousands: the Nippon Electric Company’s 5120-node Earth simula- tor producing 35.86 teraflops, the 8192-node Q-machine at Los Alamos with 13.88 teraflops, Virginia Tech’s X machine, which is a 2200-node apple G5 cluster with 10.28 teraflops, and so on. IBM is currently building the Blue Gene/L parallel computer with 360 teraflops and 65,000 nodes. Blue Gene/P, the next-generation computer, is expected to surpass the petaflop barrier in 2006. The design of efficient, scalable update schemes for performing PDES on these large parallel computers is a rather challenging problem because the simulation scheme itself becomes a complex system whose properties are hard to deduce using classical methods of algorithm analysis. Korniss et al. (2003) introduced a less conventional approach to analyzing Figure 9. Congestion Factors for Random and Scale-Free Substrate the efficiency and scalability of paral- Networks lel discrete-event simulation schemes. Congestion factors are shown as a function of size for random graphs and scale- The authors constructed an exact free networks. For random binomial graph substrates, the jamming coefficient tends mapping between the parallel compu- to unity with increasing network size, indicating that these networks will become tational process itself and a nonequi- extremely congested in this limit. For scale-free substrates, however, the congestion factor becomes independent on the network size, and thus arbitrarily large networks librium surface growth model. As a can be considered without increasing their congestion level. result, questions about efficiency and scalability can be mapped into certain topological properties of this nonequi- We consider the class of systems nous. Simulating the systems is librium surface. Then, using methods made of a large number of interacting nontrivial, and in most cases, the from statistical mechanics, we can elements or individuals, each having a complexity of the problem requires solve the scalability problem of the finite number of attributes, or local the use of distributed computer archi- computation PDES schemes. We now state variables, that can assume a tectures. These problems define the briefly sketch the scalability problem countable number (typically finite) of field of parallel discrete-event simu- and its solution. values. The dynamics of the local lations (PDES). In order to simulate the dynamics state variables are discrete events Conceptually, the computational of the underlying system, the PDES occurring in continuous time, and the task is divided among N processing scheme must track the physical-time interactions between individuals, or elements (PEs), each of which evolves variable of the complex system. In elements, have a finite range. There the dynamics of the allocated piece of case of asynchronous dynamics on are many examples of such systems: the system. Because of the interac- distributed architectures, each PE gen- magnetic systems, epidemics, some tions among the individual elements erates its own physical (also called financial markets, wireless communi- of the real system (spins, atoms, pack- virtual) time τ, which is the physical cations, queuing systems, and so on. ets, calls, and so on), the PEs must time variable of the particular compu- Virtually all agent-based systems can coordinate with a subset of other PEs tational domain handled by that PE. be considered to belong to this class during the simulation. Because of the varying complexity of of discrete-event complex systems. At present, large parallel comput- the computation at different PEs, at a Often, the dynamics of such systems ers for performing PDES have thou- given wall-clock instant, the simulat- is inherently stochastic and asynchro- sands of nodes and soon will have ed, or virtual, times of the PEs can

106 Los Alamos Science Number 29 2005

Complex Networks

Figure 10. A Fully Scalable Small-World PDES Scheme The small-world PDES scheme is fully scalable. By introducing more shortcuts into the communication network (increasing p), the algorithm becomes measurement scalable (a), and it stays computationally scalable (b).

differ, a phenomenon called “time lated time to the new value τi(t + 1), after very long times, as the simulated horizon roughening.” Let us denote and the process is repeated for all system size and, therefore, the number the virtual time at PEi measured at active sites, generating the dynamics of PEs are taken to infinity. N wall-clock time t by τi(t). The set of of the virtual time horizon {τi(t)} i=1. We solved this computational scal- N virtual times {τi(t)}i=1 forms the vir- The average of the time horizon after ability problem by drawing an analogy tual time horizon of the PDES scheme t parallel steps is obviously with the statistical mechanics of non- after t parallel updates. In conserva- equilibrium surface-growth processes. tive PDES schemes, a PE will per- Thin films are grown on solid sub- form its next update only if it can strates by deposition of atoms or mol- obtain the correct information from its ecules from surrounding vapors. neighbors to evolve the local configu- Because the vapors are fairly hot, the ration (local state) of the underlying Thus, the rate of progress of the time atoms reaching the solid surface fol- physical system it simulates without horizon average, or the average_ uti- low a stochastic path until they are violating causality. Otherwise, it idles. lization_ of the PEs 〈u(t)〉 = 〈τ (t + 1)〉 incorporated into the surface, typically Specifically, the PEi can only update – 〈τ (t)〉 is proportional to the number in an irregular fashion. The resulting (become “active”) at wall-clock of nonidling, or active, PEs. The aver- thin film has mounds and valleys that instant t if age 〈·〉 is taken over the stochastic can be described by the fluctuations of event dynamics. The PDES scheme is the local height variable h(x,t) of the (6) computationally scalable if there is a film measured from the surface of the constant c > 0, such that substrate. Using an approach based on the Langevin equation, physicists have That is, the PE’s virtual time is a local (7) developed extended theoretical minimum among the virtual times of machinery to describe the statistics of its neighboring PEs (specified as the the fluctuations of the variable h(x,t). set ). Once the PE at site i can That is, the average rate of progress of The simulated time variable τi(t) in update, it will advance its local simu- the time horizon does not vanish even the computational scalability problem

Number 29 2005 Los Alamos Science 107

Complex Networks

behaves much like the surface height elements N in the long time limit causality is preserved in just the same variable h(x,t) in that τi(t) evolves (t →∞). Therefore, if we try to meas- way. These checks only synchronize according to the stochastic update ure a global property of simulated the PEs. Using the same coarse-grain- dynamics of the PDES scheme with system at a given simulated time τ ing methods as for the basic PDES the index i of the PE corresponding to and wait until all processors have sim- scheme, we now find that the time x in the height variable. In many large ulated their local state corresponding horizon fluctuations are described by complex systems, the dynamics of the to that time, the waiting period in stochastic events can be characterized wall-clock time would diverge with by a Poisson distributed stream. This the number of processing elements! means that, when simulating such sys- In other words, even though a parallel tems, the updates at individual PEs computer with infinitely many pro- correspond to adding height incre- cessing elements can simulate the (9) ments that follow a Poisson distribu- dynamics of an infinitely large system tion. Using statistical mechanics at nonzero speed (computational scal- methods to analyze the resulting ability), the basic PDES scheme could with γ(0) = 0 and γ(p) > 0 for surface-growth model, one can show not produce a single measurement of 0 < p ≤ 1. This equation differs from that the fluctuations of the virtual the global state of the system! The Equation (8) in the strong damping time horizon in the continuum limit basic conservative scheme is compu- term, –γ(p)τ^, which is ultimately can be described by the so-called tationally scalable but measurement responsible for the nondivergence of Kardar-Parisi-Zhang (KPZ) equation nonscalable. the average spread, and thus the new of surface growth: How can we surmount this prob- update scheme is measurement scala- lem? Can the PDES scheme be modi- ble as shown in Figure 10. fied such that the new update scheme (8) is also measurement scalable? The answer is affirmative, and the key to Concluding Remarks the solution is the notion of the small- where τ^ is a coarse-grained form of world property of complex networks The list of problems, challenges, and the virtual-time variable and η is a (Korniss et al 2003). applications that I presented above is white noise term.2 We then use the In order to decorrelate the fluctua- rather biased toward my particular KPZ equation to verify that the uti- tions in the time horizon, we modify research interests, and it is not, by far, lization of the PEs satisfies Equation the update topology in the following exhaustive of this area. My goal was to (7). The existence of a constant c > 0, way: for every node i, we assign a give the reader a feeling for the type of as in (7), is the result of the slope- randomly chosen communication link, complexity one encounters when deal- slope correlations of the surface being r(i). According to its definition, the ing with networks. I also wanted to short ranged and not scaling with N. resulting communication topology (a show that this is a novel area with many Our numerical evaluation of this con- regular lattice plus random links) interesting and potentially powerful –6 stant yields c = 0.2461 ± (7 × 10 ), forms a small-world applications awaiting discovery. n which shows that the basic conserva- network. When a node is allowed to tive PDES scheme is indeed computa- update—its virtual time satisfies the Further Reading tionally scalable. condition in (6)—it will make, with There is, however, a fundamental probability p, an extra check for the Albert, R., and A.-L. Barabási. 2002. Statistical problem with the basic PDES update condition τi(t) ≤ τr(i)(t) and update if Mechanics of Complex Networks. Rev. scheme. The KPZ equation for the that condition is satisfied. With proba- Mod. Phys. 74: 47. time horizon fluctuations predicts that bility 1 – p, it will make this extra Albert, R., H. Jeong, and A.-L. Barabási. 1999. the average spread of those fluctua- check and thus behave as the basic Diameter of the World-Wide Web. Nature 2 tions, w (N,t), diverges with an PDES scheme. Here p has the role of 401: 130. increasing number of processing a tuning parameter: For p = 0, we have the basic PDES scheme, whereas Anghel, M., Z. Toroczkai, K. E. Bassler, and G. p = 1 corresponds to the fully scalable Korniss. 2004. Competition-Driven Network Dynamics: Emergence of a Scale- small-world PDES scheme. Note that 2 The deterministic part of this equation is easi- Free Leadership Structure and Collective ly related to the well-known Burgers equation these extra checks do not affect the Efficiency. Phys. Rev. Lett. 92: 058701. through the slope variables φ^ = ∂τ^/∂x. correctness of the simulation, and

108 Los Alamos Science Number 29 2005

Complex Networks

Bollobás, B. 1998. Modern Graph Theory: Lakhina, A., J. W. Byers, M. Crovella, and P. Graduate Texts in Mathematics. Vol. 184. Xie. 2003. Sampling Biases in IP Topology Edited by F. W. Gehring, and S. Axler. New Measurements. In Proceedings of the York: Springer-Verlag. Annual Joint Conference of the IEEE Computer and Communications Societies ———. 2001. Random Graphs. Second (INFOCOM). p. 332. New York: IEEE. Edition. Cambridge; New York: Cambridge University Press. Liljeros, F., C. R. Edling, L. A. N. Amaral, H. E. Stanley, and Y. Åberg. 2001. The Web of Caldarelli, G., A. Capocci, P. De Los Rios, and Human Sexual Contacts. Nature 411: 907. M. A. Muñoz. 2002. Scale-Free Networks from Varying Vertex Intrinsic Fitness. Phys. Lloyd, S. 2000. Ultimate Physical Limits to Rev. Lett. 89: 258702. Computation. Nature 406: 1047.

Chung, F., and L. Lu. 2002. Connected Menczer, F. 2002. Growing and Navigating the Components in Random Graphs with Given Small World Web by Local Content. Proc. Expected Degree Sequences. Ann. Comb. 6: Natl. Acad. Sci. U.S.A. 99 (22): 14014. 125. Redner, S. 1998. How Popular Is Your Paper? Clauset, A. and C. Moore. 2004. Traceroute An Empirical Study of the Citation Sampling Makes Random Graphs Appear Distribution. Eur. Phys. J. B, 4 (2): 132. to Have Power Law Degree Distributions. [Online]: Solomonoff, R., and A. Rapoport. 1951. http://arxiv.org/abscondmat/0312674 Connectivity of Random Nets. Bull. Math. Biophys. 13: 107. Dorogovtsev, S. N., J. F. F. Mendes, and A. N. Samukhin. 2000. Structure of Growing Toroczkai, Z., and K. E. Bassler. 2004. Networks with Preferential Linking. Phys. Jamming is Limited in Scale-Free Systems. Rev. Lett. 85 (21): 4633. Nature 428: 716.

Eubank, S., H. Guclu, V. S. A. Kumar, M. V. Maratho, A. Srinivasan, Z. Toroczkai, and N. Wang. 2004. Modelling Disease For further information, contact Outbreaks in Realistic Urban Social Zoltán Toroczkai (505) 667-3218 Networks. Nature 429: 180. ([email protected]).

Faloutsos, M., P. Faloutsos, and C. Faloutsos. 1999. On Power-Law Relationships of the Internet Topology. Comput. Commun. Rev. 29 (4): 251.

Graham, R. L., B. L. Rothschild, and J. H. Spencer. 1990. Ramsey Theory. Second Edition. New York: John Wiley and Sons, Inc.

Jeong, H., B. Tombor, R. Albert, Z. N. Oltvai, and A.-L. Barabási. 2000. The Large-Scale Organization of Metabolic Networks. Nature 407: 651.

Kilian, J., Y. Rabani, A. Sinclair, and R. Wanka. 1998. Local Divergence of Markov Chains and the Analysis of Iterative Load Balancing Schemes. In Proceedings of the 39th Annual Symposium on the Foundations of Computer Science (FOCS). p. 694. Los Alamitos, CA: IEEE Computer Society.

Korniss, G., M. A. Novotny, H. Guclu, Z. Toroczkai, and P. A. Rikvold. 2003. Suppressing Roughness of Virtual Times in Parallel Discrete-Event Simulations. Science 299 (5607): 677.

Number 29 2005 Los Alamos Science 109