The Information-Theoretic Structure of Classical Spin Systems

Home , Spin model

Santa Fe Institute Working Paper 15-10-042 arXiv:1510.08954 [cond-mat.stat-mech] Anatomy of a Spin: The Information-Theoretic Structure of Classical Spin Systems

Vikram S. Vijayaraghavan,∗ Ryan G. James,† and James P. Crutchfield‡ Complexity Sciences Center and Physics Department, University of California at Davis, One Shields Avenue, Davis, CA 95616 (Dated: August 16, 2016) Collective organization in matter plays a significant role in its expressed physical properties. Typically, it is detected via an order parameter, appropriately defined for each given system’s observed emergent patterns. Recent developments in information theory, however, suggest quantifying collective organization in a system- and phenomenon-agnostic way: decompose the system’s thermodynamic entropy density into a localized entropy, that solely contained in the dynamics at a single location, and a bound entropy, that stored in space as domains, clusters, excitations, or other emergent structures. We compute this decomposition and related quantities explicitly for the nearest-neighbor Ising model on the 1D chain, the Bethe lattice with coordination number k = 3, and the 2D square lattice, illustrating its generality and the functional insights it gives near and away from phase transitions. In particular, we consider the roles that different spin motifs play (in cluster bulk, cluster edges, and the like) and how these affect the dependencies between spins.

PACS numbers: 05.20.-y 89.75.Kd 05.45.-a 02.50.-r 89.70.+c Keywords: Ising spin model, thermodynamic entropy density, dual total correlation, entropy rate, elusive information, enigmatic information, predictable information rate, complex system

I. INTRODUCTION statistical mechanics. Both Shaw [9] and Arnold [10] studied information in 1D spin-strips within the 2D Ising Collective behavior underlies a vast array of fascinating model. Feldman and Crutchfield [11–13] explored several phenomena, many of critical importance to contemporary generalizations of the excess entropy, a well known mutual science and technology. Unusual material properties— information measure for time series, in 1D and 2D Ising such as superconductivity, metal-insulator transitions, models as a function of coupling strength, showing that and heavy fermions—have been attributed to collective they were sensitive to spin patterns and can be used behavior arising from the compounded interaction of sys- as a generalized order parameter. In one of the more tem components [1]. Collective behavior is by no means thorough-going studies to date, Lau and Grassberger [14] limited to complex materials, however: the behavior of computed the excess entropy using adjacent rings of spins financial markets [2], epileptic seizures [3] and conscious- to probe criticality in the 2D cylindrical Ising model. ness [4], and animal flocking behavior [5] are all now also Barnett et al. [6] tracked information flows in a kinetic seen as examples. And, we now appreciate that it appears Ising system using both the pairwise and global mutual most prominently via phase transitions [6, and references informations (generalized as the total correlation [15], it therein]. appears below) and the transfer entropy [16]. Abdallah and Plumbley [17] employed an extensive version of the Operationally, collective behavior is detected and quan- dual total correlation [15] on small, random spin glasses. tified using correlations or, more recently, by mutual Despite their successful application, the measures used information [7], estimated either locally via pairwise com- to monitor collective phenomena in these studies were ponent interactions or globally. Exploring spin systems motivated information theoretically rather than physically, familiar from statistical mechanics, here we argue that and so their thermodynamic relevance and structural these diagnostics can be substantially refined to become interpretation have remained unclear. more incisive tools for quantifying collective behavior. This is part of the larger endeavor of discovering ways To address this we decompose the thermodynamic entropy density for spin systems into a set of information mea-

arXiv:1510.08954v2 [cond-mat.stat-mech] 13 Aug 2016 to automatically detect collective behavior and the emergence of organization [8]. sures, some already familiar in information theory [18], Along these lines, much eﬀort has been invested to explore complex systems [19], and elsewhere [17]. For example, information-theoretic properties of the Ising model of one measure that falls out naturally—the bound entropy— is that part of the thermodynamic entropy accounting for entropy shared between spins. This is in contrast to mon-

∗ [email protected] itoring collective behavior as the difference between the † [email protected] thermodynamic entropy and the entropy of a hypothetical ‡ [email protected] distribution over uncorrelated spins—the total correla- 2 tion mentioned above. In this way, the decomposition are uncorrelated. To determine it, we use the isolated provides a physical basis for informational measures of col- spin entropy: lective behavior, in particular showing how the measures lead to interpretations of emergent structure in system H[σ0] = p ( ) log2 p ( ) p ( ) log2 p ( ) , − ↑ ↑ − ↓ ↓ configurations. where p ( ) = (1 + m)/2 is the probability of a spin being To make this argument and illustrate its consequences, ↑ up, p ( ) = (1 m)/2 is the probability of a spin being SectionII first defines notation and lays the groundwork ↓ − down, and m = (# # )/ is average magnetization for our decomposition. SectionIII then derives the decom- ↑ − ↓ |L| in a configuration. The site index 0 was chosen arbitrarily position. Though this decomposition is given for regular and represents any single spin in the lattice. The system’s spin lattices, in principle it is meaningful for any system Boltzmann entropy is then the extensive quantity HB = with a well defined thermodynamic entropy. SectionIV H[σ0]. outlines the computational methods for estimating the |L| · resulting quantities, using them to interpret emergent As Jaynes [20] emphasizes, though correct for an ideal organization in the nearest-neighbor ferromagnetic Ising gas, HB is not the empirically correct system entropy if model in one dimension (SectionIVA), in the Bethe lat- a system develops internal correlations. In our case, if tice with coordination number k = 3 (SectionIVB), and spins are correlated across the lattice, one must consider Gibbs entropy in the two-dimensional square Ising lattice (SectionIVC). entire configurations, leading to the : Having established the different types of entropy in spin X H[σ] = p (σ) log p (σ) (3) systems and connections to their underlying physical struc- − 2 σ∈σ tures, we conclude by suggesting applications and future directions. and the thermodynamic entropy density h, the entropy per spin when the entire lattice is considered:

H[σ] II. SPIN ENTROPIES h = . (4) |L|

We write σ to denote a configuration of spins on a lattice Note that we dropped the factor kB and used the base-2 , σ for the particular spin state at lattice site i , logarithm. Our use of the letter h as opposed to s is meant L i ∈ L and σ\i the collection of all spin states in a configuration to reflect this multiplicative constant difference. It is easy excluding σ at site i. The random variable over all to see that H[σ] HB. Thus, the Boltzmann entropy i ≤ possible spin configurations is denoted σ, for a particular is typically an overestimate of the true thermodynamic spin variable σi, and for all spin variables but the one at entropy. site i, σ . As a shorthand, we write σ σ and similar to \i ∈ More to the point, in the thermodynamic limit the Boltz- mean indexing into the event space of the random variable mann entropy HB and Gibbs entropy H[σ] differ sub- σ. stantially. As Jaynes shows [20], the difference directly We study the ferromagnetic spin-1/2 Ising model with measures the effect of internal correlations on total energy nearest-neighbor interactions in the thermodynamic limit and other thermodynamic state variables. Moreover, the whose Hamiltonian is given by: difference does not vanish, rather it increases proportion- ally to system size . X |L| (σ) = J σ σ , (1) H − i j Before leaving the differences between entropy defini- hi,ji tions, it is important to note that Boltzmann’s more familiar definition—S = k ln W —via the number W where i, j denotes all pairs (i, j) such that the sites i and B h i of microstates associated with a given thermodynamic j are directly connected in and the interaction strength L macrostate is consistent with Gibbs’ definition [20]. This J is positive. We assume the system is in equilibrium follows from Shannon’s deep insight on the asymptotic and isolated. And so, the probability of configuration σ equipartition property that 2h|L| measures the volume of occurring is given by the Boltzmann distribution: typical microstates: the set of almost-equiprobable con- 1 figurations that are simultaneously most numerous and p (σ) = e−H(σ)/kBT , (2) Z capture the bulk of the probability—those that are typically realized. (See Refs. [21, Sec. 21], [7, Ch. 3] and where Z is the partition function. [22, 23].) Thus, Boltzmann’s W should be interpreted as The Boltzmann entropy of a statistical mechanical system the size (phase space volume) of the typical set associated assumes the constituent degrees of freedom (here spins) with a particular thermodynamic macrostate. Given this 3 agreement, we focus Gibbs’ approach. Though, as we measures: now note, the contrast between Gibbs’ entropy H[σ] and |L| " # Boltzmann’s HB leads directly to our larger goals. X X R[σ] = p (σ) log p σi σ\ − 2 i i=1 σ∈σ |L| X = H σi σ\i , (7) i=1

and III. DECOMPOSING A SPIN’S THERMODYNAMIC ENTROPY X p (σ) B[σ] = p (σ) log (8) − 2 |L| σ∈σ Y p σi σ\i Since we are particularly interested here in monitoring i=0 |L| the appearance of internal correlations, the difference be- X = H [σ] H σi σ\ . tween the Boltzmann and Gibbs entropies suggests itself − i i=0 as our first measure of a system’s internal organization. If each spin were independent, H[σ0] = h. For example, Note that both R[σ] and B[σ] are nonnegative and bound this is true at T = for the Ising model and for site σ ∞ from above by H[ ]. Spatial densities are denoted in percolation models [24]. If there are correlations between lower case: h = H[σ]/|L|, r = R[σ]/|L|, and b = B[σ]/|L|. We spins, however, then h < H[σ0], as just noted for the ex- consider their thermodynamic limit, where . tensive quantities. Their difference is the total correlation |L| → ∞ Similarly, we can begin with the total correlation of spins density [15]: T[σ] and break it in to two terms: ρ = H [σ ] h (5) 0 − X p (σ) T[σ] = p (σ) log (9) T[σ] 2 |L| = , σ∈σ Y |L| p (σi) i=1 T[σ] here is the Kullback-Leibler divergence [7] between = B [σ] + Q [σ] , (10) the distribution over entire configurations and the product of isolated-spin marginal distributions. Now called where B[σ] is as above. Though perhaps not clear here, total correlation the [15], this measure of internal corre- B[σ] is naturally a component of T[σ]; see Ref. [15, lation was wholly anticipated by Gibbs, as Jaynes notes Figs. 7c and 7d]. Q[σ] is the enigmatic information [15]: [20, Eq. (4)]. Since ρ vanishes only when each spin is independent, one might consider it a measure of pattern X p (σ)2 Q[σ] = p (σ) log (11) or structure in a spin system. While this is certainly 2 |L| |L| reasonable as an operational definition, our next step σ∈σ Y Y p (σi) p σj σ\j decomposes both h and ρ into more nuanced quantities. i=1 j=1 = T [σ] B[σ] , − Continuing, we recast the Gibbs thermodynamic entropy H[σ] as the sum of two nonnegative terms; starting from and again, we consider the spatial density q = Q[σ]/|L| in the configuration entropy in its standard statistical form the thermodynamic limit. Eq. (3) and manipulating it into two terms: It can be difficult to initially intuit the difference between

 |L|  T[σ] and B[σ]. Both measure dependencies among spins, Y but against different reference ensembles. T[σ] is the dif-  p σi σ\i  X   ference between the Boltzmann and Gibbs entropies (Eq. H[σ] = p (σ) log p (σ) i=1  − 2  |L|  (5)) and, therefore, quantifies the spin distribution’s devia- σ∈σ  Y   p σi σ\i  tion from a hypothetical distribution in which each spin is i=1 independent from the others. Since R[σ] is the amount of = R [σ] + B [σ] , (6) actual independent entropy in the spin distribution, B[σ] quantifies the amount of spin-spin dependency in configu- where R[σ] and B[σ], known as the residual entropy rations not in reference to a hypothetical independent-spin and dual total correlation [15, 17, 18], are the following distribution, but rather to the actually realized amount 4

H[X1 X2 X3] ⊗ ⊗ H[σ0]

T[T[T[XXX123123123]]]] ρ H[H[H[XXX111]]]] h

r b b q B[B[B[XXX123123123]]]] FIG. 2. Decomposition of the isolated-spin (Boltzmann) entropy H[σ ]. First, into the Gibbs thermodynamic entropy H[H[H[XXX22]]]] H[H[H[XXX33]]]] 0 H[H[H[XXX222]]]] H[H[H[XXX333]]]] density h and the total correlation ρ. Second, the thermodynamic entropy into the localized density r and the dual total correlation information density b, and the total correlation density ρ into b and the enigmatic information density q.

FIG. 1. Total correlation T[X] (north-west hashing) and dual total correlation B[X] (north-east hashing) for three spin entropy and how much entropy (h) there actually spins X1, X2, and X3 depicted via a 3-variable Venn diagram [25] in which areas correspond to the magnitudes of is. Whereas, b is the difference between the thermody- information measures. The joint entropy H[X123] is the union namic entropy h and the amount of independent entropy of the three single-spin marginal entropies H[Xi] (full circles). r there is. In addition, b is also part of ρ [15], leav- The outer rectangle represents an entropy H[X1 ⊗ X2 ⊗ X3] ing q which therefore quantifies potential dependencies with all dependencies between spins removed; each marginal (explained shortly) not present in the thermodynamic entropy H[Xi] still matches its original marginal entropy, but does not overlap that of the other spins. This corresponds to entropy h. the distribution whose Gibbs entropy is the Boltzmann en- Let’s explore the consequences of this decomposition. Al- tropy of X123. The diagram directly demonstrates that T[X] though r and b are components of the thermodynamic measures dependence as a distance from the external reference entropy density h—itself a measure of the irreducible per- H[X ⊗ X ⊗ X ]. Whereas, B[X] measures with respect to 1 2 3 spin randomness—individually they have very different an internal reference H[X123]. The unshaded region is residual entropy R[X123]. meanings that give insight into the interplay of spin ther- malization and spin ordering. The first, r, is the average amount of unshared entropy, quantifying randomness re- of independence. This difference is pictorially represented maining in a single spin given the exact configuration of in Fig.1. the rest of the lattice. For this reason we refer to it as the Let us now interpret the physics captured by these en- localized entropy. In the limit of high and low temperature, tropy components. First, H[σ0] quantifies the entropy r’s behavior is intuitive. At low temperature, spins in the per spin ignoring any dependencies (e.g. correlations) lattice are entirely aligned, and so there is no randomness between spins. The thermodynamic entropy density h, in individual sites given the global alignment. At high however, is the entropy per spin including dependencies. temperatures, each spin is independent, and so there is Continuing in this way, from Eq. (7) r is the entropy full randomness in a site, even knowing its surroundings. per spin remaining after these dependencies have been The other quantity, b, is the entropy remaining in a spin factored out, meaning it is the average amount of inde- when the localized entropy is deducted, and so it captures pendent entropy—the entropy per spin when we know the the thermodynamic entropy shared between a spin and state of the other spins in the lattice. With this logic, the rest of the lattice. We refer to it as the bound entropy. b is that portion of the thermodynamic entropy density Its limits are similarly intuitive. At low temperatures coming from dependencies and is, therefore, the average there is no randomness to share, and so b = 0. Similarly, amount of dependent entropy per spin. As we shall see, in at high temperatures all randomness is localized, and so comparison to ρ, b provides a complementary, quantita- no entropy can be shared and bound entropy also vanishes. tively different, and more physically grounded approach At intermediate temperatures, though spins are in flux, a to quantifying dependencies among spins. spin’s context does at least in part influence its alignment This completes, in effect, our decomposition of the and so b > 0. We refer to the temperature at which b is isolated-spin entropy H[σ0], shown schematically in Fig.2. maximized by Tb. To take stock, let’s back up a bit. Recall that ρ is the Finally, we consider the enigmatic information density difference between H[σ0], which ignores the dependen- q. As a component of ρ it also reflects dependencies, but cies between the spins, and the thermodynamic entropy in a way different from b. The bound entropy captures density h. That is, ρ is the difference between the isolated- dependencies that are also part of the thermodynamic 5 entropy, but q captures those that are not. As we shall estimator Hb2 was used to decrease the number of con- see in SectionV, this means it is sensitive to the interior figuration samples while still providing convergence to of clusters. At low temperatures, i.e., temperatures below known analytic results. Overall, when comparing and Tc, we expect q to be small, since H[σ0] is small. At when analytic results are available, simulation results high temperatures, there are no dependencies between match to within standard error bars smaller than plot spins, and so we also expect q to be small. At intermediate line widths. All simulation results were obtained using temperatures, just like b, we expect q to take larger values. the Wolff algorithm [29]. In summary, in the reported The nonzero values of ρ, b, and q are all driven by Ising results for those several quantities that required estimated spin clustering. Consider for the moment randomly per- or partial estimation from simulations, sufficient care was muting the sites in the lattice, resulting in a site perco- taken that the statistical errors fall far below the level lation model with site occupancy probability given by a needed to support the main conclusions. simple function of the magnetization of the original Ising lattice. Then, the spin orientation at any particular site is now statistically independent of its neighbors, and so A. 1D Ising Model ρ = b = q = 0, and therefore H[σ0] = h = r. There are, however, clusters in this permuted lattice—in fact, Simple Ising models have been broadly adapted to un- there may exist a giant cluster. These clusters are driven derstand simple collective phenomena in fields ranging purely by the lattice topology. We therefore conclude from surface physics [30, 31] and biophysics [32, 33] to that ρ, b, and q are each sensitive to differing aspects of economics [34]. The Ising model on a one-dimensional how the pairwise Ising Hamiltonian affects the shape and lattice ( = Z) can be fully analyzed in closed form. Un- L distribution of spin clusters. fortunately, its emergent patterns are rather constrained; To summarize, the disorder generated at each site is for example, it does not exhibit a phase transition [35]. measured by the Gibbs thermodynamic entropy density h. However, the exact solutions provide an important bench- And, one portion (b) participates in spatial organization mark and so it is a necessary candidate with which to and the other portion (r) does not. Thus, b is key to start and provides a base from which to generalize [36–39]. monitoring the emergence of spatial structures during Analytic results were computed using a transfer matrix pattern formation dynamics. approach combined with the aforementioned Markovian property. For comparison, simulations were performed on a lattice of size N = 1024. IV. RESULTS Independent of sporting a phase transition or not, the argument that b maximizes between temperature extremes To explore what the entropy components reveal, we cal- still holds and Fig.3 verifies this. The bound entropy b 1 reaches a maximal value at T 1.0117 J/k and this is culate them for the ferromagnetic spin- /2 Ising model on b ≈ B one- (1D) and two-dimensional (2D) square lattices and not (and cannot be) associated with a phase transition. on the Bethe lattice. These results demonstrate that our At lower temperatures, spin-spin shared entropy b is the intuitions regarding the behaviors of the information mea- dominant contributor to the Gibbs thermodynamic en- sures are correct, but also raise subtle questions regarding tropy density h = r + b. Whereas at higher temperatures, the details of exactly what they capture. the localized entropy r is the dominant contributor. Simi- larly, at low temperatures q is the dominant contributor to As part of this, we report analytical results for the 1D ρ, and b is its dominant contributor at high temperatures. chain and the Bethe lattice and show that they match SectionV below isolates why b peaks where it does and results from simulation. And, we use analytic results to which spin-configuration features each measure is sen- whenever possible for the 2D lattice. For example, we sitive. For now, let’s continue with system informational report analytical results for h, H[σ ], and ρ. Then, only 0 phenomenology. r needs to be estimated from simulation and that result is then combined with analytic quantities to compute b and q. To provide a comparison, estimating b purely from B. Bethe Lattice Ising Model simulation requires accurate h and r. In this, we make use of the global Markovian property of spin lattices [26]. To obtain r we need only condition a spin on all its nearest What role does the underlying lattice topology play in neighbors—spins directly coupled via the Hamiltonian determining the balance between local randomness and of Eq. (1). For h estimated purely from simulation, we spatially shared information that equilibrium configura- use Ref. [27]’s method. In both cases, Ref. [28]’s entropy tions achieve? Spins on the Bethe lattice, in which is L 6

Tb Tc = Tb

1 1 H[σ0] h 3 3 ρ 4 H[σ0] 4 r h b 1 ρ 1 q 2 r 2

1 b 1 Entropy [bits] 4 q Entropy [bits] 4

0 0

0 1 2 3 4 5 0 1 2 3 4 5 J J Temperature [ /kB] Temperature [ /kB]

FIG. 3. Boltzmann and Gibbs thermodynamic entropy decom- FIG. 4. Thermodynamic entropy decompositions for the Ising positions for the 1D, nearest neighbor, ferromagnetic spin-1/2 model on a Bethe lattice. By far and away, and unlike the Ising model: Gibbs entropy density h and the localized entropy 1D spin lattice, the individual-spin disorder r is the dominant density r both monotonically increase with temperature, but contributor to Gibbs entropy density h over the entire temper- the bound entropy density b is maximal near Tb ≈ 1.0117 J/kB. ature range. Of the information generated, relatively little (b) Below this temperature, the dominant contribution to the is stored in spatial patterns. Gibbs thermodynamic entropy switches to the entropy shared between nearby spins. Whereas, in the Bethe lattice deviations from uniformity come in the form of isolated spins differing from their sur- an infinite Cayley tree with coordination number k, is roundings. Also, unlike 1D, the spin-spin shared entropy an ideal candidate since the absence of loops makes it b is the dominant contributor to ρ at both low and high possible to compute quantities analytically. Moreover, temperatures. spin configurations exhibit a phase transition at critical temperature [36]: C. 2D Ising Model k −1 Tc = 2J k ln . (12) B k 2 − Unlike its 1D sibling, the nearest-neighbor ferromagnetic We analytically calculated the entropy decomposition for Ising model in two dimensions ( = Z2) has a phase tran- L √ a Bethe lattice with coordination number k, the details of sition at a finite critical temperature Tc = 2/ln(1+ 2) ≈ which appear in AppendixA. Simulation results, matching 2.2692 J/kB. In the 2D case, although Onsager’s solution the analytic to high precision, were performed on a 1, 000 lets us calculate H[σ0], h, and ρ [36], we do not have node random 3-regular graph [40], which has no boundary an analytic form for r and so the curves for r, b, and and is locally Bethe lattice-like. q in Fig.5 partly rely on estimates from simulation, as Figure4 presents the results for a Bethe lattice with coor- explained above. The simulations were conducted on a 128 128 lattice and quantities averaged over 200, 000 dination number k = 3, though other ks behave similarly. × Interestingly, all information measures have a disconti- configuration updates. Again, the resulting standard error nuity in their first derivatives, and this happens at the bars were smaller than the plotted line widths. phase transition at Tc. Furthermore, the bound entropy Phase transition aside, the behaviors of h, r, and b, seen b is maximized there: T = T 1.8205 J/k . Opposite in Fig.5, are qualitatively similar to those in the 1D b c ≈ B the 1D lattice, at small T the dominant contributor to lattice. However, unlike 1D, at low temperatures r is the Gibbs entropy h is the local randomness r. Thus, not only dominant contributor to h, similar to the Bethe lattice does the change in lattice topology induce a phase tran- below its transition. Also, paralleling the Bethe lattice, sition, but it also inverts the informational components’ b is the dominant contributor to ρ at both low and high contributions to thermodynamic entropy density. This, in temperatures. Like the 1D system, bound entropy is max- turn, indicates a rather different underlying mechanism imized within the disordered phase at T 2.675 J/k , b ≈ B that supports entropy production in the low temperature above Tc. Let us now turn to what configurational struc- regime. In the 1D case entropy is spatially extended and tures lead to the overall behaviors of these measures and configurations of spins are dominated by large clusters. how they contribute to the Gibbs entropy density h. 7

T J J Tc b T = 1 /kB T = 3 /kB

1 H[σ0] h h ρ 3 ρ 4 r b r 1 q 2 b

1 q Entropy [bits] 4

0 FIG. 6. Motif entropy-component analysis of the 1D Ising 0 1 2 3 4 5 model at two temperatures. A segment of a spin conﬁguration showing up spins (white cells) and down spins (black cells) is J Temperature [ /kB] shown at the bottom.

FIG. 5. Thermodynamic entropy decompositions for the 2D, nearest neighbor, ferromagnetic Ising model. Curves are from to be [41]: numerical simulation with sufficient size that standard errors are much smaller than the line widths. As in 1D, the entropy X h = H σi σi = p (σ) log p (σi σi ) density h and the localized entropy density r monotonically ←− − 2 |←− σ∈σ increase with temperature. Here, also, the bound entropy den- X sity b reaches a maximal value at a nonextremal temperature: = p (σ) hi(σ) (13) near Tb ≈ 2.675 J/kB, but this peak value does not occur σ∈σ at the critical temperature Tc ≈ 2.2692 J/kB, where domain sizes become scale-free. where h (σ) = log p (σ σ ) and σ is the set of spins i − 2 i|←−i ←−i whose indices are “lexicographically” less than i. hi(σ) is a spatially local or pointwise measure and quantifies how V. DISCUSSION much the individual spin σi contributes to the global entropy. We can more directly understand how local motifs contribute to the average entropy by plotting the local At first, it is striking that Tb is not generically identical to measures at each site within a given lattice configuration. Tc. To understand why, we need to investigate to which This approach roughly parallels that in, e.g., Ref. [42], spin-configuration motifs the various measures are sensi- but here we employ different information measures and do tive. To accomplish this, we take a local approach. Each not assume directionality, since we average the pointwise of the measures examined above is an average density, values resulting from each of the possible orientations which is important when discussing the lattice as a whole centered at the given spin. and its bulk thermodynamic properties. However, each cluster spin configuration motif contributes differently to this In the following, we define a to be a maximal set edge average. For example, an up spin surrounded by down of contiguous spins oriented identically. The of a spins contributes to the measures differently than an up cluster is that set of spins in a cluster with at least one spin surrounded by other up spins. neighbor not in the cluster. The remaining spins in the cluster are its bulk. An isolated spin is a cluster consisting of a single spin. Figures6 and7 show the results of the motif entropy analysis in 1D and 2D lattices. In all cases, the spatial A. Motif Entropies average of the displayed quantities corresponds (in the thermodynamic limit) with the values reported in Figs.3 To quantify this, we appeal to spatial- and temporal- and5. local forms of the averaged densities considered so far. There are several immediate similarities that allow for This is possible on square lattices = Zd, due to the structural interpretation. For example, motif q is positive L existence of conditional forms of the entropy density; see within the bulk of a cluster and negative on its edges. Ref. [41, Thm. 2.9]. The conditional forms for each of the Highlighting opposite features, motif r is negligible within component quantities are configuration-weighted averages the bulk, but positive along edges, particularly corners over quantities of the form log ( ) [15]. For example, the and isolated spins. The motif bound entropy b, however, 2 • thermodynamic entropy density of Eq. (4) can be shown is more nuanced in its behavior. Considering Figure6, at 8

16 more generally, correlations and information measures hhhh ρρρρ hhhh 12 on network dynamical systems. If local information estimates, such as those used for the motif entropy analyses 8 above, existed, then studies can be undertaken that de- 4 termine, say, which nodes in a lattice or network graph contribute most to collective behavior. Failing such mea- 0 r qqqq sures, though, one must be wary of claims to answer rrr qqq 4 rrrr bbbb qqqq such questions. These concerns are touched on by the

8 informational analyses of frustration in spin glasses by Robinson et al. [44], whose local entropies are temporally 12 averaged, spatially local entropies in a heterogeneous sys-

16 tem. They are unlike our motif entropies, though, which are both spatially and temporally local and, therefore, FIG. 7. Motif entropy-component analysis of the 2D Ising have meaning only in a homogeneous system. More study model at Tb. is required, however. lower temperatures it is sensitive to cluster edges more so B. Actively Storing Information versus than cluster bulk. At higher temperatures, however, this Communicating It relationship flips and it is more sensitive to the bulk. For all temperatures, it is negative for isolated spins—clusters of size 1. Assuming Glauber dynamics for the nearest-neighbor 2D Ising system, Barnett et al. [6] computed a global transfer In contrast, we see that at not-too-high temperatures entropy [16]—the average information communicated be- motif b is largely sensitive to cluster boundaries. This tween the entire lattice and a single spin. They found that leads one to speculate that, as in Ref. [6], b’s maximization it peaks within the disordered phase at a temperature in the disordered phase is due to complex interactions J TT 2.354 /kB. Although there are serious concerns gl ≈ between cluster sizes and their surface area. This may also with how the transfer entropy conflates two-spin (dyadic) shed light on why b does in fact peak at Tc on the Bethe and multispin (polyadic) dependencies [45], those con- lattice. On the square lattice, on the one hand, cluster cerns are not relevant in these Ising systems due to their boundaries have a tendency to smooth due to the presence known, dyadic Hamiltonian. of correlations flowing along short loops in the lattice To compare our measures with theirs, we examine the ac- topology that tend to align spins on a boundary. On the tive storage of randomly generated information in spatial other, the Bethe lattice has no such loops, and so there is patterns, as measured by the ratio of b/h—how much of no pressure to reduce surface area. This suggests that b the localized randomness (h) is converted by a system will generically peak at T for systems where boundaries c into spin-spin (spatially) shared information (b). Figure8 are not constrained by system energetics and that the plots the ratio as a function of temperature for the 2D energetic smoothing of cluster boundaries drives b to peak Ising system: the ratio peaks in the disordered phase and, in the more “textured” disordered phase. surprisingly, it appears to do so at TTgl . This implies that Interestingly, for general lattices such as the Bethe lat- there is a strong connection between b (storing thermal tice, but especially for random graph topologies, local fluctuations as spatial correlation) and the potential for (conditional) forms of the thermodynamic entropy density information communication in a system. Hopefully, the (analogs of Eq. (13)) and other motif measures are un- result suggests that Barnett et al.’s TTgl is not a result known and may simply not exist. While techniques, such specific only to their choice of Glauber dynamics, but as Ref. [43]’s, exist for estimating thermodynamic entropy rather an intrinsic property of the 2D Ising model. It density for a ferromagnetic Ising model on an arbitrary remains to be seen why spatially shared information is graph, the interpretation of a thermodynamic entropy maximized within the disordered phase of the 2D Ising density in such systems is problematic as each spin site system, though. may have differing connectivity. Therefore, while global averages may exist for arbitrary topologies and may even be tractably estimated, their structural meaning is vastly VI. CONCLUSIONS more challenging. And, this is critical to understanding thermodynamic As noted at the beginning, even the earliest debates over and statistical mechanical properties of such systems and, entropy’s statistical foundation turned on contrasting its 9

T Tc Tgl Tb of spin states exceeds four. Another setting of particular 0.19 interest is to study the behavior of b in a frustrated b/h system, such as the antiferromagnetic Ising model on a triangular lattice, paralleling Ref. [44]. Finally, as alluded to above, although beyond the present scope of this work, 0.18 a next step is to consider informational measures for spins on arbitrary graphs, with the goal of providing insight into the roles that diﬀerent nodes play in information

Entropy [bits] processing and storage in complex dynamical networks. 0.17

ACKNOWLEDGMENTS 2.3 2.4 2.5 2.6 2.7

J Temperature [ /kB] We thank Cina Aghamohammadi, Adam Dioguardi, FIG. 8. Storing locally generated, thermal randomness as spa- Raissa D’Souza, Pierre-André Noël, Márton Pósfai, Paul tial correlation: The ratio of b/h as a function of temperature Riechers, Adam Rupe, Rajiv Singh, and Dowman Varn for in the 2D Ising system. It peaks within the disordered phase helpful conversations. We thank the Santa Fe Institute J at a temperature T ≈ 2.354 /kB. Compare to [6, Fig. 2]. Tgl for its hospitality during visits. JPC is an SFI Exter- nal Faculty member. This material is based upon work supported by, or in part by, the U. S. Army Research thermodynamic components—Boltzmann’s isolated-spin Laboratory and the U. S. Army Research Office under entropy versus the Gibbs global entropy. From a modern contracts W911NF-13-1-0390 and W911NF-13-1-0340. perspective their difference, well known to Gibbs, is a generalized mutual information that measures the degree of individual-component independence, now called the total Appendix A: Bethe Lattice Spin-Neighborhood correlation. Keying off this, we showed that the Gibbs Probabilities thermodynamic entropy density naturally decomposes further into two functionally distinct informational com- Consider a Cayley tree with coordination number k ponents: one quantifying independence among constituent consisting of n shells, or layers of spins centered at spins and the other, dependence. The one quantifying the root. The total number of spins in such a tree is dependence, the bound entropy b, captures collective be- N = k[(k 1)n 1]/(k 2). We denote the spins in the havior by expressing how much of the thermodynamic − − − lattice using σ , i [0,N 1]. Let us use the index 0 entropy density is locally shared. We then demonstrated i ∈ − for the central spin and indices [1, k] for its immediate the behavior of the bound entropy and related quantities neighbors. We use σ to denote a configuration—the state for the nearest-neighbor ferromagnetic Ising model on a of all spins present in the Cayley tree. The Bethe lattice variety of lattices. We found that it tends to a maximum is the singular limit of a Cayley tree as n . Following at intermediate temperatures, though not always at the → ∞ Ref. [36], we write the partition function for the Ising magnetic phase transition. Our analyses support our ear- model on the Bethe lattice as: lier hypothesis [46] that q, as the dominant component of the persistent mutual information [47], should generically X X Z = exp(βJ σiσj) , be maximized at critical points. Though not detailed σ∈σ hi,ji here, this observation holds in three dimensions and in P simulations of the Potts model [48] with the number of where exp(βJ hi,ji σiσj) is the Boltzmann factor corre- states s = 3 on lattices with 1, 2, and 3 dimensions. sponding to the system being in a state σ and σ denotes the set of all possible configurations. As shown in Ref. [36], This brief phenomenological study of thermodynamic due to the Cayley tree topology, the above expression can entropy components served to give a physical grounding be written as: for information measures and what they reveal in spin systems on various lattice topologies. The results suggest X k Z = gn(σ0) , many avenues of potential research. One topic to explore σ0∈σ0 is the behavior of b in the Potts model, which switches from exhibiting a second-order phase transition in the where gn(σ0) is the partition function of a branch of magnetization to a first-order transition when the number the Cayley tree with its root at σ0. Starting with this 10 expression we derive the joint probability of the central where the product is essentially over all branches 1 step spin and its neighbors. away from the central spin, and σi is the spin to which it Let g (σ ) g(σ ) in the limit of n shells. Then, is anchored. From this, we obtain the joint probability of n 0 → 0 → ∞ the spins not close to the Cayley tree leaves behave like the central spin σ0 and its k neighbors as: those on the Bethe lattice with the same coordination p σ ,σ , . . . , σ number. In this limit, by explicitly accounting for the ( 0 1 k)   bonds between the central spin and its neighbors, we can X Y k−1 write the partition function as: = exp βJ σ0σi [g(σi)] /Z . i∈[1,k] i∈[1,k]     X X Y k−1 k−1 Z = exp βJ σ0σi [g(σi)]  , Dividing both numerator and denominator by [g(+1)] σ0∈σ0 i∈[1,k] i∈[1,k] we obtain: σ1∈σ1 . . σk∈σk

  k−1 X Y g(σi) exp βJ σ0σi g(+1) i∈[1,k] i∈[1,k] p(σ0, σ1, . . . , σk) = . (A1) " − #k " − #k g( 1)k 1 g( 1)k 1 eβJ + e−βJ − + e−βJ + eβJ − g(+1) g(+1)

This is the joint probability distribution of the central is stable. Below Tc, however, there are there are 3 −1 spin and its neighbors. We evaluate the above expression roots: x0, 1, x0 where x0 < 1. The stable roots are numerically by deﬁning g( 1)/g(+1) = x. From Ref. [36, x and x −1. Setting x = x provides the distribution − 0 0 0 Eq. (4.3.14)], we know that x is the “stable” root of the p(σ0, σ1, . . . , σk) where the symmetry breaking prefers −1 equation: spins aligned up and where setting x = x0 provides the distribution with the symmetry broken such that e−βJ + eβJ xk−1 x = . (A2) downward spins are preferred. eβJ + e−βJ xk−1

Above Tc, Eq. (A2) has only one root: x = 1, which

[1] V. N. Antonov, L. V. Bekenov, and A. N. Yaresko. Elec- disorder of consciousness. BMC Neuroscience, 14(Suppl tronic structure of strongly correlated systems. Adv. Cond. 1):P83, 2013. Matter Physics, 2011(298928):1–207, 2011. [5] I. Couzin. Collective minds. Nature, 445(7129):715–715, [2] R. N. Mantegna and H. E. Stanley. Introduction to Econo- 2007. physics: Correlations and complexity in finance. Cam- [6] L. Barnett, J. T. Lizier, M. Harré, A. K. Seth, and bridge University Press, Cambridge, United Kingdom, T. Bossomaier. Information flow in a kinetic Ising 1999. model peaks in the disordered phase. Phys. Rev. Lett., [3] S. Raiesdana, M. R. Hashemi Golpayegani, and A. M. 111(17):177203, 2013. Nasrabadi. Complexity evolution in epileptic seizure. In [7] T. M. Cover and J. A. Thomas. Elements of Information Engineering in Medicine and Biology Society, 2008. EMBS Theory. Wiley-Interscience, New York, second edition, 2008. 30th Annual International Conference of the IEEE, 2006. pages 4110–4113. IEEE, 2008. [8] J. P. Crutchfield. Between order and chaos. Nature [4] V. Mäki-Marttunen, J. M. Cortes, M. F. Villarreal, and Physics, 8(January):17–24, 2012. D. R. Chialvo. Disruption of transfer entropy and inter- [9] R. Shaw. The Dripping Faucet as a Model Chaotic System. hemispheric brain functional connectivity in patients with Aerial Press, Santa Cruz, California, 1984. 11

[10] D. Arnold. Information-theoretic analysis of phase transi- one-dimensional Ising model for C 70 molecular ordering tions. Complex Systems, 10:143–155, 1996. in C 70-peapods. New J. Physics, 5(1):127, 2003.

[11] J. P. Crutchfield and D. P. Feldman. Statistical complexity [31] F. M. Zimmermann and X. Pan. Interaction of H2 with of simple one-dimensional spin systems. Phys. Rev. E, Si(001) − (2 × 1): Solution of the barrier puzzle. Phys. 55(2):R1239, 1997. Rev. Lett., 85:618–621, 2000. [12] D. P. Feldman and J. P. Crutchfield. Structural informa- [32] B. H. Zimm. Theory of “melting” of the helical form tion in two-dimensional patterns: Entropy convergence in double chains of the DNA type. J. Chem. Physics, and excess entropy. Phys. Rev. E, 67(5):051104, 2003. 33(5):1349–1356, 1960. [13] D. P. Feldman, C. S. McTague, and J. P. Crutchfield. [33] R. M. Wartell and A. S. Benight. Thermal denaturation of The organization of intrinsic computation: Complexity- DNA molecules: A comparison of theory with experiment. entropy diagrams and the diversity of natural information Phys. Reports, 126(2):67 – 107, 1985. processing. CHAOS, 18(4):043106, 2008. [34] S. N. Durlauf. How can statistical mechanics contribute to [14] H. W. Lau and P. Grassberger. Information theoretic social science? Proc. Natl. Acad. Sci. USA, 96(19):10582– aspects of the two-dimensional Ising model. Phys. Rev. 10584, 1999. E, 87(2):022128, 2013. [35] R. K. Pathria and P. D. Beale. Statistical Mechanics. [15] R. G. James, C. J. Ellison, and J. P. Crutchfield. Anatomy Elsevier Science, Amsterdam, Netherlands, 1996. of a bit: Information in a time series observation. CHAOS, [36] R. J. Baxter. Exactly solved models in statistical mechan- 21(3):037109, 2011. ics. Academic Press, New York, New York, 1982. [16] T. Schreiber. Measuring information transfer. Phys. Rev. [37] P. Pfeuty. An exact result for the 1D random Ising model Lett., 85:461–464, 2000. in a transverse field. Phys. Lett. A, 72(3):245–246, 1979. [17] S. A. Abdallah and M. D. Plumbley. A measure of [38] D. P. Feldman. Computational Mechanics of Classical statistical complexity based on predictive information Spin Systems. PhD thesis, University of California, Davis, with application to finite spin systems. Phys. Lett. A, 1998. Published by University Microfilms Intl, Ann Arbor, 376(4):275–281, 2012. Michigan. [18] S. Verdú and T. Weissman. Erasure entropy. In Informa- [39] M. B. Yilmaz and F. M. Zimmermann. Exact cluster size tion Theory, 2006 IEEE Intl. Symp., pages 98–102. IEEE, distribution in the one-dimensional Ising model. Phys. 2006. Rev. E, 71:026127, Feb 2005. [19] R. G. James, K. Burke, and J. P. Crutchfield. Chaos [40] Béla Bollobás. Random Graphs, volume 73 of Cambridge forgets and remembers: Measuring information creation, studies in advanced mathematics. Cambridge University destruction, and storage. Phys. Lett. A, 378(30):2124– Press, Cambridge„ 2001. 2127, 2014. [41] B. Marcus and R. Pavlov. Computing bounds for entropy [20] E. T. Jaynes. Gibbs versus Boltzmann entropies. Am. J. of stationary Zd Markov random fields. SIAM J. Discr. Physics, 33(5):391–398, 1965. Math., 27(3):1544–1558, 2013. [21] C. E. Shannon. A mathematical theory of communication. [42] J. Lizier, M. Prokopenko, and A. Zomaya. Information Bell Sys. Tech. J., 27:379–423, 623–656, 1948. modification and particle collisions in distributed compu- [22] W. T. Grandy Jr. Entropy and the time evolution of tation. CHAOS, 20(3):037109, 2010. macroscopic systems, volume 141. Oxford University [43] M. Jerrum and A. Sinclair. Polynomial-time approxima- Press, Oxford, United Kingdom, 2008. tion algorithms for the Ising model. SIAM J. Computing, [23] D. Ruelle. Statistical Mechanics: Rigorous results. World 22(5):1087–1116, 1993. Scientific, Singapore, 1969. [44] M. D. Robinson, D. P. Feldman, and S. R. McKay. Local [24] G. R. Grimett. Percolation. Springer, Berlin, second entropy and structure in a two-dimensional frustrated edition, 1999. system. Chaos, 21:037114, 2011. [25] F. M. Reza. An introduction to information theory. [45] R. G. James, N. Barnett, and J. P. Crutchfield. Informa- Courier Corporation, 1961. tion flows? a critique of transfer entropies. Phys. Rev. [26] S. Goldstein, R. Kuik, and A. G. Schlijper. Entropy Lett., 116:238701, Jun 2016. and global Markov properties. Comm. Math. Physics, [46] P. M. Ara, R. G. James, and J. P. Crutchfield. The elusive 126(3):469–482, 1990. present: Hidden past and future dependence and why we [27] A. G. Schlijper and B. Smit. Two-sided bounds on the build models. Phys. Rev. E, 93(2):022143, 2016. free energy from local states in Monte Carlo simulations. [47] R. C. Ball, M. Diakonova, and R. S. MacKay. Quantifying J. Stat. Phys., 56(3-4):247–260, 1989. emergence in terms of persistent mutual information. Adv. [28] T. Schürmann. A note on entropy estimation. arXiv Complex Systems, 13(03):327–338, 2010. preprint arXiv:1503.05911, 2015. [48] R. B. Potts. Some generalized order-disorder transforma- [29] U. Wolff. Collective Monte Carlo updating for spin sys- tions. In Math. Proc. Cambridge Phil. Soc., volume 48, tems. Phys. Rev Lett., 62(4):361, 1989. pages 106–109, Cambridge, United Kingdom, 1952. Cam- [30] M. Yutaka, H. Kataura, K. Matsuda, and Y. Okabe. A bridge University Press.