Ising models and neural networks H.G. Schaap The work described in this thesis was performed at the Center for Theoretical Physics in Groningen, with support from the “Stichting voor Fundamenteel Onderzoek der Ma- terie” (FOM). Printed by Universal Press - Science Publishers / Veenendaal, The Netherlands. Cover design by Karlijn Hut. Copyright © 2005 Hendrikjan Schaap. Rijksuniversiteit Groningen Ising models and neural networks Proefschrift ter verkrijging van het doctoraat in de Wiskunde en Natuurwetenschappen aan de Rijksuniversiteit Groningen op gezag van de Rector Magnificus, dr. F. Zwarts, in het openbaar te verdedigen op maandag 23 mei 2005 om 16.15 uur door Hendrikjan Gerrit Schaap geboren op 7 mei 1977 te Emmen Promotores: Prof. dr. A. C. D. van Enter Prof. dr. M. Winnink Beoordelingscommissie: Prof. dr. A. Bovier Prof. dr. H. W. Broer Prof. dr. W. Th. F. Den Hollander ISBN: 90-367-2260-8 Contents 1 Introduction 9 2 General overview 19 2.1 Gibbs measures: Ising model . 19 2.1.1 The Ising model . 19 2.1.2 Thermodynamical limit . 23 2.1.3 Some choices of boundary conditions . 24 2.2 Spin glasses . 29 2.2.1 Ising spin glasses . 29 2.2.2 Mean field: SK-model . 31 2.3 Hopfield model . 31 2.3.1 Setting . 32 2.3.2 Dynamics and ground states . 32 2.3.3 System-size-dependent patterns . 35 2.3.4 Some generalizations . 36 2.4 Scenarios for the spin glass . 37 2.4.1 Droplet-picture short-range spin-glasses . 37 2.4.2 Parisi’s Replica Symmetry breaking picture . 39 2.4.3 Chaotic Pairs . 40 2.5 Metastates . 40 3 Gaussian Potts-Hopfield model 43 3.1 Introduction . 43 3.2 Notations and definitions . 44 3.3 Ground states . 46 3.3.1 Ground states for 1 pattern . 46 3.3.2 Ground states for 2 patterns . 47 3.4 Positive temperatures . 50 5 3.4.1 Fixed-point mean-field equations . 50 3.4.2 Induced measure on order parameters . 51 3.4.3 Radius of the circles labeling the Gibbs states . 53 3.5 Stochastic symmetry breaking for q = 3 . 54 4 The 2d Ising model with random boundary conditions 59 4.1 Introduction . 59 4.2 Set-up . 62 4.3 Results . 63 4.4 Geometrical representation of the model . 66 4.5 Cluster expansion of balanced contours . 73 4.6 Absence of large boundary contours . 77 4.7 Classification of unbalanced contours . 79 4.8 Sequential expansion of unbalanced contours . 82 4.8.1 Renormalization of contour weights . 84 4.8.2 Cluster expansion of the interaction between n-aggregates . 85 4.8.3 Expansion of corner aggregates . 87 4.8.4 Estimates on the aggregate partition functions . 89 η 4.9 Asymptotic triviality of the constrained Gibbs measure νΛ . 89 4.10 Random free energy difference . 91 4.11 Proofs . 95 4.11.1 Proof of Proposition 4.20 . 95 4.11.2 Proof of Proposition 4.24 . 98 4.11.3 Proof of Proposition 4.25 . 103 4.11.4 Proof of Lemma 4.26 . 103 4.11.5 Proof of Lemma 4.28 . 105 4.12 High field . 106 4.12.1 Contour representation . 108 4.12.2 Partitioning contour families . 110 4.13 Concluding remarks and some open questions . 114 4.14 Appendix on cluster models . 116 4.15 Appendix on interpolating local limit theorem . 120 A Cluster expansions for Ising-type models 123 A.1 High temperature results . 123 A.1.1 1D Ising model by Mayer expansion . 123 A.1.2 Uniqueness of Gibbs measure for high temperature or d=1 . 124 A.1.3 High temperature polymer expansions . 126 A.2 Low temperature expansions of 2D Ising model . 129 6 A.2.1 Upper bound on the pressure by cluster expansion . 129 A.2.2 Site percolation . 133 A.3 Nature of the clusters . 134 A.4 Multi-scale expansion for random systems . 136 Publications 141 Bibliography 143 Samenvatting 149 Dankwoord 157 7 8 Chapter 1 Introduction Imagine a river, continuously flowing on, in a constant way running its course; the water in the river always tends to flow in the direction with the minimal resistance possible. Suppose during heavy rain the water in the river becomes higher and higher. Then, suddenly, the water becomes so high that the river overflows its banks, allowing the water to flow in the nearby meadows. Then it takes some time before the river is adjusted to this new situation. Statistical physics is an attempt for modelling natural processes. It tries to connect the microscopic properties with the macroscopic properties. For the river the movement of the water molecules is connected to the overall flow. The global properties of the river are given by some universal laws for some characteristics, notwithstanding the huge number of involved water molecules. Because of this huge number we have to apply probabilistic methods (stochastics) on the underlying microscopic differential equations defining the movement of all of the water molecules. Then if we look at the macroscopic properties, we know almost for certain its global behavior. This global behavior can be described by equations only depending on macroscopic properties. The underlying microscopic part is removed by the performed stochastics. Let us take a die to demonstrate some of the involved stochastic principles. As we know, we have the same probability to throw a 1 or a 6. However, experience tells us that after a small number of throws, the resulting relative frequencies of individual numbers can significantly differ from each other. Only when we throw a die a large number of times, the resulting relative frequencies of the numbers become more and more equal. The same result can be arranged when we throw not one die many times, but rather when we throw a lot of dice at once and then look at the relative number frequencies. Obviously throwing one die 1000 times is equivalent to throwing 1000 dice one time. 1 Eventually all of the relative frequencies approach 6 : on average every number appears 9 once if we throw a die six times. This relative frequency of a number is sometimes identified as the probability of the number to appear. To shorten notation it is denoted 1 by P (i). For every number i on our die, P (i) = 6 . The function which assigns to each number i the corresponding probability, we call the probability distribution of the property. Suppose we have two dice. Then the combined probability equals P (n1, n2), where ni represents the numbers on the two dice. For instance P (1, 1), the probability of 1 1 1 throwing with both dice a 1 equals 6 · 6 = 36 . Note that the throw of one die does not depend on the outcome of the throw of the other die. We say that the outcome of die 1 is independent of the outcome of die 2. When two properties are dependent on each other the expression for the combined probability is in general more complicated. All matter around us is made out of atoms. Every gram of matter contains around 1023 atoms. Often one considers a collection of some global (bulk) properties in addi- tion to the atomic properties. In the description of matter, all the atoms together with the mentioned global properties define the system. Any particular realization of the corresponding atomic values is called a configuration of the system. Determining the configuration resembles the throwing of 1023 dice at once. Suppose one wants to measure a macroscopic property, for instance the average density. Because of the large number of atoms, in the probability distribution of the atomic values there is no need of the possibility of tracking the locations of the single atoms. In case of the river: during the heavy rainfall, the river has a way of flowing which does change in time. After some adjustments are made, the changes do stop; the river flow becomes stationary. The time scale of the adjustments is extremely large compared to the scale of the local movements of the water molecules. The way in which the system properties are evolving we call the dynamics of the system. In the global flow of the river the microscopic movements of the water molecules have been averaged out. In the next chapter we give a general overview of the part of statistical physics which is important for the two particular kind of models we study in this thesis: neural networks and Ising models. Neural networks The first subject of the thesis is about a model originating in the theory of neural net- works. In particular we like to understand the concept of memory. Our brain is built up out of billions of neurons connected in a highly non-trivial way. This structure we call a neural network. It is difficult to study it directly, because of the huge number of neu- rons involved in a relatively small area. In order to understand how the memory works, a common approach is to build a simpler model which captures its main features. Just as the neural network of the brain, the model should be sufficiently robust: in transmit- 10 Figure 1.1: Components of a neuron (taken from [19]) ting signals between neurons there are always some small errors involved. Given this slightly deformed signal, the brain is able to remove the noise and to reconstruct the pure signal. For a good general overview of neural networks see [19]. A neuron is build up of three parts: the cell body, the dendrites and the axon, see Figure 1.1.