UAI2003 WANG ET AL. 567
Boltzmann Machine Learning with the Latent Maximum Entropy Principle
Shaojun Wang Dale Schuurmans Fuchun Peng Yunxin Zhao University of Toronto University of Waterloo University of Waterloo University of Missouri Toronto, Canada Waterloo, Canada Waterloo, Canada Columbia, USA
Abstract configurations of the hidden variables (Ackley et al. 1985, Welling and Teh 2003). We present a new statistical learning In this paper we will focus on the key problem of esti paradigm for Boltzmann machines based mating the parameters of a Boltzmann machine from on a new inference principle we have pro data. There is a surprisingly simple algorithm (Ack posed: the latent maximum entropy principle ley et al. 1985) for performing maximum likelihood (LME). LME is different both from Jaynes' estimation of the weights of a Boltzmann machine-or maximum entropy principle and from stan equivalently, to minimize the KL divergence between dard maximum likelihood estimation. We the empirical data distribution and the implied Boltz demonstrate the LME principle by deriving mann distribution. This classical algorithm is based on new algorithms for Boltzmann machine pa a direct gradient ascent approach where one calculates rameter estimation, and show how a robust (or estimates) the derivative of the likelihood function and rapidly convergent new variant of the with respect to the model parameters. The simplic EM algorithm can be developed. Our exper ity and locality of this gradient ascent algorithm has iments show that estimation based on LME attracted a great deal of interest. generally yields better results than maximum However, there are two significant problems with the likelihood estimation when inferring models standard Boltzmann machine training algorithm that from small amounts of data. we address in this paper: First, the convergence rate of the standard algorithm is extremely slow. It is well known that gradient ascent converges only linearly, 1 Introduction with a rate that depends critically on the condition Boltzmann machines are probabilistic networks of bi of the Hessian matrix of the likelihood function. In nary valued random variables that have wide applica order to achieve faster convergence one generally has tion in areas of pattern recognition and combinatorial to adjust the step size. In practice, gradient ascent optimization (Aarts and Korst 1989). A Boltzmann methods often select step sizes by performing an ex machine can be represented as an undirected graph plicit line search in the gradient direction, which can ical model whose associated probability distribution lead to non-trivial overhead. has a simple quadratic energy function, and hence has Second, it is known that the true distribution is multi the form of a Boltzmann-Gibbs distribution. modal, and the likelihood function has multiple lo Typically, the random variables are divided into a set cal maxima. This raises the question of which local of visible variables, whose states are clamped at ob maximizer to choose as the final estimate. Fisher's served data values, and a separate set of hidden vari classical maximum likelihood estimation (MLE) prin ables, whose values are unobserved. Inference, there ciple states that the desired estimate corresponds to fore, usually consists of calculating the conditional the global maximizer of the likelihood function. How probability of a configuration over the hidden vari ever, in practice it is often observed that MLE leads ables, or finding a most likely configuration of the to over-fitting (poor generalization) particularly when hidden variables, given observed values for the visible faced with limited training data. variables. Inference in Boltzmann machines is known To address both of the above problems in the context to be hard, because this generally involves summing of Boltzmann machines, we have recently proposed a or maximizing over an exponential number of possible 568 WANG ET AL. UA12003
new statistical machine learning framework for density 2 The latent maximum entropy estimation and pattern classification, which we refer principle to as the latent maximum entropy (LME) principle (Wang et a!. 2003).1 Although classical statistics is To formulate the LME principle, let X E X be a ran heavily based on parametric models, such an approach dom variable denoting the complete data, Y E Y be can sometimes be overly restrictive and prevent accu the observed incomplete data and Z E Z be the miss rate models from being developed. The alternative ing data. That is, X = (Y, Z). If we let p(x) and principle we propose, LME, is a non-parametric ap p(y) denote the densities of X and Y respectively, and proach based on matching a set of features in the data let p(zjy) denote the conditional density of Z given Y, (i.e. sufficient statistics, weak learners, or basis func where = then p(y) = fzEZp(x) f.l(dz) p(x) p(y)p(zjy). tions). This technique becomes parametric when we LME principle Given features !J, ... specifying necessarily have to approximate the principle. LME is JN, the properties we would like to match in the data, an extension to Jaynes' maximum entropy (ME) prin select a joint probability model from the space of all ciple that explicitly incorporates latent variables in the p* distributions over to maximize the joint entropy formulation, and thereby extends the original principle P X to cases where data components are missing. The re sulting principle is different from both maximum likeli H(p) = p(x) logp(x) f.l(dx) (1) hood estimation and standard maximum entropy, but -1x EX often yields better estimates in the presence of hidden subject to the constraints variables and limited training data.
In this paper we demonstrate the LME principle by de fi(x)p(x) f.l(dx)= Lfi(y) { fi(x)p(zJy) f.l(dz) 1x z riving new algorithms for Boltzmann machine param EX yEY f EZ eter estimation, and show how a robust and rapidly i = l...N, Y and Z not independent (2) convergent new variant of the EM algorithm can be de veloped. Our experiments show that estimation based where x = (y, z). Here ji(y) is the empirical distribu on LME generally yields better results than maximum tion over the observed data, and Y denotes the set of likelihood estimation, particularly when inferring hid observed Y values. Intuitively, the constraints specify den units from small amounts of data. that we require the expectations of fi(X) in the com The remainder of the paper is organized as follows. plete model to match their empirical expectations on First, in Section 2 we introduce the latent maximum the incomplete data Y, taking into account the struc entropy principle, and then outline a general training ture of the dependence of the unobserved component algorithm for this principle in Section 3. Once these Z on Y. preliminaries are in place, we then present the main Unfortunately, there is no simple solution for p* in contribution of this paper-a new training algorithm (1,2). However, a good approximation can be obtained for Boltzmann machine estimation-in Section 4. Sec by restricting the model to have an exponential form tion 5 compares the new parameter optimization pro cedure to standard procedures, and Section 6 compares ( ) P>..(x) = P 1 exp >. i(x) the generalization performance of the new estimation :\ t d principle to standard maximum likelihood. We con clude the paper with a brief discussion in Section 7. where P>.. f exp is a nor = xEX (2::�1 >.;J;(x)) f.l(dx) malizing constant that ensures = 1. fx x P>.. (x) f.l(dx) This restriction provides a free parameterE >.; for each feature function k By adopting such a "log-linear" restriction, it turns out that we can formulate a prac
1 A third problem, which we do not directly address in tical algorithm for approximately satisfying the LME this paper, is that calculating a gradient involves summing principle. over exponentially many configurations of the hidden vari ables, and thus becomes intractable in large networks. It is therefore usually impractical to perform exact gradient 3 A general training algorithm for ascent in these models, and some form of approximate in log-linear models ference (Welling and Teh 2003) or Monte Carlo estimation (Ackley et a!. 1985) is needed to approximate the gradi To derive a practical training algorithm for log-linear ent. For the purposes of this paper, we simply assume that some sort of reasonable approximation technique is avail models, we exploit the following intimate connection able. See (Southey et a!. 2003) for an attempt to improve between LME and maximum likelihood estimation classical estimators for this problem. (MLE). UAI2003 WANG ET AL. 569
Theorem 1 Under the log-linear assumption, maxi longer depends on >.. but rather on the fixed constants mizing the likelihood of log-linear models on incom from the previous iteration >..U). This means that plete data is equivalent to satisfying the feasibility con maximizing (4) subject to (5) with respect to >.. is straints of the LME principle. That is, the only dis now a convex optimization problem with linear con tinction between MLE and LME in log-linear models straints. The generalized iterative scaling algorithm is that, among local maxima (feasible solutions), LME (GIS) (Darroch et al. 1972) or improved iterative scal selects the model with the maximum entropy, whereas ing algorithm (liS) (Della Pietra et al. 1997) can be MLE selects the model with the maximum likelihood used to maximize Q ( >.., >..(j)) very efficiently. (Wang et al. 2003). From these observations, we can recover feasible log This connection allows us to exploit an EM algorithm linear models by using an algorithm that combines EM (Dempster et al. 1977) to findfeasible solutions to the with nested iterative scaling to calculate the M step. LME principle. It is important to emphasize, how EM-IS algorithm: ever, that EM will only find alternative feasible solu E Btep: Given >..Ul, for each feature /;, i = 1, tions, while the LME and MLE principles will differ ... , N, calculate its current expectation j) with respect to markedly in the feasible solutions they prefer. We il 17i by: lustrate this clistinrtion helow. )..U) To formulate an EM algorithm for learning log-linear i) 17i = LP(Y) 1 f;(x)p-\
>.. >.. >..') M Btep: Perform S iterations of full parallel update L( ) L ii(y) log p"(y) = Q(>..,') >.. + H( , of parameter values >..1, ...,A N either by GIS or liS as yEY follows. Each update is given by where Q(>.., ')>.. = p(y) PA'(zi y) logpA(x )Jl(dz), L: f di s _ di+(s-1)/S) . yEY zEZ + (S) - + (7) H(>.. (y) y) log ( y) (d . Ai /\ i ft , >..') = - L:yEY p f Z P-\'(zi p,\ zi Jl z) This is a standard decompositionzE used for deriving such that satisfies EM. For log-linear models, in particular, we have /i
Q(.\, >..(j) ) = -log( .\ ( f p d ) ; p(y) ;(x) -\ ME-EM-IS algorithm: Initialization: Choose random initial values for A. EM-IS: Run EM-IS to convergence, to obtain feasible A*. Entropy calculation: Calculate the entropy of PA·. Model selection: Repeat the above steps several times to produce a set of distinct feasible candidates. Choose the feasible candidate that achieves the highest entropy. I Figure 1: Boltzmann machine model: the nodes Y are This leads to a new estimation technique that we will observed and the nodes Z are unobserved. compare to standard MLE below. One apparent corn plication, first, is that we need to calculate the en abies in the model. Specifically, we consider the fea tropies of the candidate models produced by EM-IS. tures fkt(x) = YkY£, fkm(x) = YkZm, fmn(x) = ZmZn, However, it turns out that we do not need to calculate for 1 :<:; k < £ :<:; J and 1 :<:; m < n :<:; L, where entropies explicitly because one can recover the en x = (y,z) = (y1,... ,y,,z1, ... ,zL)· Note that the fea tropy of feasible log-linear models simply as a byprod tures are all binary, and therefore we can represent the uct of running EM-IS to convergence. structure of the log-linear model by a graph, as shown in Figure 1. Corollary 1 If A* is feasible, then Q(A*, A*) Given a sequence of observed data 1 we -H(PA· ) and L(A*) = -H(PA•) + H(A*, A*). Y = (y ,... , yr), formulate the LME principle as Therefore, at a feasible solution A*, we have already maxH(X) = H(Y) + H(ZIY) (9) calculated the entropy, -Q(A*, A*), in the M step of p(x) EM-IS. subject to To draw a clear distinction between LME and MLE, assume that the term H (A*, A*) from Corollary 1 L YkYf p(x) = L YkYt fJ(y) is constant across different feasible solutions. Then xEX MLE, which maximizes L(A*), will choose the model L YkZm p(x) = LYk fJ(y) L Zm p(ziy) that has lowest entropy, whereas LME, which maxi xEX yEY zE{O,t}L mizes H (PA. ) , will chose a model that has least like lihood. (Of course, H(A*, A*) will not be constant in L ZmZn p(x) = L ZmZn p(z) practice and the comparison between MLE and LME xEX zE{O,t}L is not so straightforward, but this example does high for 1 :<:; k <£ :<:; J and 1 :<:; m < n :<:; L light their difference.) The fact that LME and MLE are different raises the question of which method is where x = (y, z) = (y1,... ,YJ, z1, ... ,zL) and jj(y)= �· the most effective when inferring a model from sample Again we can apply EM-IS to find a feasible log-linear data. model. To execute the E step, calculate the feature expectations according to (6) 4 Application to Boltzmann machine 1 T ,.,Ul __ " t t training ·•k,t - T L....- YkY t t=l Consider a graphical model with M binary nodes tak 1 T ing on values in {0, 1 }. Assume that among these 7]��= T LY� L Zm p(ziy1) t=! nodes there are J observed nodes Y = ( Yt, ... , Y,), zE{O,l}L and L = M- J hidden nodes Z Z Let = (Zt, ..., L)· ryg�n = L ZmZn p(z) X = (Y, Z). Thus, Y Z {0 L and = {0, 1}', = ,1} zE{O,l}L X 1}'+L {0, 1}M. For this problem, the ob = {0, = for :<:; :<:; J and :<:; n served data has the form of a J dimensional vector 1 k < £ 1 m < :<:; L y = (y1,.. . ,YJ) E {0,1 }'. Given an observed sequence To execute the M step we then formulate the simpler of T ]-dimensional vectors Y = (y1,... , yr), where maximum entropy problem with linear constraints, as y1 E {0, 1 }' for t = 1, ..., T, we attempt to infer a in ( 4) and (5) latent maximum entropy model that matches expec rnaxH(X) = H(Y) + H(ZIY) (10) tations on features defined between every pair of vari- p(x) UAI2003 WANG ET AL. 571 subject to LYkY£ p(x) TJk,((j) Carlo estimation (Ackley et a!. 1985), mean field the xEX ory (Kappen ad Rodriguez, 1998) or generalized belief propagation (Wainwright et a!. 2003, Welling and Teh, LYkZm p(x) = TJ(j) k,m 2003,Yeddida et a!. 2002) when M is large. xEX L ZmZn p(x) TJm,n(j) 5 Comparing EM-IS training to xEX standard EM for 1 ::; k < f ::; J and 1 ::; m < n ::; L To compare EM-IS to standard Boltzmann machine where .•. ). In this = (y,z) = (y1, ...,yJ ,z1, ,z£ estimation techniques, first consider the derivation of case, thex probability distribution for the complete data a direct EM approach. In standard EM, given the pre model can be written vious parameters (j), one solves for new parameters by maximizing theA auxiliary Q function with respect PA(x) = PA(z,y) A to A =_!__e�YT i\yy+�zT Azz+yT Ayzz A 1 = LT L PA'(zjyt)iogpA(Yt,z) -e2lxTAx Q(A,A') � 1 T = ( A) + L L PA'(ziy') where = is the M M symmetric -log T A [ ��zA�; ] x 2 xTAx t=l zE{O,l}L matrix of .X parameters corresponding to the features over the variable pairs (with the diagonal elements of A Taking derivatives with respect to A gives equal to zero), and = e�xTAx is the nor A I::xE{D,l}M malization factor. This graphical model corresponds : Q(A, A') (11) to a Boltzmann machine (Ackley et a!. 1985). To solve A ) 1 T 1 T for the optimal Lagrange multipliers A (j in the M step = + L L T we once again need to use iterative scaling. Following -2 EPA[xx ] 2T xx PA'(zly') t=l zE{O,l}L (7), we iteratively improve A(J) by adding the update parameters l(j+s/S) that satisfy (8). These can be cal Apparently there is no closed form solution to the M culated by by using Newton's method or the bisection step and a generalized EM algorithm has to be used in method to solve for l(j+s/S) in this case. The standard approach is to use a gradient ascent to approximately solve the M step. However, the step size needs to be controlled to ensure a mono 1 tonic improvement in Q. L YkYI xE{0,1)M Here 1 is the M dimensional vector with all 1 ele We note that in previous work, Byrne (1992) has pro ments, and IM is the M x M identity matrix. The posed a sequential update algorithm for the M step required expectations can be calculated by direct enu in a Boltzmann machine parameter estimation algo meration when M is small, or approximated by Monte rithm. However, to maintain monotonic convergence, 572 WANG AL. ET UA12003 hood. Corollary 1 shows that these are not equivalent. We are interested in determining which method yields better estimates of various underlying models p* used to generate the data. We measure the quality of an estimate PA by calculating the cross entropy from the -310 .... correct marginal distribution p* (y) to the estimated ' ' marginal distribution (y) on the observe data com _,, ' PA d gredien1Uoet11 :,' ponent Y -330 ·, ' ' p*(y) (p*(y) P ( )) p*(y) log (dy) D ii AY = 1 P ( ) f..l yEY A Y The goal is to minimize the cross entropy between the Figure 2: Convergence evaluation: log-likelihood ver marginal distribution of the estimated model PA and sus iteration, solid curve denotes EM-IS with S=4, dot the correct marginal p*. A cross entropy of zero is ted curve denotes EM-IS with S=1,and dashed curve obtained only when PA(Y) matches p*(y). denotes gradient ascent. We consider a series of experiments with differentmod els and different sample sizes to test the robustness of Byrne's algorithm requires a large number of itera both LME and MLE to sparse training data. In par tions in the M step to ensure that a maximum is ticular, we used the following experimental design. achieved. Otherwise the monotonic convergence prop erty can be violated for the sequential updates he pro 1. Fix a target Boltzmann machine model p*(x) poses. In our case, EM-IS uses a parallel update proce p*(y,z). dure that avoids this difficulty. A sequential algorithm that maintains the monotonic convergence property 2. Generate a sample of observed data Y can also be achieved along the lines of (Collins et al. (YI, ... ,YT) according to p*(y). 2002). 3. Run EM-IS to generate multiple feasible solutions by starting from 100 random initial parameters A. 6 Comparing ME-EM-IS estimation to maximum likelihood 4. Calculate the entropy and likelihood for each fea sible candidate. Even assuming that one has an effective algorithm for local parameter optimization, there remains the issue 5. Select the maximum entropy candidate PLME as of coping with multiple local maxima. To select from the LME estimate, and the maximum likelihood local maxima we propose to use the new latent maxi candidate PMLE as the MLE estimate. mum entropy (LME) technique instead of the classical 6. Calculate the cross entropy from p*(y) to the maximum likelihood (MLE) approach. To ascertain whether LME or MLE yields better estimates when marginals PLME(Y) and PMLs(y) respectively. inferring models from sample data that has missing 7. Repeat Steps 2 to 6 5 times and compute the av component, we conducted a simple experiment. In erage of the respective cross entropies. That is, particular, we considered a simple eight nodes Boltz average the cross entropy over 5 repeated trials mann machine with 5 observable and 3 hidden units for each sample size and each method, in each ex as a case study. periment. The basis for comparison between LME and MLE is to realize that by the discussion in Section 3, any fea 8. Repeat Steps 2 to 7 for different sample sizes T. sible solution to the LME principle corresponds to a locally maximum likelihood model as specifiedby ( 11). 9. Repeat Steps 1 to 8 for differentgenerative models Therefore, we can implement EM-IS as outlined in Sec p*(x) . tion 3 and generate feasible candidates for the LME and MLE principles simultaneously. From Theorem 1, In this experiment we generated the data according we know that LME and MLE consider the same set of to a Boltzmann machine with 5 observable and 3 hid feasible candidates, except that among feasible solu den units, and attempted to learn the parameters for a tions, LME selects the model with the highest entropy, Boltzmann machine that assumed the same architec whereas MLE selects the model with the highest likeli- ture. UAI2003 WANG ET AL. 573 -2.35,------,, .. ,------�------, " "" 0.18 "" •. 5 0.16 ] -25 l -2.6 ! '" -2.65 '" ..:!.7 5 " -28� 'c------�,-;-, ------J , o.o�'c, -�-----�-;- , ------...... J , ,. , 0 , , S•mples�te s,mpleaiz& Figure 3: Average log-likelihood of the MLE estimate Figure 5: Average cross entropy between the true dis versus the LME estimates in Experiment 1. tribution and the MLE estimate versus the LME esti mates in Experiment 1. about the underlying model. Figure 6 shows that LME obtained a significantlylower cross entropy than MLE. 0.32,------�------, '' "'-' 0.28 "' ' �0.:!4 3-�L,------� ,--, ------J , � 0.22 , , , Sampltloiz• g " 0.18 Figure 4: Average entropy of the MLE estimate versus the LME estimates in Experiment 1. 0.16 0.14 0.1� , 0 ,. Figures 3 and 4 first show that the average log Swnple•lz• likelihoods and average entropies of the models pro duced by LME and MLE respectively behave as ex Figure 6: Average cross entropy between the true dis pected. MLE clearly achieves higher log-likelihood tribution and the MLE estimate versus the LME esti mates in Experiment 2. than LME, however LME clearly produces models that have significantly higher entropy than MLE. The in teresting outcome is that the two estimation strategies In our third experiment we used a generative model obtain significantly different cross entropies. Figure 5 that was a Boltzmann machine with five observable reports the average cross entropy obtained by MLE and one hidden and the data were generated by this and LME as a function of sample size, and shows the architecture. Again the LME and MLE estimators result that LME achieves substantially lower cross en inferred Boltzmann machine with five observable and three tropy than MLE. LME's advantage is especially pro hidden in this case, and hence were making an nounced at small sample sizes, and persists even when incorrect 'overcomplete' assumption about the under sample sizes as large as 1,000 are considered (Figure 5). lying model. In our second experiment we used a generative model Figure 7 shows that LME still obtained a significantly that was a Boltzmann machine with five observable lower cross entropy than MLE. and five hidden units. Specifically, we generated data Although these results are anecdotal, we have wit with this architecture. The LME and MLE estima nessed a similar outcome on several other models as tors still only inferred a Boltzmann machine with five well. Wider experimentation on synthetic data and observable and three hidden in this case, and hence real MRF application and theoretical analysis are nec were making an incorrect 'undercomplete' assumption essary to confirm this as a general conclusion. More- 574 WANG ET AL. UAI2003 on,------�------� and Boltzmann Machines, John Wiley & Sons. 0.28 W. Byrne (1992). Alternating minimization and ,, Boltzmann machine learning. IEEE Trans. on Neural "" Networks, 4) :612-620. "' 3( f 0.25 M. Collins, R. Schapire and Y. Singer (2002). Logis � tic regression, AdaBoost and Bregman distances. Ma chine Learning, 48(1-3):253-285 '" J. Darroch and D. Ratchliff (1972). Generalized it 0.22 erative scaling for log-linear models. The Annals of Mathematical Statistics, 43-5, 1470-1480. , ______02�0';-, ------..';- _J.. . S•mpktslze S. Della Pietra, V. Della Pietra and J. Lafferty (1997). Inducing features of random fields. IEEE Transactions Figure 7: Average cross entropy between the true dis on Pattern Analysis and Machine Intelligence, 19-4, tribution and the MLE estimate versas the LME esti 380-393. mates in Experiment 3. A. Dempster, N. Laird and D. Rubin (1977). Maxi mum likelihood estimation from incomplete data via over, it is worthwhile to discuss and compare with the EM algorithm. J. Royal Statis. Soc. B, 39, 1-38. training by minimizing contrastive divergence (Hinton G. Hinton (2002). Training products of experts by 2002). minimizing contrastive divergence Neural Computa tion, 14, 1771-1800. 7 Conclusion H. Kappen and F. Rodriguez (1998). Efficientlearning in Boltzmann machines using linear response theory. We have introduced a new inference technique, the Neural Computation, 10, 1137-1156. latent maximum entropy principle, and used it to de S. Lauritzen (1995). The EM-algorithm for graphi rive a new local parameter optimization method, EM cal association models with missing data. Comput. IS, and also a new selection technique, ME-EM-IS, Statist. Data Analysis, 1, 191-201. for Boltzmann machines. These principles yield new training and estimation algorithms that perform very S. Riezler (1999). Probabilistic constraint logic pro effectively when compared to the standard methods. gramming. Ph.D. Thesis, University of Stuttgart. There remain several avenues to extend this work. T. Sejnowski (1986). Higher order Boltzmann ma More generally, by allowing binary-valued features of chines. Neural Networks for Computing, J. Denker, Ed. American Institute of Physics, 386-403. the form fv(w) = Wv1Wv2 ...Wv. for v = (vJ, ... vn ) , a path in G = (E, V), we construct models that are F. Southey, D. Schuurmans and A. Ghodsi (2003). essentially "higher-order" Boltzmann machines (Se Regularized greedy importance sampling. Advances jnowski 1986). in Neural Information Processing Systems 15. Also, the training and estimation methods can still N. Ueda and R. Nakano (1998). Deterministic anneal be improved by tackling them simultaneously. In this ing EM algorithm. Neural Networks, 11, 272-282. paper, by randomly choosing different starting points, M. Wainwright, T. Jaakkola and A. Willsky (2003). we take the feasible log-linear model with maximum Tree-based reparameterization framework for analysis entropy value as the LME estimate. This procedure is of belief propagation and related algorithms. IEEE computationally expensive. Thus it is worthwhile to Trans. on Information Theory, 49, 1120-1146. develop an analogous deterministic annealing ME-EM IS algorithm to automatically find feasible maximum S. Wang, D. Schuurmans and Y. Zhao (2002). The entropy log-linear model for LME (Ueda and Nakano latent maximum entropy principle. manuscript 1998). M. Welling andY. Teh (2003). Approximate inference in Boltzmann machines. Artificial Intelligence , 143, References 19-50. D. Ackley, G. Hinton and T. Sejnowski (1985). A J. Yedidia, W. Freeman, and Y. Weiss (2002). Con learning algorithm for Boltzmann machines. Cogni structing free energy approximations and generalized tive Science, 9(1):147-169 belief propagation algorithms. submitted to IEEE Trans. on Information Theory. E. Aarts and J. Karst (1989). Simulated Annealing