From Random Fields to Networks Ibrahim M. Elfadel RLE Technical Report No. 579 June 1993 The RESEARCH LABORATORY of ELECTRONICS MASSACHUSETTS INSTITUTE of TECHNOLOGY CAMBRIDGE, MASSACHUSETTS 02139-4307 From Random Fields to Networks Ibrahim M. Elfadel RLE Technical Report No. 579 June 1993 Research Laboratory of Electronics Massachusetts Institute of Technology Cambridge, Massachusetts 02139-4307 This work was supported in part by the National Science Foundation under Grant MIP 91-17724. From Random Fields to Networks by Ibrahim M. Elfadel Submitted to the Department of Mechanical Engineering on February 1, 1993, in partial fulfillment of the requirements for the degree of Doctor of Philosophy Abstract Gibbs random fields (GRF's) are probabilistic models inspired by statistical mechanics and used to model images in robot vision and image processing. In this thesis, we bring the analytical methods of statistical mechanics to bear on these models. Specifically, we address and solve the following fundamental problems: 1. Mean-field estimation of a constrained GRF model: The configuration space of a GRF model is often constrained to produce interesting" patterns. We develop mean-field equations for estimating the means of these constrained GRF's. The novel feature of these equations is that the finiteness of graylevels is incorporated in a "hard" way in the equations. 2. Correlation-field estimation of a GRF model: GRF correlation functions are generally hard to compute analytically and expensive to compute numerically. We use the mean- field equations developed above to propose a new procedure for estimating these corre- lation functions. Our procedure, which is valid for both unconstrained and constrained models, is applied to the quadratic interaction model, and a new closed-form approxima- tion for its correlation function in terms of the model parameters is derived. 3. Network representation: We show how the mean-field equations of the GRF model can be mapped onto the fixed-point equations of an analog winner-take-all (WTA) network. The building block of this network is the generalized sigmoid mapping, a natural general- ization of the sigmoidal function used in artificial neural networks. This mapping turns out to have a very simple VLSI circuit implementation with desirable circuit-theoretic properties. 4. Solution algorithms: Iterated-map methods and ordinary differential equations (ODE's) are proposed to solve the network fixed-point equations. In the former, we show, using Lyapunov stability theory for discrete systems, that the worst that could happen during synchronous iteration is an oscillation of period 2. In the latter, we show that the ODE's are the gradient descent equations of energy functions that can be derived from the mean- field approximation. One of our gradient descent algorithms can be interpreted as the generalization to analog WTA networks of Hopfield's well-known algorithm for his analog network. 5. Temperature dependence: The GRF temperature parameter reflects the thermodynamic roots of the model. Using eigenstructure analysis, we study the temperature effect on 2 the stability of the WTA network fixed points. In particular, we derive new closed-form formulas for the critical temperature of a large class of models used in grayscale texture synthesis. The stability study is used to gain insight into the phase transition behavior of GRF's. The implications of these results for image modeling, optimization, and analog hardware im- plementation of image processing and optimization algorithms are discussed. Thesis Supervisor: John L. Wyatt, Jr. Professor of Electrical Engineering and Computer Science 3 Acknowledgments First and foremost, I would like to acknowledge the continuous encouragement and support I have received from my thesis supervisor, Professor John L. Wyatt, Jr. It has been for me an honor and a privilege to work with John both as his research and teaching assistant. His emphasis on clarity, depth, and rigor has certainly shaped my own approach to academic research. I would also like to express my gratitude for his close and interested reading of the thesis document. I am thankful to my thesis committee members: Professor Berthold Horn for asking the right questions and Professor Alan Yuille for helping me answer them. I am also grateful to Alan for the time he gave me and for the many illuminating discussions we had. Special thanks go to the chairman of my committee, Professor Derek Rowell, for his help in getting me better organized and more goal-oriented. Mission accomplished. Witness this document! Many MIT faculty members contributed both academically and financially to my graduate education while at MIT. In particular, I would like to acknowledge the financial support that I received, in the form of research assistantships, from Professors David Hardt, Sanjoy Mitter, and Ronald Rivest, and in the form of a teaching assistantship from Professor Munther Dahleh. I would also like to acknowledge and thank Professor Jacob White for his readiness to help and for his advice on both academic and non-academic matters. Doctor Robert Armstrong, my officemate for the past three years, was always there to patiently answer my ill-posed questions about computing and text formatting with the well-thought replies of an outstanding expert. Bob also ended up proofreading parts of this thesis. All the remaining bugs and typos are solely my responsibility. A zillion thanks, Bob! While working on my own dissertation, I had the unique privilege of collaborating with two excellent colleagues: Professor Rosalind Picard and Professor Andrew Lumsdaine. It is a pleasure to thank them for sharing research problems and ideas with me. 4 To past and present members of the VLSI-CAD group at RLE: thanks for your friendship and for sharing your thoughts, stories, jokes, and math questions with me. To past and present members of the MIT Vision Chip Project: thanks for being so patient with this "theory stuff." To my friends and colleagues in Syria, France, Japan, and the US: thanks for your cheers along the long road. To my relatives in Syria who have been calling me Doctor" since I was in junior high: sorry you had to wait so long! Without my family's financial sacrifices, emotional support, and unconditional love, this thesis would have remained no more than a possibility. To them I say:"I love you all." 1, - 4D L VI - . 0 I I II 1P, ,_,. P J The work described in this thesis was supported in part by the National Science Foun- dation and the Defense Advanced Research Projects Agency (DARPA) under Grant No. MIP-88-14612, and by the National Science Foundation under Grant No. MIP-91-17724. 5 to my mother my sister my brothers and the blessed memory of my father 6 _____ --------- - - _ __ _____ - - Contents 1 Introduction 11 1.1 Notation and Definitions ........................... 13 1.2 Markov Random Fields and Gibbs Distributions ............. ..... 15 1.3 Contributions ................................. 20 1.4 Overview ......................................... 22 2 Correlation Functions of Gaussian and Binary Gibbs Distributions 24 2.1 Need for Correlation Functions ....................... 26 2.1.1 Image Coding ............................. 26 2.1.2 Retinal Information Processing ................... 26 2.1.3 Pattern Analysis ........................... 27 2.1.4 Pattern Synthesis ........................... 27 2.2 One-Dimesional Case: Gaussian Processes ................. 28 2.2.1 Causal Case: Gauss-Markov Processes ............... 28 2.2.2 Noncausal Case: Gaussian Reciprocal Processes ......... ... 30 2.2.3 Noncausal Autoregressive Process ................. ....... 38 2.2.4 Conditional Probability of the Autoregressive Model ...... 39 2.3 Two-Dimensional Case: Gauss-Markov Random Fields ......... ... 43 2.3.1 Correlation Function ....................... ........... 44 2.3.2 Power Spectrum . 46 2.3.3 Gauss-Gibbs Distribution: Hammersley-Clifford Revisited ..... 48 2.4 Ising Models .................................. 49 2.4.1 One-Dimensional Ising Model .................... 50 2.4.2 Two-Dimensional Ising Model .................... 54 2.4.3 Other Exactly Solvable Models ........................... 58 2.4.4 Effect of Constraints ........................ .......... 58 2.5 General Case: Mean-Field Approximation ........................ 61 7 3 Mean-Field Approximations of Gibbs Distributions 64 3.1 Unconstrained Binary Quadratic Case ................... 65 3.1.1 Effective Energy ........................... 66 3.1.2 Approximation of the Partition Function .............. 68 3.1.3 Effective Energy in the Presence of an External Field ...... 69 3.1.4 Approximation of the Mean-Field ................. ....... 71 3.1.5 Effective Energy and Free Energy ................. ....... 73 3.1.6 Remarks ........................................ 75 3.2 Unconstrained Multilevel Case ................................ 76 3.2.1 Effective Energy ........................... 76 3.2.2 Effective Energy and Free Energy ................. ....... 80 3.2.3 Generalization ............................. 81 3.3 Constrained Multilevel Case ......................... 84 3.3.1 Saddle Point Method ......................... 84 3.3.2 Probability Factorization Method ................. ....... 91 3.4 On the Legendre Transformation ..................... ........ 94 4 Correlation-Field Approximations of Gibbs Distributions 96 4.1 Unconstrained Binary Case ......................... 98 4.2 Unconstrained Multilevel Case ....................... ........ 100 4.2.1 A General