i i i i Pattern Theory i i i i i i i i APPLYING MATHEMATICS A Collection of Texts and Monographs ADVISORY BOARD Robert Calderbank Jennifer Chayes David Donoho Weinan E George Papanicolaou i i i i i i i i Pattern Theory The Stochastic Analysis of Real-World Signals David Mumford Agnes` Desolneux A K Peters, Ltd. Natick, Massachusetts i i i i Cover illustrations courtesy of Alex May CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2010 by Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works Version Date: 20150515 International Standard Book Number-13: 978-1-4398-6556-9 (eBook - PDF) This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint. Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, trans- mitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copyright. com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com i i i i To Ulf Grenander, who has dared to propose mathematical models for virtually all of life, pursuing the motto on the wall of his study Using Pattern Theory to create mathematical structures both in the natural and the man-made world i i i i i i i i i i i i i i i i Contents Preface ix Notation xi 0 What Is Pattern Theory? 1 0.1TheManifestoofPatternTheory................ 1 0.2TheBasicTypesofPatterns................... 5 0.3 Bayesian Probability Theory: Pattern Analysis andPatternSynthesis...................... 9 1 English Text and Markov Chains 17 1.1BasicsI:EntropyandInformation............... 21 1.2 Measuring the n-gramApproximationwithEntropy...... 26 1.3 Markov Chains and the n-gramModels............ 29 1.4Words.............................. 39 1.5 Word Boundaries via Dynamic Programming and Maximum Likelihood . ................... 45 1.6MachineTranslationviaBayes’Theorem........... 48 1.7Exercises............................ 51 2 Music and Piecewise Gaussian Models 61 2.1 Basics III: Gaussian Distributions ............... 62 2.2BasicsIV:FourierAnalysis................... 68 2.3GaussianModelsforSingleMusicalNotes........... 72 2.4 Discontinuities in One-Dimensional Signals . .... 79 2.5TheGeometricModelforNotesviaPoissonProcesses.... 86 2.6RelatedModels......................... 91 2.7Exercises............................ 100 3 Character Recognition and Syntactic Grouping 111 3.1FindingSalientContoursinImages.............. 113 3.2StochasticModelsofContours................. 122 3.3TheMedialAxisforPlanarShapes............... 134 3.4 Gestalt Laws and Grouping Principles . ........... 142 3.5GrammaticalFormalisms.................... 147 3.6Exercises............................ 163 vii i i i i i i i i viii Contents 4 Image Texture, Segmentation and Gibbs Models 173 4.1BasicsIX:GibbsFields..................... 176 4.2 (u + v)-ModelsforImageSegmentation............ 186 4.3SamplingGibbsFields..................... 195 4.4 Deterministic Algorithms to Approximate the Mode of a GibbsField........................... 202 4.5TextureModels......................... 214 4.6 Synthesizing Texture via Exponential Models . ........ 221 4.7TextureSegmentation...................... 228 4.8Exercises............................ 234 5 Faces and Flexible Templates 249 5.1ModelingLightingVariations.................. 253 5.2ModelingGeometricVariationsbyElasticity.......... 259 5.3 Basics XI: Manifolds, Lie Groups, and Lie Algebras . .... 262 5.4ModelingGeometricVariationsbyMetricsonDiff...... 276 5.5ComparingElasticandRiemannianEnergies......... 285 5.6 Empirical Data on Deformations of Faces . ........ 291 5.7TheFullFaceModel...................... 294 5.8Appendix:GeodesicsinDiffandLandmarkSpace...... 301 5.9Exercises............................ 307 6 Natural Scenes and their Multiscale Analysis 317 6.1HighKurtosisintheImageDomain.............. 318 6.2 Scale Invariance in the Discrete and Continuous Setting .... 322 6.3 The Continuous and Discrete Gaussian Pyramids . .... 328 6.4Waveletsandthe“Local”StructureofImages......... 335 6.5DistributionsAreNeeded.................... 348 6.6 Basics XIII: Gaussian Measures on Function Spaces . .... 353 6.7 The Scale-, Rotation- and Translation-Invariant GaussianDistribution...................... 360 6.8ModelII:ImagesMadeUpofIndependentObjects...... 366 6.9FurtherModels......................... 374 6.10 Appendix: A Stability Property of the Discrete GaussianPyramid........................ 377 6.11Exercises............................ 379 Bibliography 387 i i i i i i i i Preface TheNatureofThisBook This book is an introduction to the ideas of Ulf Grenander and the group he has led at Brown University for analyzing all types of signals that the world presents to us. But it is not a basic introductory graduate text that systematically expounds some area of mathematics, as one would do in a graduate course in mathematics. Nor is our goal to explain how to develop all the basic tools for analyzing speech signals or images, as one would do in a graduate course in signal processing or computer vision. In our view, what makes applied mathematics distinctive is that one starts with a collection of problems from some area of science and then seeks the appropriate mathematics for clarifying the experimental data and the under- lying processes producing this data. One needs mathematical tools and, almost always, computational tools as well. The practitioner finds himself or herself en- gaged in a lifelong dialog: seeking models, testing them against data, identifying what is missing in the model, and refining the model. In some situations, the challenge is to find the right mathematical tools; in others, it is to find ways of efficiently calculating the consequences of the model, as well as finding the right parameters in the model. This book, then, seeks to actively bring the reader into such a dialog. To do this, each chapter begins with some example, a class of signals whose variability and patterns we seek to model. Typically, this endeavor calls for some mathematical tools. We put these in sections labeled Basics I, II, etc., and in these sections we put on our mathematical hats and simply develop the math. In most cases, we give the basic definitions, state some results, and prove a few of them. If the reader has not seen this set of mathematical ideas at all, it is probably essential to do some background reading elsewhere for a full exposition of the topic. But if the reader has some background, this section should serve as a refresher, highlighting the definitions and results that we need. Then we compare the model with the data. Usually, new twists emerge. At some point, questions of computability emerge. Here we also have special sections labeled Algorithm I, II, etc. Each chapter ends with a set of exercises. Both of the authors have taught some or part of this material multiple times and have drawn on these classes. In fact, the book is based on lectures delivered in 1998 at the Institut Henri Poincar´e (IHP) by the senior author and the notes made at the time by the junior author. ix i i i i i i i i x Preface The chapters are ordered by the complexity of the signal being studied. In Chapter 1, we study text strings: discrete-valued functions in discrete time. In Chapter 2, we look at music, a real-valued function of continuous time. In Chap- ter 3, we examine character recognition, this being mostly about curvilinear struc- tures in the domain of a function of two variables (namely, an image I(x, y)). In Chapter 4, we study the decomposition of an image into regions with distinct colors and textures, a fully two-dimensional aspect of images. In Chapter 5, we deal with faces, where diffeomorphisms of planar domains is the new player. In Chapter 6, we examine scaling effects present in natural images caused by their statistical self-similarity. For lack of time and space, we have not gone on to the logical next topic: signals with three-dimensional domains, such as various med- ical scans, or the reconstruction of three-dimensional scenes from one or more two-dimensional images