Pattern Theory and Its Applications

Rudolf Kulhavý Institute of Information Theory and Automation Academy of Sciences of the Czech Republic, Prague Ulf Grenander

ƒ A Swedish mathematician, since 1966 with Division of Applied Mathematics at Brown University ƒ Highly influential research in time series analysis, probability on algebraic structures, pattern recognition, and image analysis. ƒ The founder of Pattern Theory 1976, 1978, 1981: Lectures in Pattern Theory 1993: General Pattern Theory 1996: Elements of Pattern Theory 2007: Pattern Theory

2 See initial chapters for good introduction

Further referred to as [PT07]

Ulf Grenander Michael Miller

3 Agenda

1. Intuitive concepts 2. General pattern theory 3. Illustrative applications 4. Takeaway points

4 Why should one pay attention?

ƒ Pattern Theory is a mathematical representation of objects with large and incompressible complexity in terms of atom-like blocks and bonds between them, similar to chemical structures.1 ƒ Pattern Theory gives an algebraic framework for describing patterns as structures regulated by rules, both local and global. Probability measures are sometimes superimposed on the image algebras to account for variability of the patterns.2

1 Yuri Tarnopolsky, Molecules and Thoughts, 2003 2 Ulf Grenander and Michael I. Miller, Representations of knowledge in complex systems, 1994. 5 Board game analogy

ƒ Game board

6 Board game analogy

ƒ Game board ƒ Game stones

7 Board game analogy

ƒ Game board ƒ Game stones ƒ Game situations

8 Board game analogy

ƒ Game board ƒ Game stones ƒ Game situations ƒ Game rules Which situations - are allowed? - are likely?

9 Sample "games"

ƒ Molecular geometry ƒ Chemical structures ƒ Biological shapes and anatomies ƒ Pixelated images and digital videos ƒ Visual scene representation ƒ Formal languages and grammars ƒ Economies and markets ƒ Human organizations ƒ Social networks ƒ Protein folding ƒ Historic events ƒ Thoughts, ideas, theories … 10 Agenda

1. Intuitive concepts 2. General pattern theory 3. Illustrative applications 4. Takeaway points

11 Configurations

ƒ Configurations

ƒ Connector graph

ƒ Generators

ƒ Bonds

12 Typical connector types Source: [PT07]

13 Strict regularity

ƒ Bond function

ƒ Regular configuration

ƒ Space of regular configurations

Single graph

Multiple graphs

14 Example: Mitochondria shape Source: [PT07] ƒ Generators

ƒ Bonds

ƒ Regularity conditions

15 Weak regularity

ƒ Probability of configuration

Matching bonds

ƒ Normalization What’s the motivation behind this form?

16 Markov random field

ƒ A set of random variables, X, indexed over is a Markov random field on a graph if 1. for all 2. for each

Neighborhood

Two sites are neighbors if connected by an edge in the connector graph.

17 Gibbs distribution

ƒ Gibbs distribution

Normalization ƒ Energy function constant

A system A set of sites forms of cliques a clique if every two ƒ Clique potentials distinct sites in the set are neighbors.

Only sites

in the clique 18 Hammersley-Clifford theorem Equivalent ƒ Markov random field properties for all

for each

ƒ Gibbs distribution

19 Canonical representation

Pairwise generator cliques are enough

Source: [PT07] 20 Weak regularity

ƒ Gibbs distribution

Pairwise potentials

Energy function ƒ Normalization

21 Board game analogy

ƒ Connector graph ~ Game board ƒ Generators and bonds ~ Game stones ƒ Regularity conditions ~ Game rules ƒ Regular configurations ~ Permitted game situations ~ Likely game situations

22 Images and deformed images

ƒ Identification rule The concept of image formalizes the idea of an observable ƒ Image algebra (quotient space)

ƒ Image (equiv. class)

Think of 3D scene ƒ Deformed image by means of 2D images

23 Patterns

ƒ Similarity group

ƒ Pattern family (quotient space)

"Square" pattern ƒ Patterns are thought of as images modulo the invariances represented by the similarity group

"Rectangle" pattern 24 Bayesian inference

ƒ Bayes rule

ƒ Numerical implementation – Markov chain Monte Carlo methods y Gibbs sampler y Langevin sampler y Jump-diffusion dynamics – Mean field approximation – Renormalization group

25 Agenda

1. Intuitive concepts 2. General pattern theory 3. Illustrative applications 4. Takeaway points

26 Ising model

A row of binary elements ƒ Energy of configuration

+1 for positive spin –1 for negative spin

Neighbors External magnetic field Configur- >0 if attractive ation <0 if repulsive Property of material, >0

ƒ Probability of configuration ConfigurationsConfigurations thatthat requirerequire highhigh energyenergy areare lessless likelylikely

Temperature Normalizing constant Universal constant 27 Pixelated image model

ƒ Black-and-white image s, t : sites s~t : neighbors

gs : original image pixels Is : noisy image pixels ƒ Energy of configuration given a noisy image

Neighbors Smoothing factor, >0 +1 for black –1 for white ƒ Probability of configuration

28 Market network model

Actors I Consumer (C) I N Producer (P) P I Labor (L) L P C Investor (I) L Nature (N) P C N Exchanges I I P Consumers’ goods C or goods of the 1st order P L L Producers’ goods L or goods of higher order L or factors of production

29 Sample markets

30 Phrase structure grammars Source: [PT07]

31 MRI images

Whole brain maps in the macaque monkey

Transformed templates Target sections

Template

Source: Ulf Grenander and Michael I. Miller, Computational anatomy: an emerging discipline, 1996 32 Agenda

1. Intuitive concepts 2. General pattern theory 3. Illustrative applications 4. Takeaway points

33 When to think of Pattern Theory

ƒ Representations of complex systems composed of atomic blocks interacting locally ƒ Representations that accommodate structure and variability simultaneously ƒ Typical applications include – Speech recognition – Computational linguistics – Image analysis – Computer vision – Target recognition

34 Further reading

ƒ Julian Besag, Spatial interaction and the statistical analysis of lattice systems. Journal of the Royal Statistical Society, Series B, Vol. 36, No. 2, pp. 192-236, 1974. ƒ David Griffeath, Introduction to random fields, in J.G. Kemeny, J. L. Snell and A. W. Knapp, Denumerable Markov Chains, 2nd ed., Springer-Verlag, New York, 1976. ƒ Ross Kindermann and J. Laurie Snell, Markov Random Fields and Their Applications. American Mathematical Society, Providence, RI, 1980. ƒ Stuart Geman and Donald Geman, Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 6, No. 6, pp. 721-741, 1984. ƒ Ulf Grenander and Michael I. Miller, Representations of knowledge in complex systems. Journal of the Royal Statistical Society, Series B, Vol. 56, No. 4, pp. 549-603, 1994.

35 Further reading (contd.)

ƒ Gerhard Winkler, Image Analysis, Random Fields and Dynamic Monte Carlo Methods: A Mathematical Introduction, Springer-Verlag, Berlin, 1995. ƒ Ulf Grenander, Geometries of knowledge, Proceedings of the National Academy of Sciences of the USA, vol. 94, pp. 783- 789, 1997. ƒ Anuj Srivastava et al., Monte Carlo techniques for automated pattern recognition. In A. Doucet, N. de Freitas and N. Gordon (eds.), Sequential Monte Carlo Methods in Practice, Springer-Verlag, Berlin, 2001. ƒ David Mumford, Pattern theory: the mathematics of perception. Proceedings of the International Congress of Mathematicians, Beijing, 2002. ƒ Ulf Grenander and Michael I. Miller, Pattern Theory from Representation to Inference, Oxford University Press, 2007. ƒ Yuri Tarnopolsky, Introduction to Pattern Chemistry, URL: http://spirospero.net, 2009.

36