2. Graphical Models

Undirected graphical models

Factor graphs

Bayesian networks

Conversion between graphical models

Graphical Models 2-1 Graphical models

There are three families of graphical models that are closely related, but suitable for different applications and different distributions:

I Undirected graphical models (also known as Markov Random Fields) I Factor graphs I Bayesian networks we will learn what they are, how they are different and how to switch between them.

consider a over x = (x1, x2, . . . , xn)

µ(x1, x2, . . . , xn)

a is a graph and a set of functions over a subset of random variables which define the probability distribution of interest

Graphical Models 2-2 graphical model is a marriage between and graph theory that allows compact representation and efficient inference, when the probability distribution of interest has special independence and structures for example, consider a random vector x = (x1, x2, x3) and a given distribution µ(x1, x2, x3) we use (with a slight abuse of notations) X µ(x1) , µ(x1, x2, x3) , and x2,x3 X µ(x1, x2) , µ(x1, x2, x3) x3 to denote the first order and the second order marginals respectively we can list all possible independence structures

x1 ⊥ (x2, x3) ↔ µ(x1, x2, x3) = µ(x1)µ(x2, x3) (1)

x1 ⊥ x2 ↔ µ(x1, x2) = µ(x1)µ(x2) (2)

x1 ⊥ x2|x3 ↔ x1–x3–x2 ↔ µ(x1, x2|x3) = µ(x1|x3)µ(x2|x3)(3) and various permutations and combinations of these Graphical Models 2-3 warm-up exercise

I (1) ⇒ (2) proof: (1) µ(x , x ) = P µ(x , x , x ) = P µ(x )µ(x , x ) = µ(x )µ(x ) 1 2 x3 1 2 3 x3 1 2 3 1 2 I (2) 6⇒ (3) counter example: X1 ⊥ X2 and X3 = X1 + X2 I (2) 6⇐ (3) counter example: Z1,Z2,X3 are independent and X1 = X3 + Z1, X2 = X3 + Z2 this hints that different graphical models are required to represent different notions of independence in fact, there are exponentially many possible independencies, resulting in doubly exponentially many possible independence structures in a distribution however, there are only 2n2 undirected graphs (4n2 for directed) hence, graphical models only capture (important) subsets of possible independence structures

Graphical Models 2-4 a probabilistic graphical model is a graph G(V,E) representing a family of probability distributions 1. that share the same factorization of the probability distribution; and 2. that share the same independence structure. I undirected graphical model = Markov (MRF) 1 Y µ(x) = ψ (x ) Z c c c∈C(G) where C(G) is the set of all maximal cliques in the undirected graph G(V,E) I model (FG) 1 Y µ(x) = ψ (x ) Z a ∂a a∈F where F is the set of factor nodes in the undirected G(V,F,E) and ∂a is the set of neighbors of the node a I directed graphical model = (BN) Y µ(x) = µ(xi|xπ(i)) i∈V where π(i) is the set of parent nodes in the directed graph G(V,E) Graphical Models 2-5 warm-up example: Markov Random Fields (MRF) and Factor Graphs (FG) MRF FG independencefactorization

x1 x2

x3 µ(x1, x2, x3) none

ψ(x1, x2)ψ(x1, x3) x2 ⊥ x3|x1

[HW1.1]

ψ(x1, x2)ψ(x3) x3 ⊥ (x1, x2)

ψ(x1)ψ(x2)ψ(x3) all indep.

Graphical Models 2-6 warm-up example: Bayesian Network (BN) of ordering (x1 → x2 → x3) BN independencefactorization

x1 x2 µ(x1)µ(x2|x1)µ(x3|x1, x2) none

x3

µ(x1)µ(x2|x1)µ(x3|x1) x2 ⊥ x3|x1

µ(x1)µ(x2)µ(x3|x1, x2) x1 ⊥ x2

µ(x1)µ(x2|x1)µ(x3|x2) x1 ⊥ x3|x2

µ(x1)µ(x2|x1)µ(x3) x3 ⊥ (x1, x2)

µ(x1)µ(x2)µ(x3) all indep.

Graphical Models 2-7 Family #1: Undirected Pairwise Graphical Models

Graphical Models 2-8 Family #1: Undirected Pairwise Graphical Models (a.k.a. Pairwise MRF)

x3 x2 x10 x x 4 11 x1

x12 x9 x5

x7 x8 x6

G = (V,E), V = [n] , {1, . . . , n}, x = (x1, . . . , xn), xi ∈ X

Graphical Models 2-9 x3 x2 x10 x x 4 11 x1

x12 x9 x5

x7 x8 x6 Undirected pairwise graphical models are specified by

I Graph G = (V,E) I Alphabet X I Compatibility function ψij : X × X → R+, for all (i, j) ∈ E 1 Y µ(x) = ψ (x , x ) Z ij i j (i,j)∈E

I pairwise MRF only allow compatibility functions over two variables Graphical Models 2-10 Undirected Pairwise Graphical Models Alphabet X

I Typically |X | < ∞ I Occationally X = R and 1 Y µ(dx) = ψ (x , x ) dx Z ij i j (i,j)∈E

(all formulae interpreted as densities)

Compatibility function ψij

1 Y µ(x) = ψ (x , x ) Z ij i j (i,j)∈E

Partition function Z plays a crucial role! X Y Z = ψij(xi, xj) x∈X V (i,j)∈E

Graphical Models 2-11 Graph notation 3 2 10 4 11 1

12 9 5

7 8 6 ∂i ≡ {neighborhood of vertex i}, ∂9 = {5, 6, 7} deg(i) = |∂i|, deg(9) = 3

xU ≡ (xi)i∈U , x{1,5} = (x1, x5)

x∂9 = (x5, x6, x7) Complete graph Clique Graphical Models 2-12 Example

x3 x2 x10 x x 4 11 x1

x12 x9 x5

x7 x8 x6 Coloring (e.g. ring tone) Given graph G = (V,E) and a set of colors X = {R, G, B} Find a coloring of the vertices such that no two adjacent vertices have the same color Fundamental question: Chromatic number Graphical Models 2-13 x3 x2 x10 x x 4 11 x1

x12 x9 x5

x7 x8 x6 A (joint) probability of interest is uniform measure over all possible colorings: 1 Y µ(x) = (x 6= x ) Z I i j (i,j)∈E

Z = total number of colorings = coloring similarly, independent set problem [HW1.3, 1.4] Graphical Models 2-14 (General) Undirected Graphical Model

x3 x2 x10 x x 4 11 x1

x12 x9 x5

x7 x8 x6 Undirected graphical models are specified by I Graph G = (V,E) I Alphabet X c I Compatibility function ψc : X → R+, for all maximal cliques c ∈ C 1 Y µ(x) = ψ (x ) Z c c c∈C Graphical Models 2-15 Family #2: Factor Graph Models

Graphical Models 2-16 Family #2: Factor graph models ∂4 b x5 x6 ←variable x ∈ X c i x4

←factor ψa(x1, x5, x6) a | {z } x3 ∂a d x 2 x1

Factor graph G = (V,F,E) I Variable nodes i, j, k, · · · ∈ V I Function nodes a, b, c, · · · ∈ F Variable node xi ∈ X , for all i ∈ V ∂a Function node ψa : X → R+, for all a ∈ F 1 Y µ(x) = ψ (x ) Z a ∂a a∈F Graphical Models 2-17 Factor graph models ∂4 b x5 x6 ←variable x ∈ X c i x4

←factor ψa(x1, x5, x6) a | {z } x3 ∂a d x 2 x1

Factor graph model is specified by I Factor graph G = (V,F,E) I Alphabet X ∂a I Compatibility function ψa : X → R+, for a ∈ F 1 Y µ(x) = ψ (x ) Z a ∂a a∈F P Q Partition function: Z = x∈X V a∈F ψa(x∂a) Graphical Models 2-18 Conversion between factor graphs and pairwise models From pairwise model to factor graph A pairwise model on G(V,E) with alphabet X can be represented by a factor graph G0(V 0,F 0,E0) with V 0 = V , F 0 ' E, |E0| = 2|E|, X 0 = X .

Put a factor node on each edge From factor graph to a general undirected graphical model A factor model on G(V,F,E) with alphabet X can be represented by a 0 0 0 0 0 P 2 0 pairwise model on G (V ,E ) with V = V , E ' a∈F |∂a| , X = X .

A factor node is turned into a clique From factor graph to a pairwise model A factor model on G(V,F,E) can be represented by a pairwise model on 0 0 0 0 0 0 ∆ G (V ,E ) with V = V ∪ F , E = E, X = X , ∆ = maxa∈F deg(a).

A factor node is represented by a large variable node Graphical Models 2-19 Family #3: Bayesian Networks

Graphical Models 2-20 Family #3: Bayesian networks x1 x2 x3

x4 x5

x6 x7 x8 DAG: G = (V,D) Variable nodes V = [n], xi ∈ X , for all i ∈ V Define π(i) ≡ {parents of i} Set of directed edges D Y µ(x) = µi(xi|xπ(i)) i∈V

Graphical Models 2-21 x1 x2 x3

x4 x5

x6 x7 x8 Bayesian network is specified by I directed acyclic graph G = (V,D) I alphabet X π(i) I conditional probability µi(·|·): X × X → R+, for i ∈ V Y µ(x) = µi(xi|xπ(i)) i∈V

I we do not need normalization (1/Z) since X X µi(xi|xπ(i)) = 1 ⇒ µ(x) = 1

xi∈X x∈X V

Graphical Models 2-22 Conversion between Bayes networks and factor graphs from Bayes network to factor graph A Bayes network G = (V,D) with alphabet X can be represented by a factor graph model on G0 = (V 0,F 0,E0) with V 0 = V , |F 0| = |V |, |E0| = |D| + |V |, X 0 = X .

represent by a factor node each conditional probability moralization for conversion from BN to MRF from factor graph to Bayes network A factor model on G = (V,F,E) with alphabet X can be represented by a Bayes network G0 = (V 0,D0) with V 0 = V and X 0 = X .

take a topological ordering, e.g. x1, . . . , xn for each node i, starting from the first node, find a minimal set U ⊆ {1, . . . , i − 1} such that xi is conditionally independent of x{1,...,i−1}\U given xU . (we will learn how to do this) in general the resulting Bayesian network is dense Graphical Models 2-23 Because MRF and BN are incomparable, some independence structure is lost in conversion

µ(x) = ψ(x1, x2)ψ(x1, x3)ψ(x2, x4)ψ(x3, x4) x1 ⊥ x4|(x2, x3) x2 ⊥ x3|(x1, x4) x2 ⊥ x3|(x1, x4)

µ(x) = µ(x2)µ(x3)µ(x1|x2, x3) x2 ⊥ x3 no independence

Graphical Models 2-24 Factor graphs are more ‘fine grained’ than undirected graphical models

1 3 1 3 1 3

2 2 2

ψ(x1, x2, x3) ψ12(x1, x2)ψ23(x2, x3)ψ31(x3, x1) ψ123(x1, x2, x3) all three encodes same independencies, but different factorizations (in particular the degrees of freedom in the compatibility functions are 3|X |2 vs. |X |3) set of independencies represented by MRF is the same as FG but FG can represent a larger set of factorizations

Graphical Models 2-25 undirected graphical models can be represented by factor graphs

I we can go from MRF to FG without losing any information on the independencies implies by the model

Bayesian networks are not compatible with undirected graphical models or factor graphs

I if we go from one model to the other, and then back to the original model, then we will not, in general, get back the same model as we started out with I we lose any information on the independencies implies by the model, when switching from one model to the other

Graphical Models 2-26 Bayes networks with observed variables

V = H ∪ O

Hidden variables: x = (xi)i∈H

Observed variables: y = (yi)i∈O

Y Y µ(x, y) = µ(xi|xπ(i)∩H , yπ(i)∩O) µ(yi|xπ(i)∩H , yπ(i)∩O) i∈H i∈O

Typically interested in µy(x) ≡ µ(x|y) and

arg max µy(x) x

Graphical Models 2-27 Example

Forensic Science [Kadane, Shum, A probabilistic analysis of the Sacco and Vanzetti evidence, 1996] [Taroni et al., Bayesian Networks and Probabilistic Inference in Forensic Science, 2006]

Graphical Models 2-28 Example

diseases

‘soft ORs’

symptoms

Medical Diagnosis [M. Shwe, et al., Methods of Information in Medicine, 1991]

Graphical Models 2-29 Roadmap

Cond. Indep. Factorization Graphical Graph Cond. Indep. µ(x) µ(x) Model G implied by G 1 Q x1–{x2, x3}–x4; Z ψa(x∂a) FG Factor Markov 1 Q x4–{}–x7; Z ψC (xC ) MRF Undirected Markov . Q . ψi(xi|xπ(i)) BN Directed Markov

Any µ(x) can be represented by {FG,MRF,BN} A µ(x) can be represented by multiple {FG,MRF,BN} with multiple graphs (but same µ(x)) We want a ‘simple’ graph representation (sparse, small alphabet size)

I Memory to store the graphical model I Computations for inference µ(x) with some conditional independence structure can be represented by simple {FG,MRF,BN}

Graphical Models 2-30