Lecture 5: Intrinsic GMRFs Gaussian Markov random fields David Bolin Chalmers University of Technology February 2, 2015 Intrinsic GMRFs (IGMRF) Intrinsic GMRFs are • improper, i.e., they have precision matrices not of full rank. • Often used as prior distributions in various applications. Of particular importance are IGMRFs that are invariant to any trend that is a polynomial of the locations of the nodes up to a specific order. Today we will define IGMRFs and look at some popular constructions: • Random walks models on the line • ICAR models on regular lattices • Besag models on irregular graphs Introduction David Bolin Improper GMRF Definition Let Q be an n × n SPSD matrix with rank n − k > 0. Then T x = (x1; : : : ; xn) is an improper GMRF of rank n − k with parameters (µ; Q), if its density is −(n−k) ∗ 1=2 1 T π(x) = (2π) 2 (jQj ) exp − (x − µ) Q(x − µ) : (1) 2 Further, x is an improper GMRF wrt to the labelled graph G = (V; E), where Qij 6= 0 () fi; jg 2 E for all i 6= j: j · j∗ denote the generalised determinant: product of the non-zero eigenvalues. Definition of IGMRFs — Improper GMRFs David Bolin Comments • We will continue denoting (µ; Q) “mean” and “precision”, even if these quantities formally do not exist. • The precision matrix has at least one zero eigenvalue. • An improper GMRF is always proper on an appropriate subspace. • We define sampling from an IGMRF as sampling from the proper density on the appropriate subspace. • The eigenvectors corresponding to the zero eigenvalues define directions that the GMRF ’does not care about’ • This is the important property: We can use this indifference! • If we use an improper GMRF as a prior, the posterior is typically proper. Definition of IGMRFs — Improper GMRFs David Bolin Example: GMRFs under linear constraints Consider a GMRF x under the hard contraint Ax = a. ∗ n We can define the density π (x) = π(xjAx = a) for any x 2 R . The null space of the precision of π∗(x) is spanned by the columns of AT . Take x = x~ + x^, where x~ is in the null space of the precision matrix x^ is in the orthogonal subspace. We then have π∗(x) = π∗(x^) = π(xjAx = a) Thus, π∗ is invariant to the addition of any x~ from the space spanned by the rows of A. Definition of IGMRFs — Improper GMRFs David Bolin Markov properties The Markov properties are interpreted as those obtained from the limit of a proper density. Let the columns of AT span the null space of Q Q(γ) = Q + γAT A: Each element in Q(γ) tends to the corresponding one in Q as γ ! 0. 1 X E(x j x ) = µ − Q (x − µ ) i −i i Q ij j j ii j∼i is interpreted as γ ! 0. Definition of IGMRFs — Improper GMRFs David Bolin Polynomial IGMRFs The order of the IGMRF is the rank deficiency of the precision matrix. Polynomial IGMRFs We use the term polynomial IGMRF of order k if the field is invariant to the addition of a polynomial of order < k. An intrinsic GMRF of first order is an improper GMRF of rank n − 1 where Q1 = 0. P The condition Q1 = 0 means that j Qij = 0; i = 1; : : : ; n Definition of IGMRFs — Improper GMRFs David Bolin Local behaviour Consider a first order IGMRF. With µ = 0, we have 1 X E(x j x ) = − Q x i −i Q ij j ii j:j∼i and X − Qij=Qii = 1 j:j∼i • The conditional mean of xi is a weighted mean of its neighbours. • No shrinking towards an overall level.This is important! • We can concentrate on the deviation from any overall level without having to specify what it is • This is a perfect property for a prior • Many IGMRFs are constructed such that the deviation from the overall level is a smooth curve in time or a smooth surface in space. Definition of IGMRFs — Improper GMRFs David Bolin Building IGMRFs on the line Definition (Forward difference) Define the first-order forward difference of a function f(·) as ∆f(z) = f(z + 1) − f(z): Higher-order forward differences are defined recursively: ∆kf(z) = ∆∆k−1f(z) so ∆2f(z) = f(z + 2) − 2f(z + 1) + f(z) (2) and in general for k = 1; 2;:::, k X k ∆kf(z) = (−1)k (−1)j f(z + j): j j=0 IGMRFs on the line — Forward differences David Bolin IGMRFs of first order on the line Location of the node i is i (think “time”). Assume independent increments iid −1 ∆xi ∼ N (0; κ ); i = 1; : : : ; n − 1; so (3) −1 xj − xi ∼ N (0; (j − i)κ ) for i < j: (4) The density for x is n−1 ! κ X π(x j κ) / κ(n−1)=2 exp − (∆x )2 (5) 2 i i=1 n−1 ! κ X = κ(n−1)=2 exp − (x − x )2 ; (6) 2 i+1 i i=1 or κ π(x) / κ(n−1)=2 exp − xT Rx (7) 2 IGMRFs on the line — On the line with regular locations David Bolin The structure matrix The structure matrix 0 1 −1 1 B−1 2 −1 C B C B −1 2 −1 C B C B .. .. .. C R = B . C : (8) B C B −1 2 −1 C B C @ −1 2 −1A −1 1 • The n − 1 independent increments ensure that the rank of Q is n − 1. • Denote this model by RW1(κ) or short RW1. IGMRFs on the line — On the line with regular locations David Bolin Properties Full conditionals 1 x j x ; κ ∼ N ( (x + x ); 1=(2κ)); 1 < i < n; (9) i −i 2 i−1 i+1 If the intersection between fi; : : : ; jg and fk; : : : ; lg is empty for i < j and k < l, then Cov(xj − xi; xl − xk) = 0: (10) Properties coincide with those of a Wiener process. IGMRFs on the line — On the line with regular locations David Bolin Example Samples with n = 99 and κ = 1 by conditioning on the constraint P xi = 0. IGMRFs on the line — On the line with regular locations David Bolin Example Marginal variances Var(xi) for i = 1; : : : ; n. IGMRFs on the line — On the line with regular locations David Bolin Example Corr(xn=2; xi) for i = 1; : : : ; n IGMRFs on the line — On the line with regular locations David Bolin An alternative first order IGMRF Let n ! κ X π(x) / κ(n−1)=2 exp − (x − x)2 (11) 2 i i=1 where x is the empirical mean of x. Assume n is even and ( 0; 1 ≤ i ≤ n=2 xi = : (12) 1; n=2 < i ≤ n Thus x is locally constant with two levels. Evaluate the density at this configuration under the RW1 model and the alternative (11), we obtain κ κ κ(n−1)=2 exp − and κ(n−1)=2 exp −n (13) 2 8 The log ratio of the densities is then of order O(n). IGMRFs on the line — On the line with regular locations David Bolin The local structure of the RW1 model • The RW1 model only penalises the local deviation from a constant level. • The alternative penalises the global deviation from a constant level. This local behaviour is advantageous in applications if the mean level of x is approximately or locally constant. A similar argument also apply for polynomial IGMRFs of higher order constructed using forward differences of order k as independent Gaussian increments. IGMRFs on the line — On the line with regular locations David Bolin The RW2 model for regular locations Let si = i for i = 1; : : : ; n, with a constant distance between consecutive nodes. Use the second order increments 2 iid −1 ∆ xi ∼ N (0; κ ) (14) for i = 1; : : : ; n − 2, to define the joint density of x n−2 ! κ X π(x) / κ(n−2)=2 exp − (x − 2x + x )2 (15) 2 i i+1 i+2 i=1 κ = κ(n−2)=2 exp − xT Rx (16) 2 IGMRFs on the line — RW2 model for regular locations David Bolin The structure matrix The structure matrix is now 0 1 −2 1 1 B−2 5 −4 1 C B C B 1 −4 6 −4 1 C B C B 1 −4 6 −4 1 C B C B .. .. .. .. .. C R = B . C : B C B 1 −4 6 −4 1 C B C B 1 −4 6 −4 1 C B C @ 1 −4 5 −2A 1 −2 1 IGMRFs on the line — RW2 model for regular locations David Bolin Remarks The conditional mean and precision is 4 1 E(x j x ; κ) = (x + x ) − (x + x ); i −i 6 i+1 i−1 6 i+2 i−2 Prec(xi j x−i; κ) = 6κ, respectively for 2 < i < n − 2. • Verify directly that QS1 = 0 and that the rank of Q is n − 2. • IGMRF of second order: invariant to the adding line to x. • Known as the second order random walk model, denoted by RW2(κ) or simply RW2 model. IGMRFs on the line — RW2 model for regular locations David Bolin Example Samples with n = 99 and κ = 1 by conditioning on the constraints. IGMRFs on the line — RW2 model for regular locations David Bolin Marginal variances Var(xi) for i = 1; : : : ; n. IGMRFs on the line — RW2 model for regular locations David Bolin Corr(xn=2; xi) for i = 1; : : : ; n IGMRFs on the line — RW2 model for regular locations David Bolin Smoothness and limiting properties The forward difference of kth order as an approximation to the kth derivative of f(z), f(z + h) − f(z) f 0(z) = lim h!0 h We can think of smoothness of the in the limit as the distance between the points goes to zero relative to the domain size In the limit we are asking that the kth derivative isn’t too large.