<<

Spatial GMRF Q Model INLA Extensions References

Gaussian Markov Random Fields

Johan Lindstrom¨ 1

1Centre for Mathematical Sciences Lund University

Pan-American Advanced Study Institute B ´uzios June 18, 2014

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 1/33 Spatial GMRF Q Model INLA Extensions References Interpolation Big N Alternatives

Spatial interpolation —

Given observations at some locations, Y(si), i = 1 ... n we want to make statements about the value at unobserved location(s), X(s).

In the simplest case we assume a Gaussian model for the data

Y μy Σyy Σyx ∼ N , ⊤ , X μx Σyx Σxx with some parametric form for the matrix and mean

Y ∼ N (μ(θ), Σ(θ)) .

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 2/33 Spatial GMRF Q Model INLA Extensions References Interpolation Big N Alternatives The “Big N” problem

The log-likelihood becomes

1 1 ⊤ l (θ|Y) = − log |Σ(θ)|− Y − μ(θ) Σ(θ)−1 Y − μ(θ) . 2 2 Given (estimated) parameters, predictions at the unobserved locations are given by

θ = + −1 E X Y, μx ΣxyΣyy (Y − μy ). The “Big N” problem

Given N observations: ◮ The has O N2 unique elements. ◮ Computations scale as O N3 (due to |Σ| and Σ−1).

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 3/33 Spatial GMRF Q Model INLA Extensions References Interpolation Big N Alternatives Getting around “Big N”

Spectral representation (Whittle, 1954; Fuentes, 2007) Uses discrete Fourier transforms; limited to regular lattices. Covariance tapering (Furrer et al., 2006; Kaufman et al., 2008) Set small values in the covariance matrix to zeros. Likelihood approximation Various statistical or numerical approximation methods ◮ By sequential matrix approx. (Stein et al., 2004) ◮ Block composite likelihoods (Eidsvik et al., 2014) ◮ Krylov based numerical approx. (Stein et al., 2013; Aune et al., 2014) Low rank approximations Exact computations on a simplified model of reduced rank/size ◮ Predictive processes (Banerjee et al., 2008; Eidsvik et al., 2012) ◮ Fixed rank kriging (Cressie and Johannesson, 2008) ◮ Process convolution or kernel methods (Higdon, 2001)

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 4/33 Spatial GMRF Q Model INLA Extensions References Markov Computations Gaussian Markov random fields (Rue and Held, 2005)

Let the neighbours Ni to a point si be the points {sj, j ∈ Ni} that are “close” to si. Gaussian Markov random field (GMRF)

A Gaussian random field x ∼ N (μ, Σ) that satisfies

p xi {xj : j = i} = p xi {xj : j ∈ Ni} is a Gaussian Markov random field.

The simplest example of a GMRF is the AR(1)-process 2 xt = axt−1 + εt, εt ∼ N(0, σ ) and independent.

xt−2 xt−1 xt xt+1 xt+2

Nt

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 5/33 Spatial GMRF Q Model INLA Extensions References Markov Precision Computations Good neighbours

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 6/33 Spatial GMRF Q Model INLA Extensions References Markov Precision Computations −1 Let me introduce: The precision matrix Q = Ë

Using the precision matrix the model becomes

X ∼ N μ, Q−1 cf. X ∼ N (μ, Σ) With conditional expectation

= −1 E (X|Y) μx − Qxx Qxy (Y − μy).

instead of

= + −1 E (X|Y) μx Σxy Σyy (Y − μy).

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 7/33 Spatial GMRF Q Model INLA Extensions References Markov Precision Computations

The conditional expectation for a single location is

j=i Qij(xj − μj) E x x , j = i = μ − i j i Q ii 1 = μi −  Qij(xj − μj) + Qij(xj − μj) Qii j∈N j∈{N/,i}  i i 

If Qij = 0 for all j ∈/ Ni then

E xi xj, j = i = E xi xj, j ∈ Ni . The precision matrix is sparse

Elements in the precision matrix of a Gaussin Markov random field are non-zero only for neighbours and diagonal elements.

j ∈{/ i, Ni} ⇐⇒ Qij = 0.

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 8/33 Spatial GMRF Q Model INLA Extensions References Markov Precision Computations Computational details

If Q is a sparse matrix then (under mild conditions) the Cholesky factorisation Q = R⊤R will also be sparse. ◮ Simulation of X ∈ N μ, Q−1 . X = μ + R−1E, E ∈ N (0, I)

◮ Conditional expectations

= −1 −⊤ E (X|Y) μx − Rxx Rxx Qxy(Y − μy) ◮ Computing the

1 1 ⊤ log |Q| = log R R = log |R| = log Rii 2 2 i ◮ Never compute R−1, use back substitution for triangular systems instead.

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 9/33 Spatial GMRF Q Model INLA Extensions References Mat´ern SPDE Basis functions Lattice How to create Q?

The Mat´ern covariance family (Mat´ern, 1960)

The covariance between two points at distance h is

2 = σ Ò Rd r(h) (κ h) KÒ (κ h), h ∈ 2Ò−1Γ(ν)

Fields with Matern´ are solutions to a Stochastic Partial Differential Equation (SPDE) (Whittle, 1954, 1963),

a/2 κ2 − Δ x(s) = W(s). = ∂2 = + Here W(s) is , Δ i 2 , and α ν d/2. ∂si

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 10/33 Spatial GMRF Q Model INLA Extensions References Mat´ern SPDE Basis functions Lattice Does the Mat´ern covariance produce Markov fields?

The Matern´ covariance has wave number spectrum

1 a/2 2 = R(k) ∝ 2 cf. κ − Δ x(s) W(s). 2 + a (κ k )

Spectral density for Markov fields

According to Rozanov (1977) a stationary field is Markov if and only if the spectral density is a reciprocal of a polynomial.

For the SPDE this implies α ∈ Z (or ν ∈ Z for R2).

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 11/33 Spatial GMRF Q Model INLA Extensions References Mat´ern SPDE Basis functions Lattice

Basic idea Construct a discrete approximation of the continious field using basis functions, {ψk}, and weights, {wk},

x(s) = ψk(s)wk k

2 a/2 Find the distribution of wk by solving κ − Δ x(s) = W(s)

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 12/33 Spatial GMRF Q Model INLA Extensions References Mat´ern SPDE Basis functions Lattice Solving the SPDE

A stochastic weak solution to the SPDE is given by

2 a/2 =D φk, κ − Δ x [φk, W]k=1,...,n k=1,...,n

for each set of test functions {φk}

Replacing x with k ψkwk gives 2 a/2 =D φi, κ − Δ ψj w [φk, W]k i,j

Study the case α = 2 and φi = ψi (Galerkin)

2 D κ ψi, ψj + ψi, −Δψj  w = [ψk, W]    C G  N(0,C)

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 13/33 Spatial GMRF Q Model INLA Extensions References Mat´ern SPDE Basis functions Lattice

Solution to the SPDE A weak solution to the SPDE

κ2 − Δ x(s) = W(s). is given by

2 x(s) = ψk(s)wk where κ C + G w ∼ N (0, C) k The precision of the weights, w, is

−1 2 ⊤ −1 2 V (w) = Q2 = κ C + G C κ C + G

2 Q1 =κ C + G 2 ⊤ −1 2 Q2 = κ C + G C κ C + G

= 2 + ⊤ −1 −1 2 + = a Qa κ C G C Q −2C κ C G , α 3, 4, 5,...

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 14/33 Spatial GMRF Q Model INLA Extensions References Mat´ern SPDE Basis functions Lattice What’s a good basis?

Several possible basis functions exist: Harmonic functions gives a spectral representation (Thon et al., 2012; Sigrist et al., 2012). Eigenfunctions of the covariance matrix give Karhunen-Loeve.´ Piecewise linear basis gives (almost) a GMRF (Lindgren et al., 2011). Waveletts also gives a GMRF (Bolin and Lindgren, 2013).

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 15/33 Spatial GMRF Q Model INLA Extensions References Mat´ern SPDE Basis functions Lattice But it’s not a Markov field! — Yet

Using a piecewise linear basis only neighbouring basis functions overlap, so both

Gij = ψi, −Δψj and Cij = ψi, ψj are sparse. However, C−1 is not sparse. GMRF approximation

To obtain sparse precision matrices we replace the C-matrix with a diagonal matrix C with elements Ci,i = ψi(s) ds The resulting approximation error is small (Bolin and Lindgren, 2013; Simpson et al., 2010)

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 16/33 Spatial GMRF Q Model INLA Extensions References Mat´ern SPDE Basis functions Lattice Regular lattice in R2

Solving the SPDE gives a GMRF with precision elements

Order α = 1 (ν = 0):

−1 κ2  1  + −1 4 −1 −1    

≈−D Order α = 2 (ν = 1):

1 −1  2 −8 2  κ4  1  + 2κ2 −1 4 −1 + 1 −8 20 −8 1   −1  2 −8 2         1  ≈−D   2 ≈D Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 17/33 Spatial GMRF Q Model INLA Extensions References Observations E(Y|X) Hierarchical Posterior Observations— The A-matrix The field is created as a weighted sum of basis functions.

N

x(s) = ψk(s) wk, k=1 The locations of the basis functions do not need to match observation locations. Observations

y(s) = x(s) + ε

= ψk(s)wk + ε k

We introduce a sparse matrix

Ai = ψ1(si) ψN(si) linking the field to the observation. Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 18/33 Spatial GMRF Q Model INLA Extensions References Observations E(Y|X) Hierarchical Posterior Conditional expectation

Given a matrix A with rows

Ai = ψ1(si) ψN(si) we can write the observation equation on matrix form as

Y|w ∈ N Aw, Q−1 w ∈ N , Q−1

e μ Kriging with GMRF

= + −1 ⊤ E (w|y) μ Qw|y A Qe (y − Aμ) −1 = −1 = + ⊤ V (w|y) Qw|y Q A Qe A

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 19/33 Spatial GMRF Q Model INLA Extensions References Observations E(Y|X) Hierarchical Posterior

Bayesian hierarchical modelling using GMRF

Data model, p (y|x, θ): Describing how observations arise assuming a known latent field x. Latent model, p (x|θ): Describing how the latent field behaves.

X = Aw + Bβ w ∼ N 0, Q−1(θ) Parameters, p (θ): Describing our, sometimes vauge, prior knowledge of the parameters.

For INLA we require that

p (y|x, θ) = p (yi|xi, θ) i

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 20/33 Spatial GMRF Q Model INLA Extensions References Observations E(Y|X) Hierarchical Posterior Inference

Given a Bayesian hierarchical model we are interested in Posteriors for the parameters

p (θ|y) ∝ p (y|θ) p (θ)

Posteriors for the latent field

p (x|y) ∝ p (x|y, θ) p (θ|y) dθ

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 21/33 Spatial GMRF Q Model INLA Extensions References Observations E(Y|X) Hierarchical Posterior Computing the posterior — A “trick”

Conditional distributions provide the equality

p (y|x, θ) p (x|θ) = p (y, x|θ) = p (x|y, θ) p (y|θ)

This gives

p (y|x, θ) p (x|θ) p (θ|y) ∝ p (θ) for any x. p (x|y, θ)

p(y|θ)

For non-Gaussian observations we need a good approximation of

p (x|y, θ)

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 22/33 Spatial GMRF Q Model INLA Extensions References Observations E(Y|X) Hierarchical Posterior Approximating the posterior

For non-Gaussian observations we have that

log p (x|y, θ) = log p (y|x, θ) + log p (x|θ) + const.

Using a second order Taylor approximation of

f(x) = log p (y|x, θ) around x0

we can obtain a Gaussian approximation pG(x|y, θ), with

−1 θ ′′ + ′ ′′ Ex0 (x|y, ) ≈ Q − diag (f (x0)) Qμ f (x0) − f (x0)x0 −1 θ ′′ Vx0 (x|y, ) ≈ Q − diag (f (x0))

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 23/33 Spatial GMRF Q Model INLA Extensions References Integrated Nested Laplace Approximation – INLA

To evaluate p (θ|y) using the Taylor/Laplace approximation do: 1. For a given θ find the mode

x0 = argmax p (x|y, θ) x 2. Compute the Taylor expansion of f(x) around x0 3. The approximation of p (θ|y) is p (y|x , θ) p (x |θ) p (θ) p (θ|y) ∝ 0 0 pG (x0|y, θ) The (approximate) maximum likelihood estimate is

θML ≈ argmax p (θ|y) θ Use numerical integration over θ to obtain posteriors for [x|y]

p (xi|y) = p (xi|θ, y) p (θ|y) dθ ≈ pG (xi|θk, y) p (θk|y) k Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 24/33 Spatial GMRF Q Model INLA Extensions References INLA (Rue et al., 2009)

◮ Posteriors computed include

◮ p(θi|y) ◮ p(θ|y) ◮ p(xi|y) ◮ Some linear combinations of the field. ◮ The full joint posterior p(x|y) is not computed. ◮ Errors due to the Taylor expansion and numerical integration are usually smaller than the MCMC errors. ◮ R-package: http://www.r-inla.org ◮ The package is fast, but memory intensive for large problems. ◮ See also Lindgren and Rue (2013) for applications to the SPDE-model.

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 25/33 Spatial GMRF Q Model INLA Extensions References Global Data General SPDEs Global Temperature Data

January 2003 July 2003

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 26/33 Spatial GMRF Q Model INLA Extensions References Global Data General SPDEs Global Temperature Data

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 27/33 Spatial GMRF Q Model INLA Extensions References Global Data General SPDEs GMRFs based on SPDEs (Lindgren et al., 2011)

GMRF representations of SPDEs can be constructed for to oscillating, anisotropic, non-stationary, non-separable spatio-temporal, and multivariate fields on .

(κ2 − Δ)(τ x(s)) = W(s), s ∈ Rd

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 28/33 Spatial GMRF Q Model INLA Extensions References Global Data General SPDEs GMRFs based on SPDEs (Lindgren et al., 2011)

GMRF representations of SPDEs can be constructed for to oscillating, anisotropic, non-stationary, non-separable spatio-temporal, and multivariate fields on manifolds.

(κ2 − Δ)(τ x(s)) = W(s), s ∈ Ω

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 28/33 Spatial GMRF Q Model INLA Extensions References Global Data General SPDEs GMRFs based on SPDEs (Lindgren et al., 2011)

GMRF representations of SPDEs can be constructed for to oscillating, anisotropic, non-stationary, non-separable spatio-temporal, and multivariate fields on manifolds.

(κ2 ei Ôθ − Δ)(τ x(s)) = W(s), s ∈ Ω

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 28/33 Spatial GMRF Q Model INLA Extensions References Global Data General SPDEs GMRFs based on SPDEs (Lindgren et al., 2011)

GMRF representations of SPDEs can be constructed for to oscillating, anisotropic, non-stationary, non-separable spatio-temporal, and multivariate fields on manifolds.

2 (κs + ∇ ms − ∇ Ms∇)(τsx(s)) = W(s), s ∈ Ω

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 28/33 Spatial GMRF Q Model INLA Extensions References Global Data General SPDEs GMRFs based on SPDEs (Lindgren et al., 2011)

GMRF representations of SPDEs can be constructed for to oscillating, anisotropic, non-stationary, non-separable spatio-temporal, and multivariate fields on manifolds.

∂ + 2 + = R ∂t κs,t ∇ ms,t − ∇ Ms,t∇ (τs,tx(s, t)) E(s, t), (s, t) ∈ Ω ×

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 28/33 Spatial GMRF Q Model INLA Extensions References Global Data General SPDEs

Questions?

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 29/33 Spatial GMRF Q Model INLA Extensions References Bibliography I

Aune, E., Simpson, D. P., and Eidsvik, J. (2014), “Parameter estimation in high dimensional Gaussian distributions,” Stat. Comput., 24, 247–263. Banerjee, S., Gelfand, A. E., Finley, A. O., and Sang, H. (2008), “Gaussian predictive process models for large spatial data sets,” J. R. Stat. Soc. B, 70, 825–848. Bolin, D. and Lindgren, F. (2013), “A comparison between Markov approximations and other methods for large spatial data sets,” Comput. Stat. Data Anal., 61, 7–21. Cressie, N. and Johannesson, G. (2008), “Fixed rank kriging for very large spatial data sets,” J. R. Stat. Soc. B, 70, 209–226. Eidsvik, J., Finley, A. O., Banerjee, S., and Rue, H. (2012), “Approximate Bayesian inference for large spatial datasets using predictive process models,” Comput. Stat. Data Anal., 56, 1362–138. Eidsvik, J., Shaby, B. A., Reich, B. J., Wheeler, M., and Niemi, J. (2014), “Estimation and Prediction in Spatial Models With Block Composite Likelihoods,” J. Comput. Graphical Stat., 23, 295–315.

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 30/33 Spatial GMRF Q Model INLA Extensions References Bibliography II

Fuentes, M. (2007), “Approximate Likelihood for Large Irregularly Spaced Spatial Data,” J. Am. Stat. Assoc., 102, 321–331. Furrer, R., Genton, M. G., and Nychka, D. (2006), “Covariance Tapering for Interpolation of Large Spatial Datasets,” J. Comput. Graphical Stat., 15, 502–523. Higdon, D. (2001), “Space and space time modeling using process convolutions,” Tech. rep., Institute of and Decision Sciences, Duke University, Durham, NC, USA. Kaufman, C. G., Schervish, M. J., and Nychka, D. W. (2008), “Covariance Tapering for Likelihood-Based Estimation in Large Spatial Data Sets,” J. Am. Stat. Assoc., 103, 1545–1555. Lindgren, F. and Rue, H. (2013), “Bayesian Spatial and Spatio-temporal Modelling with R-INLA,” Submitted to J. Stat. Softw. Lindgren, F., Rue, H., and Lindstrom,¨ J. (2011), “An explicit link between Gaussian fields and Gaussian Markov random fields: the stochastic partial differential equation approach,” J. R. Stat. Soc. B, 73, 423–498.

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 31/33 Spatial GMRF Q Model INLA Extensions References Bibliography III

Matern,´ B. (1960), “Spatial variation. Stochastic models and their application to some problems in forest surveys and other sampling investigations.” Ph.D. thesis, Stockholm University, Stockholm, Sweden. Rozanov, Y. A. (1977), “Markov random fields and stochastic partial differential equations,” Math. USSR Sb., 32, 515–534. Rue, H. and Held, L. (2005), Gaussian Markov Random Fields; Theory and Applications, vol. 104 of Monographs on Statistics and Applied , Chapman & Hall/CRC. Rue, H., Martino, S., and Chopin, N. (2009), “Approximate Bayesian inference for hierarchical Gaussian Markov random field models,” J. R. Stat. Soc. B, 71, 1–35. Sigrist, F., K¨unsch, H. R., and Stahel, W. A. (2012), “Stochastic partial differential equation based modeling of large space-time data sets,” Tech. Rep. arXiv:1204.6118v5, arXiv preprint. Simpson, D., Lindgren, F., and Rue, H. (2010), “In order to make spatial statistics computationally feasible, we need to forget about the ,” Environmetrics, 23, 65–74.

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 32/33 Spatial GMRF Q Model INLA Extensions References Bibliography IV

Stein, M. L., Chen, J., and Anitescu, M. (2013), “Stochastic approximation of score functions for Gaussian processes,” Ann. Stat., 7, 1162–1191. Stein, M. L., Chi, Z., and Welty, L. J. (2004), “Approximating likelihoods for large spatial data sets,” J. R. Stat. Soc. B, 66, 275–296. Thon, K., Rue, H., Skrøvseth, S. O., and Godtliebsen, F. (2012), “Bayesian multiscale analysis of images modeled as Gaussian Markov random fields,” Comput. Stat. Data Anal., 56, 49–61. Whittle, P. (1954), “On Stationary Processes in the Plane,” Biometrika, 41, 434–449. — (1963), “Stochastic processes in several ,” Bull. Int. Stat. Inst., 40, 974–994.

Johan Lindstr¨om - [email protected] Gaussian Markov Random Fields 33/33