Smoothing and Weighted Average Techniques

Shreyan Ganguly, Jeffrey Gory, Meng Li, Matt Pratola, and Angela Dean

October 28, 2014

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 1 / 30 General Overview Non-Stationarity

Second order stationarity implies that the covariance structure between two distinct points in the spatial domain is only a function of the distance between them Under the assumption of non-stationarity, this is not the case One method of estimating the covariance structure of a non-stationary spatial process is with smoothing or weighted average techniques

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 2 / 30 General Overview Notable Papers

Fuentes (2001) represented a non-stationary spatial process as a weighted average of orthogonal local stationary processes Fuentes and Smith (2001) extended this idea to a continuous convolution of local stationary processes Kim, Mallick, and Holmes (2005) proposed a similar approach with piecewise Gaussian processes that allows for sharp transitions in the covariance structure Nott and Dunsmuir (2002) used a weighted average of stationary processes to describe the conditional behaviour of a process given the values of the process at a set of monitoring sites

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 3 / 30 Fuentes (2001) High Frequency Kriging Procedure

Suppose Z(·) is a non-stationary process observed on a given spatial region D

The region D is divided into k disjoint subregions Si , i = 1, 2,..., k such that

k [ D = Si i=1 Here k can be considered to be a variable valid for estimation

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 4 / 30 Fuentes (2001) Representation of Z

Z is represented as a weighted average of orthogonal local stationary processes Zi k X Z(s) = Zi (s)wi (s) i=1

Each wi (·) is a weight function (e.g. the inverse of the distance between s and Si ) Each Zi is a stationary process with spatial covariance representing the spatial structure of Z in the subregion Si The non-stationary covariance of Z is defined as k X cov(Z(s), Z(t)) = wi (s)wi (t)cov(Zi (s), Zi (t)) i=1

where cov(Zi (s), Zi (t)) = Kθi (s − t), Kθi (·) is Matern SSES Reading Group Smoothing and Weighted Averages October 28, 2014 5 / 30 Fuentes (2001) Algorithm to Find k

Start with small number of subregions (i.e. a small value of k)

Increase k by dividing each subregion Si in half Repeat this until we have less than 36 observations in the subregions or until the AIC suggests no significant improvement in the estimation of θi for i = 1,..., k by increasing the number of subregions k

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 6 / 30 Fuentes (2001) More Details on Finding k

If data are in a regular grid, the Whittle (1954) approximation to the local stationary functions is used to calculate the global likelihood to obtain the AIC value If the data are not in a regular grid, then the method proposed by Kitanidis (1993) and by Mardia and Marshall (1984) is used to efficiently maximize the global likelihood to obtain the AIC The minimum number of observations in a subregion is restricted to 36 as it is not recommended to calculate the periodogram when the number of sample points available to compute the sample covariance is less than 36 (see Haas 1990, Journel and Huijbregts 1978, SAS/STAT Technical Report 1996)

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 7 / 30 Fuentes (2001) Estimation of Covariance Parameters

Recall that the non-stationary covariance of Z is represented as

k X cov(Z(s), Z(t)) = wi (s)wi (t)cov(Zi (s), Zi (t)) i=1

Assuming cov(Zi (s), Zi (t)) = Kθi (s − t) is a Matern covariance structure we have

k X cov(Z(s), Z(t)) = wi (s)wi (t)Kθi (s − t) i=1

We need to estimate θi for each Si

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 8 / 30 Fuentes (2001) Estimation of Covariance Parameters (cont.)

Using a spectral approach assume the spectral function

2 (−νi −d/2) fi (ω) = φi (|ω| )

T for Zi for each subgrid Si so that θi = (φi , νi )

Let Gi be the generalized covariance corresponding to fi ˆ φi andν ˆi are obtained using a weighted non-linear least squares technique in the spectral domain With these estimates we get the estimated generalized covariance

k ˆ X ˆ G(Z(s), Z(t)) = wi (s)wi (t)Gi (Zi (s), Zi (t)) i=1

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 9 / 30 Fuentes and Smith (2001) Continuous Case

Spatially Weighted Average of Local Stationary Processes:

k X Z(s) = Zi (s)wi (s) i=1

0 where wi is the weight function and Zi s are orthogonal stationary processes with Zi (s) ⊥ Zj (t), for any i 6= j Continuous Convolution of Local Stationary Processes: Z Z(s) = K(s − u)Zθ(u)(s)du D

where K is kernel function and Zθ(u) u ∈ D, is a family of independent stationary Gaussian processes indexed by θ, which is assumed to vary smoothly across space

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 10 / 30 Fuentes and Smith (2001) Kernel Smoothing Method

Stationary Gaussian processes can be represented in the form Z Z(s) = K(s − u)X (u)du D where K(·) is some kernel function and X (·) is a Gaussian white noise process This can be extended to non-stationary processes

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 11 / 30 Fuentes and Smith (2001) Kernel Smoothing Models

Model 1: Process Convolution Model Z Z(s) = Ks (u)X (u)du (1) D

where Ks depends on the location s Model 2: Continuous Convolution of Local Stationary Processes Z Z(s) = K(s − u)Zθ(u)(s)du (2) D

where Zθ(u)(s), s ∈ D is a family of independent stationary Gaussian processes indexed by θ(u)

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 12 / 30 Fuentes and Smith (2001) Kernel Smoothing Models (cont.)

Generally, Model (1) and (2) are not equivalent

Since Zθ(u) is a family of independent stationary Gaussian process, it can be written in the form, Z Zθ(u)(s) = Ku(x − s)Xu(x)dx R2

where Ku is a kernel for each location u, and Xu is an independent white noise process for each location u

Only in the case that the process Xu is common across all u (instead of being an independent white noise process for each location u) can Model (1) and (2) be expressed interchangeably

However, Zθ(u)(s), s ∈ D are correlated through kernel smoothing

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 13 / 30 Fuentes and Smith (2001) Covariance Structure of Non-Stationary Process Z

Let Zθ(u) be a stationary Gaussian process with covariance matrix Cθ(u), that is,

cov(Zθ(u)(x1), Zθ(u)(x2)) = Cθ(u)(x1 − x2)

The covariance between locations x1 and x2 of the non-stationary process Z is a convolution of the local covariances Cθ(u)(x1 − x2), Z C(x1, x2; θ) = K(x1 − u)K(x2 − u)Cθ(u)(x1 − x2)du D Assume that θ(u) is a continuous function of u In implementation, the integration can be approximated by Riemann sum or Monto Carlo integration.

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 14 / 30 Fuentes and Smith (2001) Hierarchical Bayesian Model

Z Z(s) = K(s − u)Zθ(u)(s)du D

Stage 1: Conditional on model parameters {θ(u), u ∈ D}, Z is Gaussian Stage 2: θ(u) = a + ri + cj + θ(u), where the process θ(u) is a stationary spatial process with mean zero and its covariance structure can be modeled by some well-known covariance function with unknown parameters Another parameter that needs to be estimated is the bandwidth h that defines the kernel function K

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 15 / 30 Kim, Mallick, and Holmes (2005) Piecewise Gaussian Processes

Much like Fuentes (2001), partition region into several subregions and fit a stationary process within each subregion Adapted to deal with sharp transitions in covariance structure, so processes assumed independent across subregions The resulting non-stationary model is not smooth One possible application is modeling soil permeability Voronoi tessellation (Green and Sibson 1978) used to determine number of subregions and their centers Fit model within a Bayesian framework

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 16 / 30 Kim, Mallick, and Holmes (2005) Voronoi Tessellation

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 17 / 30 Nott and Dunsmuir (2002) General Idea

Measurements of a spatial process are available at a set of monitoring sites An empirical covariance matrix can be estimated from these measurements Want to extend this covariance matrix to a covariance function over the entire region of interest Do this with a weighted average of local stationary processes

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 18 / 30 Nott and Dunsmuir (2002) Illustration

Image from Haas 1990

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 19 / 30 Nott and Dunsmuir (2002) Notation

Z = {Z(s), s ∈ Rd } is the spatial process of interest s1,..., sn are the sites at which Z is observed T Γ is the covariance matrix for (Z(s1),..., Z(sn))

Wi (s), i = 1,..., I are a collection of independent, zero-mean, stationary processes which describe the behavior of Z locally about a collection z1,..., zI of I (unmonitored) locations d Ri (h), h ∈ R is the covariance function corresponding to Wi (s) n Ci = [Ri (sj − sk )]j,k=1 is the covariance matrix of T Wi = (Wi (s1),..., Wi (sn)) T ci (s) is the vector (Ri (s − s1),..., Ri (s − sn)) of cross-covariances between Wi (s) and Wi

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 20 / 30 Nott and Dunsmuir (2002)

Representation of Wi (s)

We can represent Wi (s) as T −1 Wi (s) = ci (s) Ci Wi + δi (s)

where δi (s) is a zero-mean, non-stationary random field with covariance function T −1 Rδi (s, t) = Ri (t − s) − ci (s) Ci ci (t) T −1 Note that ci (s) Ci w is the simple kriging predictor of Wi (s) given Wi = w

δi (s) describes the residual variation, so δi (s) = 0 when s ∈ {s1,..., sn}

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 21 / 30 Nott and Dunsmuir (2002) Representation of Non-Stationary Process

Define a non-stationary process W ∗(s) by

∗ X X 1/2 W (s) = νi (s)µi (s) + νi (s) δi (s) i i

Each νi (s) is a weight function

Each δi (s) is a residual process

Each µi (s) is a mean process given by T −1 ∗ µi (s) = ci (s) Ci W where W ∗ is a zero-mean nx1 random vector with covariance matrix Γ uncorrelated with each δi (s)

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 22 / 30 Nott and Dunsmuir (2002) Covariance of Non-Stationary Process

With this representation, W ∗(s) has covariance function

∗ X T −1 −1 R (s, t) = νi (s)νj (t)ci (s) Ci ΓCj cj (t) i,j X 1/2 1/2 + νi (s) νi (t) Rδi (s, t) i We use a covariance function of this form to estimate the covariance of Z(s)

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 23 / 30 Nott and Dunsmuir (2002) Covariance of Non-Stationary Process (cont.)

Changing notation we can write

I ∗ ∗ X Cov(W (s), W (t)) = Σ0(s, t) + wi (s)wi (t)Cθi (s − t) i=1

Σ0(s, t) is a function of the empirical covariance matrix Γ and the local stationary models Wi (s) ∗ ∗ The Wi (s) are computed so that Cov(W (sk ), W (sl )) = Γkl The 2nd term is similar to the model proposed by Fuentes (2001) Weighted average of residual covariance functions Condition on observed values at the monitoring sites

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 24 / 30 Nott and Dunsmuir (2002) Choice of Weight Functions

The weight functions νi (s) should have the following properties d νi (s) ≥ 0 ∀i = 1,..., I and ∀s ∈ R P i νi (s) = 1 argmax(νi (s)) = zi ∀i = 1,..., I

νi (s) decays smoothly to zero as ||s − zi || −→ ∞

One choice of νi (s) is

fη(s−zi ) νi (s) = P j fη(s−zj )

where fη(·) is a kernel function given by ||t||2 fη(t) = exp(− η ) and η is a smoothing parameter

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 25 / 30 Nott and Dunsmuir (2002) Estimation of Γ

Can estimate covariance matrix Γ by averaging over time

M 1 X Γˆ = (z − z¯)(z − z¯)T nxn M i i i=1 Could alternatively use the Loader & Switzer (1992) empirical Bayes shrinkage estimator ˆ ˆ ˆ ΓLS = λΓ + (1 − λ)C

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 26 / 30 Nott and Dunsmuir (2002)

Estimation of the Ri (h)

For each i = 1,..., I fit a variogram within a window centered on the point zi

z1,..., zI could be on a grid covering the region of interest Might instead be based on knowledge of non-stationarity Recommend at least 10 sites in each window Related to moving window kriging (Haas 1990,1995) ˆ ˆ ˆ One can then estimatec ˆi (s),c ˆj (t), Ci , Cj , and Rδi (s, t) and ultimately compute Rˆ∗(s, t)

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 27 / 30 Nott and Dunsmuir (2002) Advantages of this Method

Produces a valid non-negative definite non-stationary covariance function ”Computationally attractive,” especially in comparison to deformation techniques Can be extended to multivariate processes

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 28 / 30 References

Fuentes, M. (2001). “A high frequency kriging approach for non-stationary environmental processes,” Environmetrics, 12(5), 469-483. Fuentes, M. and Smith, R.L. (2001). “A new class of nonstationary spatial models,” Technical report, North Carolina State University, Institute of Statistics Mimeo Series #2534, Raleigh, NC. Haas, T.C. (1990). “Lognormal and Moving Window Methods of Estimating Acid Deposition,” Journal of the American Statistical Association, 85, 950-963.

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 29 / 30 References (cont.)

Kim, H.M., Mallick, B.K., and Holmes, C.C. (2005). “Analyzing nonstationary spatial data using piecewise Gaussian processes,” Journal of the American Statistical Association, 100, 653-658. Nott, D.J. and Dunsmuir, W.T.M. (2002). “Estimation of nonstationary spatial covariance structure,” Biometrika, 89, 819-829. Sampson, P.D. (2010). ”Constructions for Nonstationary Spatial Processes,” in Handbook of Spatial Statistics. Boca Raton, FL: CRC Press, 119-130.

SSES Reading Group Smoothing and Weighted Averages October 28, 2014 30 / 30