Lesson 6: Estimation of the Function of a

Umberto Triacca

Dipartimento di Ingegneria e Scienze dell’Informazione e Matematica Universit`adell’Aquila, [email protected]

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Introduction

The autocovariance function plays a key role in the construction of the DGP’s model of our .

Thus an important question is:

Is it possible to obtain a ’good’ estimate of the autocovariance function?

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Introduction

This question is not trivial since we observe only one realization of the process.

We cannot, for example, observe a second realization of Italian annual GDP for the period from 2001 to 2011.

In general, with a single realization we are unable to estimate a moment function of a .

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The Mean Function of a stochastic process

First, we consider the mean function. Definition. Let {xt ; t ∈ Z} be a stochastic process such that Var(xt ) < ∞ ∀t ∈ Z. The function

µx : Z → R

defined by µx (t) = E(xt )

is called Mean Function of the stochastic process {xt ; t ∈ Z}.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The Mean Function of a weakly stationary process

Let {xt ; t ∈ Z} be a weakly stationary stochastic process. The mean function is given by

µx (t) = µ ∀t ∈ Z

Thus, for a weakly stationary process, the mean function is the constant function which takes always the same value µ.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Estimation of Mean Function

How do we estimate µ?

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Estimation of Mean Function

If were possible to obtain m independent realizations

n (j) (j)o x1 , ..., xT j = 1, ..., m

we could estimate the mean of {xt ; t ∈ Z} by averaging over the realizations

m (δ) 1 X (j) µ˜ = x , δ ∈ {1, 2, ..., T } m m δ j=1

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Estimation of Mean Function

Since  2 γ (0) E µ˜(δ) − µ = x m m

we have    (δ) 2 limm→∞E µ˜m − µ = 0

(δ) The estimatorµ ˜m converges in quadratic mean to µ as the number of realizations increase.

(δ) This implies thatµ ˜m is consistent for µ.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Estimation of Mean Function

Unfortunately, as we have already seen, it is impossible to obtain multiple realizations. We have a single finite realization

x1, x2,..., xT

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Estimation of Mean and Autocovariance Function

Then, the only way to perform the average is along the time axis.

⇓

The mean and functions can be estimated by their corresponding time averages:

T 1 X x¯ = x T T t t=1 and

T 1 X γˆ (k) = (x − x¯ )(x − x¯ ) for k = 0, 1,..., T − 1 x T t T t−k T t=k+1

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Estimation of Mean and Autocovariance Function

Now the question is: are these estimators ‘good’?

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Estimation of Mean and Autocovariance Function

The answer is yes, if the stationary process is ergodic

Ergodic comes from Greek.From ergon ‘energy, work’and hodos ‘way, path’.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic processes

What is an ergodic process?

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic processes

Definition. A stationary stochastic process is said ergodic, with respect to a given population moment, if the sample (or time) moment for a single finite realization of length T converges in quadratic mean to the population moment as T increases to ∞.

Remark. Note that we cannot simply refer to a process as ergodic. Ergodicity must be related directly to the particular population moment, e.g. mean value, covariance, etc.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic processes

In particular, a stationary process xt with mean µ is mean-ergodic if

h 2i limT →∞E (¯xT − µ) = 0

and it is covariance-ergodic if

 T !2 1 X lim E (x − x¯ )(x − x¯ ) − γ (k) = 0 T →∞  T t T t−k T x  t=k+1

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process ergodic processes

It is important to underline that covariance-ergodicity implies mean-ergodicity, but not the reverse.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process ergodic processes

A simple example of mean and covariance-ergodic process is the process ut ∼ i.i.d.N(0, 1)

Problem: Show that the process ut ∼ i.i.d.N(0, 1) is mean-ergodic

We observe that the process ut has no memory, in the sense that the value of the process at time t is uncorrelated with all past values up to time t − 1.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process ergodic processes

It is important to note that not all stationary processes are ergodic.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic processes

Consider the stationary process {xt ; t ∈ Z}, with xt = A ∀t ∈ Z, where A is a random variable with mean 3 and variance 7.

This process is not mean-ergodic since the sample meanx ¯T does not converge to µ as T → ∞

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic processes

In fact we have

 2 limT →∞E (¯xT − 3) = 7 6= 0

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The meaning of the ergodicity

The ergodicity is a matter of information contained in a single realization of a long duration of the process.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The meaning of the ergodicity

If the process is not too persistent (ergodicity), so that each element of the realization x1, x2,..., xT will contain some information not available from the other elements, then a single realization of a long duration will be sufficient to obtain a good estimate of its moments.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The meaning of the ergodicity

We have seen that:

1 the process ut ∼ i.i.d.N(0, 1) is mean-ergodic;

2 the process xt = A, where A is a random variable with mean 3 and variance 7 is not mean-ergodic.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The meaning of the ergodicity

Consider the process ut ∼ i.i.d.N(0, 1) and let

u1, u2,..., uT

be a time series from this process. Each element of the time series u1, u2,..., uT will contain some information not available from the other elements. There are new information contained in the new observations.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The meaning of the ergodicity

Now, we consider the process xt = A, where A is a random variable with mean 3 and variance 7. We note that a time series from this process will has the form

a, a, ..., a

where with a we have denoted a realization of the random variable A. Thus in the time series

x1 = a, x2 = a, ..., xT = a

there is the same infomation contained in the first observation x1. There aren’t new information contained in the new observations.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The meaning of the ergodicity

The process ut has no memory, in the sense that the value of the process at time t is uncorrelated with all past values up to time t − 1. The process xt = A, instead, is too persistent.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic Theorems

Now, we present some theoretical results, called ergodic theorems. These theorems give some necessary and/or sufficient condition for mean-ergodicity of a stationary process.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic Theorems

Theorem. (Slutsky’s Theorem) A stationary process xt with mean µ and autocovariance function γx (k) is mean-ergodic iff

T −1 1 X lim γ (k) = 0 T →∞ T x k=0

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic Theorems

We note that T −1 1 X γ(k) = Cov(¯x , x ). T T T k=0 Thus

T −1 1 X lim γ(k) = 0 ⇐⇒ lim Cov(¯x , x ) = 0 T →∞ T T →∞ T T k=0

We have that a stationary process xt is mean-ergodic if and only if as the sample size T is increased there is less and less correlation (covariance) between the sample meanx ¯T and the last observation xT .

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic Theorems

Corollary 1 (Sufficient condition for mean-ergodicity). Let xt be a stationary process with mean µ and autocovariance function γx (k). If limk→∞γx (k) = 0,

then xt is mean-ergodic.

Intuitively, this result tell us that if any two random variables positioned far apart in the sequence are almost uncorrelated, then some new information can be continually added so that the sample mean will approach the popolation mean.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic Theorems

Corollary 2. (Sufficient condition for mean-ergodicity) Let xt be a stationary process with mean µ and and autocovariance function γx (k). If ∞ X |γx (k)| < ∞, k=0

then xt is mean-ergodic.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic Theorems

If ∞ X |γx (k)| < ∞, k=0 we say that the autocovariance function is absolutely sommable

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodicity under Gaussianity

Let {xt ; t ∈ Z} be a stationary process with mean µ and autocovariance function γx (k). If the process is Gaussian, then 1. the condition of absolute summability of covariance function

∞ X |γx (k)| < ∞ k=0 is sufficient to ensure ergodicity for all moments. 2. the condition limk→∞γx (k) = 0 is necessary and sufficient.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodicity under Gaussianity

Heuristically, a Gaussian stationary process is ergodic if and only if any two random variables positioned far apart in the sequence are almost independently distributed. That is, for sufficiently large k, xt and xt−k are nearly independent.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic processes

Conclusion

For stationary ergodic processes, we do not need to observe separate independent realizations of the process in order to obtain a consistent estimate of its mean value or other moments.

A good estimate of the moments of the process can be obtained considering only one sufficiently long realization of the process.

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process