Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Umberto Triacca Dipartimento di Ingegneria e Scienze dell'Informazione e Matematica Universit`adell'Aquila, [email protected] Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Introduction The autocovariance function plays a key role in the construction of the DGP's model of our time series. Thus an important question is: Is it possible to obtain a 'good' estimate of the autocovariance function? Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Introduction This question is not trivial since we observe only one realization of the process. We cannot, for example, observe a second realization of Italian annual GDP for the period from 2001 to 2011. In general, with a single realization we are unable to estimate a moment function of a stochastic process. Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The Mean Function of a stochastic process First, we consider the mean function. Definition. Let fxt ; t 2 Zg be a stochastic process such that Var(xt ) < 1 8t 2 Z. The function µx : Z ! R defined by µx (t) = E(xt ) is called Mean Function of the stochastic process fxt ; t 2 Zg. Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The Mean Function of a weakly stationary process Let fxt ; t 2 Zg be a weakly stationary stochastic process. The mean function is given by µx (t) = µ 8t 2 Z Thus, for a weakly stationary process, the mean function is the constant function which takes always the same value µ. Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Estimation of Mean Function How do we estimate µ? Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Estimation of Mean Function If were possible to obtain m independent realizations n (j) (j)o x1 ; :::; xT j = 1; :::; m we could estimate the mean of fxt ; t 2 Zg by averaging over the realizations m (δ) 1 X (j) µ~ = x ; δ 2 f1; 2; :::; T g m m δ j=1 Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Estimation of Mean Function Since 2 γ (0) E µ~(δ) − µ = x m m we have (δ) 2 limm!1E µ~m − µ = 0 (δ) The estimatorµ ~m converges in quadratic mean to µ as the number of realizations increase. (δ) This implies thatµ ~m is consistent for µ. Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Estimation of Mean Function Unfortunately, as we have already seen, it is impossible to obtain multiple realizations. We have a single finite realization x1; x2;:::; xT Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Estimation of Mean and Autocovariance Function Then, the only way to perform the average is along the time axis. + The mean and covariance functions can be estimated by their corresponding time averages: T 1 X x¯ = x T T t t=1 and T 1 X γ^ (k) = (x − x¯ )(x − x¯ ) for k = 0; 1;:::; T − 1 x T t T t−k T t=k+1 Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Estimation of Mean and Autocovariance Function Now the question is: are these estimators `good'? Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Estimation of Mean and Autocovariance Function The answer is yes, if the stationary process is ergodic Ergodic comes from Greek.From ergon `energy, work'and hodos `way, path'. Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic processes What is an ergodic process? Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic processes Definition. A stationary stochastic process is said ergodic, with respect to a given population moment, if the sample (or time) moment for a single finite realization of length T converges in quadratic mean to the population moment as T increases to 1. Remark. Note that we cannot simply refer to a process as ergodic. Ergodicity must be related directly to the particular population moment, e.g. mean value, covariance, etc. Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic processes In particular, a stationary process xt with mean µ is mean-ergodic if h 2i limT !1E (¯xT − µ) = 0 and it is covariance-ergodic if 2 T !23 1 X lim E (x − x¯ )(x − x¯ ) − γ (k) = 0 T !1 4 T t T t−k T x 5 t=k+1 Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process ergodic processes It is important to underline that covariance-ergodicity implies mean-ergodicity, but not the reverse. Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process ergodic processes A simple example of mean and covariance-ergodic process is the process ut ∼ i:i:d:N(0; 1) Problem: Show that the process ut ∼ i:i:d:N(0; 1) is mean-ergodic We observe that the process ut has no memory, in the sense that the value of the process at time t is uncorrelated with all past values up to time t − 1: Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process ergodic processes It is important to note that not all stationary processes are ergodic. Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic processes Consider the stationary process fxt ; t 2 Zg, with xt = A 8t 2 Z, where A is a random variable with mean 3 and variance 7. This process is not mean-ergodic since the sample meanx ¯T does not converge to µ as T ! 1 Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic processes In fact we have 2 limT !1E (¯xT − 3) = 7 6= 0 Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The meaning of the ergodicity The ergodicity is a matter of information contained in a single realization of a long duration of the process. Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The meaning of the ergodicity If the process is not too persistent (ergodicity), so that each element of the realization x1; x2;:::; xT will contain some information not available from the other elements, then a single realization of a long duration will be sufficient to obtain a good estimate of its moments. Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The meaning of the ergodicity We have seen that: 1 the process ut ∼ i:i:d:N(0; 1) is mean-ergodic; 2 the process xt = A, where A is a random variable with mean 3 and variance 7 is not mean-ergodic. Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The meaning of the ergodicity Consider the process ut ∼ i:i:d:N(0; 1) and let u1; u2;:::; uT be a time series from this process. Each element of the time series u1; u2;:::; uT will contain some information not available from the other elements. There are new information contained in the new observations. Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The meaning of the ergodicity Now, we consider the process xt = A, where A is a random variable with mean 3 and variance 7. We note that a time series from this process will has the form a; a; :::; a where with a we have denoted a realization of the random variable A. Thus in the time series x1 = a; x2 = a; :::; xT = a there is the same infomation contained in the first observation x1. There aren't new information contained in the new observations. Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process The meaning of the ergodicity The process ut has no memory, in the sense that the value of the process at time t is uncorrelated with all past values up to time t − 1: The process xt = A, instead, is too persistent. Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic Theorems Now, we present some theoretical results, called ergodic theorems. These theorems give some necessary and/or sufficient condition for mean-ergodicity of a stationary process. Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic Theorems Theorem. (Slutsky's Theorem) A stationary process xt with mean µ and autocovariance function γx (k) is mean-ergodic iff T −1 1 X lim γ (k) = 0 T !1 T x k=0 Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic Theorems We note that T −1 1 X γ(k) = Cov(¯x ; x ): T T T k=0 Thus T −1 1 X lim γ(k) = 0 () lim Cov(¯x ; x ) = 0 T !1 T T !1 T T k=0 We have that a stationary process xt is mean-ergodic if and only if as the sample size T is increased there is less and less correlation (covariance) between the sample meanx ¯T and the last observation xT . Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Ergodic Theorems Corollary 1 (Sufficient condition for mean-ergodicity). Let xt be a stationary process with mean µ and autocovariance function γx (k).