Introduction to Time Series Analysis. Lecture 19. 1. Review: Spectral density estimation, sample autocovariance. 2. The periodogram and sample autocovariance.

3. Asymptotics of the periodogram.

1 Estimating the Spectrum: Outline

We have seen that the spectral density gives an alternative view of • stationary time series.

Given a realization x ,...,x of a time series, how can we estimate • 1 n the spectral density?

One approach: replace γ( ) in the definition • ∞ f(ν)= γ(h)e−2πiνh, h=−∞ with the sample autocovariance γˆ( ). Another approach, called the periodogram: compute I(ν), the squared • modulus of the discrete Fourier transform (at frequencies ν = k/n).

2 Estimating the spectrum: Outline

These two approaches are identical at the Fourier frequencies ν = k/n. • The asymptotic expectation of the periodogram I(ν) is f(ν). We can • derive some asymptotic properties, and hence do hypothesis testing.

Unfortunately, the asymptotic variance of I(ν) is constant. • It is not a consistent estimator of f(ν).

3 Review: Spectral density estimation

If a time series X has autocovariance γ satisfying { t} ∞ γ(h) < , then we define its spectral density as h=−∞ | | ∞ ∞ f(ν)= γ(h)e−2πiνh h=−∞ for <ν< . −∞ ∞

4 Review: Sample autocovariance

Idea: use the sample autocovariance γˆ( ), defined by n−|h| 1 γˆ(h)= (x x¯)(x x¯), for n

5 Discrete Fourier transform

For a sequence (x1,...,xn), define the discrete Fourier transform (DFT) as

(X(ν0),X(ν1),...,X(νn−1)), where

1 n X(ν )= x e−2πiνkt, k √n t t=1 and ν = k/n (for k = 0, 1,...,n 1) are called the Fourier frequencies. k − (Think of ν : k = 0,...,n 1 as the discrete version of the frequency { k − } range ν [0, 1].) ∈ First, let’s show that we can view the DFT as a representation of x in a different basis, the Fourier basis.

6 Discrete Fourier transform

Consider the space Cn of vectors of n complex numbers, with inner product a,b = a∗b, where a∗ is the complex conjugate transpose of the vector a Cn. ∈ Suppose that a set φ : j = 0, 1,...,n 1 of n vectors in Cn are { j − } orthonormal: 1 if j = k, φj,φk = 0 otherwise.  Then these φ span the vector space Cn, and so for any vector x, we can { j}  write x in terms of this new orthonormal basis,

n−1

x = φ , x φ . (picture) j j j=0

7 Discrete Fourier transform

Consider the following set of n vectors in Cn:

1 ′ e = e2πiνj , e2πi2νj ,...,e2πinνj : j = 0,...,n 1 . j √n − It is easy to check that these vectors are orthonormal: 1 n 1 n t e , e = e2πit(νk−νj ) = e2πi(k−j)/n j k n n t=1 t=1 1 if j = k, = 2πi(k−j)/n n 1 2πi(k−j)/n 1−(e )  e − otherwise  n 1−e2πi(k j)/n 1 if j = k, = 0 otherwise,   8 Discrete Fourier transform

n t where we have used the fact that Sn = t=1 α satisfies αS = S + αn+1 α and so S = α(1 αn)/(1 α) for α = 1. n n − n − − So we can represent the real vector x =(x ,...,x )′ Cn in terms of this 1 n ∈ orthonormal basis,

n−1 n−1 x = e , x e = X(ν )e . j j j j j=0 j=0 That is, the vector of discrete Fourier transform coefficients

(X(ν0),...,X(νn−1)) is the representation of x in the Fourier basis.

9 Discrete Fourier transform

An alternative way to represent the DFT is by separately considering the real and imaginary parts,

1 n X(ν )= e , x = e−2πitνj x j j √n t t=1 1 n 1 n = cos(2πtν )x i sin(2πtν )x √n j t − √n j t t=1 t=1 = X (ν ) iX (ν ), c j − s j where this defines the sine and cosine transforms, Xs and Xc, of x.

10 Periodogram

The periodogram is defined as

I(ν )= X(ν ) 2 j | j | 2 1 n = e−2πitνj x n t t=1 2 2 = X (νj )+ X (νj ). c s 1 n X (ν )= cos(2πtν )x , c j √n j t t=1 1 n X (ν )= sin(2πtν )x . s j √n j t t=1

11 Periodogram

Since I(ν )= X(ν ) 2 for one of the Fourier frequencies ν = j/n (for j | j | j j = 0, 1,...,n 1), the orthonormality of the e implies that we can write − j ∗ n−1 n−1 x∗x = X(ν )e X(ν )e  j j   j j  j=0 j=0 n−1 n−1  = X(ν ) 2 = I(ν ). | j | j j=0 j=0 For x¯ = 0, we can write this as

1 n 1 n−1 σˆ2 = x2 = I(ν ). x n t n j t=1 j=0

12 Periodogram

This is the discrete analog of the identity

1/2 2 σx = γx(0) = fx(ν) dν. −1/2

(Think of I(νj) as the discrete version of f(ν) at the frequency νj = j/n, and think of (1/n) as the discrete version of dν.) νj ν

13 Estimating the spectrum: Periodogram

Why is the periodogram at a Fourier frequency (that is, ν = νj ) the same as computing f(ν) from the sample autocovariance? Almost the same—they are not the same at ν = 0 when x¯ = 0. 0 But if either x¯ = 0, or we consider a Fourier frequency νj with j 1,...,n 1 , ... ∈ { − }

14 Estimating the spectrum: Periodogram

2 2 1 n 1 n I(ν )= e−2πitνj x = e−2πitνj (x x¯) j n t n t − t=1 t=1 n n 1 = e−2πitνj (x x¯) e2πitνj (x x¯) n t − t − t=1 t=1 1 1 n− = e−2πi(s−t)νj (x x¯)(x x¯)= γˆ(h)e−2πihνj , n s − t − s,t h=−n+1 n where the fact that ν = 0 implies e−2πitνj = 0 (we showed this j t=1 when we were verifying the orthonormality of the Fourier basis) has allowed us to subtract the sample mean in that case.

15 Asymptotic properties of the periodogram

We want to understand the asymptotic behavior of the periodogram I(ν) at a particular frequency ν, as n increases. We’ll see that its expectation converges to f(ν).

We’ll start with a simple example: Suppose that X1,...,Xn are i.i.d. N(0, σ2) (Gaussian white noise). From the definitions,

1 n 1 n X (ν )= cos(2πtν )x ,X (ν )= sin(2πtν )x , c j √n j t s j √n j t t=1 t=1 we have that Xc(νj) and Xs(νj ) are normal, with

EXc(νj )= EXs(νj ) = 0.

16 Asymptotic properties of the periodogram

Also,

σ2 n Var(X (ν )) = cos2(2πtν ) c j n j t=1 σ2 n σ2 = (cos(4πtν )+1)= . 2n j 2 t=1 2 Similarly, Var(Xs(νj )) = σ /2.

17 Asymptotic properties of the periodogram

Also,

σ2 n Cov(X (ν ),X (ν )) = cos(2πtν ) sin(2πtν ) c j s j n j j t=1 σ2 n = sin(4πtν ) = 0, 2n j t=1 Cov(Xc(νj),Xc(νk))=0

Cov(Xs(νj ),Xs(νk))=0

Cov(Xc(νj ),Xs(νk))=0. for any j = k.

18 Asymptotic properties of the periodogram

2 That is, if X1,...,Xn are i.i.d. N(0, σ ) 2 (Gaussian white noise; f(ν)= σ ), then the Xc(νj) and Xs(νj ) are all i.i.d. N(0, σ2/2). Thus, 2 2 I(ν )= X2(ν )+ X2(ν ) χ2. σ2 j σ2 c j s j ∼ 2 So for the case of Gaussian white noise, the periodogram has a chi-squared distribution that depends on the variance σ2 (which, in this case, is the spectral density).

19 Asymptotic properties of the periodogram

Under more general conditions (e.g., normal X , or linear process X { t} { t} with rapidly decaying ACF), the Xc(νj ), Xs(νj ) are all asymptotically independent and N(0, f(νj)/2). Consider a frequency ν. For a given value of n, let νˆ(n) be the closest Fourier frequency (that is, νˆ(n) = j/n for a value of j that minimizes ν j/n ). As n increases, νˆ(n) ν, and (under the same conditions that | − | → ensure the asymptotic normality and independence of the sine/cosine (n) transforms), f(ˆν ) f(ν). (picture) → In that case, we have

2 2 d I(ˆν(n))= X2(ˆν(n))+ X2(ˆν(n)) χ2. f(ν) f(ν) c s → 2

20 Asymptotic properties of the periodogram

Thus, f(ν) 2 EI(ˆν(n))= E X2(ˆν(n))+ X2(ˆν(n)) 2 f(ν) c s f(ν) E(Z2 + Z2)= f(ν), → 2 1 2 where Z1,Z2 are independent N(0, 1). Thus, the periodogram is asymptotically unbiased.

21 Introduction to Time Series Analysis. Lecture 19. 1. Review: Spectral density estimation, sample autocovariance. 2. The periodogram and sample autocovariance.

3. Asymptotics of the periodogram.

22