Introduction to Time Series Analysis. Lecture 18. 1. Review: Spectral density, rational spectra, linear filters.

2. Frequency response of linear filters. 3. Spectral estimation

4. Sample autocovariance

5. Discrete Fourier transform and the periodogram

1 Review: Spectral density

If a time series X has autocovariance γ satisfying { t} ∞ γ(h) < , then we define its spectral density as h=−∞ | | ∞ ∞ f(ν)= γ(h)e−2πiνh h=−∞ for <ν< . We have −∞ ∞ 1/2 γ(h)= e2πiνhf(ν) dν. −1/2

2 Review: Rational spectra

For a linear time series with MA( ) polynomial ψ, ∞ 2 2πiν 2 f(ν)= σw ψ e . If it is an ARMA(p,q), we have

2 θ(e−2πiν ) f(ν)= σ2 w φ (e−2πiν) 2 q −2πiν 2 θ e zj = σ2 q j=1 − , w 2 p −2πiν 2 φp e pj j=1 | − | where z1,...,zq are the zeros (roots of θ(z)) and p1,...,pp are the poles (roots of φ(z)).

3 Review: Time-invariant linear filters

A filter is an operator; given a time series X , it maps to a time series { t} Y . A linear filter satisfies { t} ∞ Yt = at,jXj. j=−∞ time-invariant: at,t−j = ψj : ∞ Yt = ψjXt−j. j=−∞ causal: j < 0 implies ψj = 0. ∞ Yt = ψj Xt−j. j=0

4 Time-invariant linear filters

The operation ∞ ψjXt−j j=−∞ is called the convolution of X with ψ.

5 Time-invariant linear filters

The sequence ψ is also called the impulse response, since the output Y of { t} the linear filter in response to a unit impulse,

1 if t = 0, Xt =  0 otherwise, is  ∞ Yt = ψ(B)Xt = ψj Xt−j = ψt. j=−∞

6 Introduction to Time Series Analysis. Lecture 18. 1. Review: Spectral density, rational spectra, linear filters.

2. Frequency response of linear filters. 3. Spectral estimation

4. Sample autocovariance

5. Discrete Fourier transform and the periodogram

7 Frequency response of a time-invariant linear filter

Suppose that X has spectral density f (ν) and ψ is stable, that is, { t} x ∞ ψ < . Then Y = ψ(B)X has spectral density j=−∞ | j | ∞ t t 2πiν 2 fy(ν)= ψ e fx(ν). The function ν ψ(e2πiν ) (the polynomial ψ(z) evaluated on the unit → circle) is known as the frequency response or transfer function of the linear filter. The squared modulus, ν ψ(e2πiν ) 2 is known as the power transfer → | | function of the filter.

8 Frequency response of a time-invariant linear filter

For stable ψ, Yt = ψ(B)Xt has spectral density

2πiν 2 fy(ν)= ψ e fx(ν). We have seen that a linear process, Yt = ψ(B)Wt, is a special case, since f (ν)= ψ(e2πiν ) 2σ2 = ψ(e2πiν ) 2f (ν). y | | w | | w When we pass a time series X through a linear filter, the spectral density { t} is multiplied, frequency-by-frequency, by the squared modulus of the frequency response ν ψ(e2πiν ) 2. → | | This is a version of the equality Var(aX)= a2Var(X), but the equality is true for the component of the variance at every frequency. This is also the origin of the name ‘filter.’

9 Frequency response of a filter: Details

2πiν 2 Why is fy(ν)= ψ e fx(ν)? First, ∞ ∞ γy(h)= E  ψj Xt−j ψkXt+h−k j=−∞ k=−∞   ∞ ∞ = ψj ψkE [Xt+h−kXt−j] j=−∞ k=−∞ ∞ ∞ ∞ ∞ = ψ ψ γ (h + j k)= ψ ψ γ (l). j k x − j h+j−l x j=−∞ k=−∞ j=−∞ l=−∞

It is easy to check that ∞ ψ < and ∞ γ (h) < imply j=−∞ | j | ∞ h=−∞ | x | ∞ that ∞ γ (h) < . Thus, the spectral density of y is defined. h=−∞ | y | ∞

10 Frequency response of a filter: Details

∞ −2πiνh fy(ν)= γ(h)e h=−∞ ∞ ∞ ∞ −2πiνh = ψj ψh+j−lγx(l)e h=−∞ j=−∞ l=−∞ ∞ ∞ ∞ 2πiνj −2πiνl −2πiν(h+j−l) = ψje γx(l)e ψh+j−le j=−∞ l=−∞ h=−∞ ∞ 2πiνj −2πiνh = ψ(e )fx(ν) ψhe h=−∞ 2πiνj 2 = ψ(e ) fx(ν).

11 Frequency response: Examples

2πiν 2 2 For a linear process Yt = ψ(B)Wt, fy(ν)= ψ e σw. For an ARMA model, ψ(B)= θ(B)/φ(B), so Y has the rational { t} spectrum

2 θ(e−2πiν ) f (ν)= σ2 y w φ (e−2πiν) 2 q −2πiν 2 θ q e zj = σ2 j=1 − , w 2 p −2πiν 2 φp e pj j=1 | − | where pj and zj are the poles and zeros of the rational function z θ(z)/φ(z). →

12 Frequency response: Examples

Consider the moving average

1 k Y = X . t 2k + 1 t−j j=−k This is a time invariant linear filter (but it is not causal). Its transfer function is the Dirichlet kernel

1 k ψ(e−2πiν )= D (2πν)= e−2πijν k 2k + 1 j=−k

1 if ν = 0, =  sin(2π(k+1/2)ν)  (2k+1) sin(πν) otherwise. 

13 Example: Moving average

Transfer function of moving average (k=5) 1

0.8

0.6

0.4

0.2

0

−0.2

−0.4 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 ν

14 Example: Moving average

Squared modulus of transfer function of moving average (k=5) 1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 ν

This is a low-pass filter: It preserves low frequencies and diminishes high frequencies. It is often used to estimate a monotonic trend component of a series.

15 Example: Differencing

Consider the first difference

Y = (1 B)X . t − t This is a time invariant, causal, linear filter. Its transfer function is

ψ(e−2πiν ) = 1 e−2πiν, − so ψ(e−2πiν) 2 = 2(1 cos(2πν)). | | −

16 Example: Differencing

Transfer function of first difference 4

3.5

3

2.5

2

1.5

1

0.5

0 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 ν

This is a high-pass filter: It preserves high frequencies and diminishes low frequencies. It is often used to eliminate a trend component of a series.

17 Introduction to Time Series Analysis. Lecture 18. 1. Review: Spectral density, rational spectra, linear filters.

2. Frequency response of linear filters. 3. Spectral estimation

4. Sample autocovariance

5. Discrete Fourier transform and the periodogram

18 Estimating the Spectrum: Outline

We have seen that the spectral density gives an alternative view of • stationary time series.

Given a realization x ,...,x of a time series, how can we estimate • 1 n the spectral density?

One approach: replace γ( ) in the definition • ∞ f(ν)= γ(h)e−2πiνh, h=−∞ with the sample autocovariance γˆ( ). Another approach, called the periodogram: compute I(ν), the squared • modulus of the discrete Fourier transform (at frequencies ν = k/n).

19 Estimating the spectrum: Outline

These two approaches are identical at the Fourier frequencies ν = k/n. • The asymptotic expectation of the periodogram I(ν) is f(ν). We can • derive some asymptotic properties, and hence do hypothesis testing.

Unfortunately, the asymptotic variance of I(ν) is constant. • It is not a consistent estimator of f(ν). We can reduce the variance by smoothing the periodogram—averaging • over adjacent frequencies. If we average over a narrower range as n , we can obtain a consistent estimator of the spectral density. → ∞

20 Estimating the spectrum: Sample autocovariance

Idea: use the sample autocovariance γˆ( ), defined by n−|h| 1 γˆ(h)= (x x¯)(x x¯), for n

21 Estimating the spectrum: Periodogram

Another approach to estimating the spectrum is called the periodogram. It was proposed in 1897 by Arthur Schuster (at Owens College, which later became part of the University of Manchester), who used it to investigate periodicity in the occurrence of earthquakes, and in sunspot activity.

Arthur Schuster, “On Lunar and Solar Periodicities of Earthquakes,” Proceedings of the Royal Society of London, Vol. 61 (1897), pp. 455–465. To define the periodogram, we need to introduce the discrete Fourier transform of a finite sequence x1,...,xn.

22 Introduction to Time Series Analysis. Lecture 18. 1. Review: Spectral density, rational spectra, linear filters.

2. Frequency response of linear filters. 3. Spectral estimation

4. Sample autocovariance

5. Discrete Fourier transform and the periodogram

23 Discrete Fourier transform

For a sequence (x1,...,xn), define the discrete Fourier transform (DFT) as

(X(ν0),X(ν1),...,X(νn−1)), where

1 n X(ν )= x e−2πiνkt, k √n t t=1 and ν = k/n (for k = 0, 1,...,n 1) are called the Fourier frequencies. k − (Think of ν : k = 0,...,n 1 as the discrete version of the frequency { k − } range ν [0, 1].) ∈ First, let’s show that we can view the DFT as a representation of x in a different basis, the Fourier basis.

24 Discrete Fourier transform

Consider the space Cn of vectors of n complex numbers, with inner product a,b = a∗b, where a∗ is the complex conjugate transpose of the vector a Cn. ∈ Suppose that a set φ : j = 0, 1,...,n 1 of n vectors in Cn are { j − } orthonormal: 1 if j = k, φj,φk =  0 otherwise. Then these φ span the vector space Cn, and so for any vector x, we can { j} write x in terms of this new orthonormal basis,

n−1

x = φ , x φ . (picture) j j j=0

25 Discrete Fourier transform

Consider the following set of n vectors in Cn:

1 ′ e = e2πiνj , e2πi2νj ,...,e2πinνj : j = 0,...,n 1 . j √n − It is easy to check that these vectors are orthonormal: 1 n 1 n t e , e = e2πit(νk−νj ) = e2πi(k−j)/n j k n n t=1 t=1 1 if j = k, =  2πi(k−j)/n n 1 2πi(k−j)/n 1−(e )  n e 1−e2πi(k−j)/n otherwise

1 if j = k, =  0 otherwise, 

26 Discrete Fourier transform

n t where we have used the fact that Sn = t=1 α satisfies αS = S + αn+1 α and so S = α(1 αn)/(1 α) for α = 1. n n − n − − So we can represent the real vector x =(x ,...,x )′ Cn in terms of this 1 n ∈ orthonormal basis,

n−1 n−1 x = e , x e = X(ν )e . j j j j j=0 j=0

That is, the vector of discrete Fourier transform coefficients

(X(ν0),...,X(νn−1)) is the representation of x in the Fourier basis.

27 Discrete Fourier transform

An alternative way to represent the DFT is by separately considering the real and imaginary parts,

1 n X(ν )= e , x = e−2πitνj x j j √n t t=1 1 n 1 n = cos(2πtν )x i sin(2πtν )x √n j t − √n j t t=1 t=1 = X (ν ) iX (ν ), c j − s j where this defines the sine and cosine transforms, Xs and Xc, of x.

28 Introduction to Time Series Analysis. Lecture 18. 1. Review: Spectral density, rational spectra, linear filters.

2. Frequency response of linear filters. 3. Spectral estimation

4. Sample autocovariance

5. Discrete Fourier transform and the periodogram

29