Time Series II -- Frequency Domain Methods

Home , Frequency domain, Time domain

Outline Introduction Stationarity The Periodogram and the Spectral Density Smoothing and Tapering Example Extensions and References

Time Series II – Frequency Domain Methods

John Fricks

Astrostatistics Summer School Penn State University University Park, PA 16802

June 8, 2007

John Fricks Time Series II – Frequency Domain Methods Outline Introduction Stationarity The Periodogram and the Spectral Density Smoothing and Tapering Example Extensions and References

Introduction

Stationarity

The Periodogram and the Spectral Density Periodogram and Regression Spectral Density

Smoothing and Tapering Smoothing Tapering

Example

Extensions and References

I In the previous tutorial, we discussed rules for evolution of a time series in the time domain.

I In this tutorial, we will discuss time series from a diﬀerent perspective; we will look at the frequency components of the data.

I The assumption of stationarity imposes regularity on a time series model.

I We will need repeated observations with the same or similar relationship to one another in order to estimate the underlying relationships between observations.

I There are other ways to do this, but stationarity is the most common and perhaps most basic.

A time series ..., x−1, x0, x1, x2, ... is strictly stationary if for a sequence of times t1, t2, ..., tk

{xt1 , ..., xtk }

has the same distributions as

{xt1+h, ..., xtk +h}

for every integer h. In other words,

P{xt1 ≤ c1, ..., xtk ≤ ck } = P{xt1+h ≤ c1, ..., xtk +h ≤ ck }.

An important measure of dependency in time series is autocovariance. This is deﬁned as

γ(t, s) = E(xt − µt )(xs − µs )

where µt = Ext .

The time series xt is weakly stationary if µt is constant and γ(s, t) depends only on the distance |s − t|.

In the case of Gaussian time series, these two concepts of stationarity overlap.

For a weakly stationary time series, the notation used for autocovariance uses only lag:

γ(h) = E(xt − µ)(xt−h − µ)

where µ is the constant variance. We also have a concept of the autocorrelation function which we saw in the ﬁrst section in the ACF plot. The autocorrelation function is deﬁned as γ(h) ρ(h) = γ(0)

John Fricks Time Series II – Frequency Domain Methods Outline Introduction Stationarity Periodogram and Regression The Periodogram and the Spectral Density Spectral Density Smoothing and Tapering Example Extensions and References Discrete Fourier Transform of the Time Series

What if we are less interested in how our underlying process evolves in time and are more interested in the variance of the time series at certain frequencies?

We may attempt to apply a Fourier transform to the data. For our time series, x1, ..., xn, the discrete Fourier transform would be

n −1/2 X d(ωj ) = n xt exp(−2πitωj ) t=1

where ωj = 0, 1/n, ..., (n − 1)/n.

Note that we can break up d(ωj ) into two parts

n n −1/2 X −1/2 X d(ωj ) = n xt cos(2πiωj t) − in xt sin(2πiωj t) t=1 t=1 which we could write as as a cosine component and a sine component d(ωj ) = dc (ωj ) − ids (ωj )

n −1/2 X 2πiωj t xt = n d(ωj )e j=1 n −1/2 X 2πiωj t = n d(ωj )e j=1 m n −1/2 X 2πiωj t −1/2 X 2πiωj t = a0 + n d(ωj )e + n d(ωj )e j=1 j=m+1 m m X 2dc (ωj ) X 2ds (ωj ) = a + cos(2πiω t) + sin(2πiω t) 0 n−1/2 j n−1/2 j j=1 j=1

n where m = b 2 c John Fricks Time Series II – Frequency Domain Methods Outline Introduction Stationarity Periodogram and Regression The Periodogram and the Spectral Density Spectral Density Smoothing and Tapering Example Extensions and References

I We can think of the Fourier transform as a regression of xt on sines and cosines. √ I The coeﬃcients of this regression are equal to 2/ n times the sine part and the cosine part of the Fourier transforms respectively.

I Also note that if our time series is Gaussian, dc (ωj ) and ds (ωj ) are Gaussian random variables.

I The periodogram is deﬁned as

2 2 2 I (ωj ) = |d(ωj )| = dc (ωj ) + dc (ωj )

I If there is no periodic trend in the data, then Ed(ωj ) = 0, and the periodogram expresses the variance of xt at frequency ωj .

I If a periodic trend exists in the data, then Ed(ωj ) will be the contribution to the periodic trend at the frequency ωj .

I What are we trying to estimate with the periodogram?

I We can use the periodogram to ﬁnd periodic trends in the data.

I Is there information left in the periodogram after the trend is removed?

I Assuming that we have a stationary time series, what does the periodogram estimate?

The spectral density is the Fourier transform of the autocovariance function h=∞ X f (ω) = e−2πiωhγ(h) h=−∞ for ω ∈ (−0.5, 0.5). Note that this is a population quantity. (i.e. This is a constant quantity deﬁned by the model.)

n n 2 −1 X −2πiωj t X −2πiω t I (ωj ) = |d(ωj )| = n xt e xt e j t=1 t=1 n n 2 −1 X X −2πiωj (t−s) = |d(ωj )| = n (xt − m)(xs − m)e t=1 s=1 (n−1) n−|h| −1 X X −2πiωj (h) = n (xt+|h| − m)(xt − m)e h=−(n−1) t=1 (n−1) X −2πiωj (h) = γˆ(h)e ≈ f (ωj ) h=−(n−1)

(note that h = t − s.) John Fricks Time Series II – Frequency Domain Methods Outline Introduction Stationarity Smoothing The Periodogram and the Spectral Density Tapering Smoothing and Tapering Example Extensions and References

I Is the periodogram a good estimator for the spectral density? Not really!

I The periodogram, I (ω1), ..., I (ωm), attempt to estimate paramters f (ω1), ..., f (ωm). We have nearly the same number of parameters as we have data.

I Moreover, the number of parameters grow as a constant proportion of the data. Therefore, the periodogram is NOT a consistent estimator of the spectral density.

I A simple way to improve our estimates is to use a moving average smoothing technique m 1 X fˆ(ω ) = I (ω ) j 2m + 1 j−k k=−m

I We can also iterate this procedure of uniform weighting to be more weight on closer observations. 1 1 1 uˆ = u + u + u t 3 t−1 3 t 3 t+1 Then, we iterate. 1 1 1 uˆ = uˆ + uˆ + uˆ t 3 t−1 3 t 3 t+1 Then, substitute to obtain better weights.

● ●

6000 ● ●

● ● 5000

● ● ● ● ● 4000 ● ● 3000 var ●

● 2000

● ● ● 1000 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 ● ● ● ● ● ● ● ●

0.0 0.1 0.2 0.3 0.4 0.5

John Fricks Timefreq Series II – Frequency Domain Methods Outline Introduction Stationarity Smoothing The Periodogram and the Spectral Density Tapering Smoothing and Tapering Example Extensions and References Smoothing Summary

I Smoothing decreases variance by averaging over the periodogram of neighboring frequencies.

I Smoothing introduces bias because the expectation of neighboring periodogram values are similar but not identical to the frequency of interest.

I Beware of oversmoothing!

I Tapering corrects bias introduced from the ﬁniteness of the data.

I The expected value of the periodogram at a certain frequency is not quite equal to the spectral density.

I It is aﬀected by the spectral density at neighboring frequency points.

I For a spectral density which is more dynamic, more tapering is required.

John Fricks Time Series II – Frequency Domain Methods Outline Introduction Stationarity Smoothing The Periodogram and the Spectral Density Tapering Smoothing and Tapering Example Extensions and References Why do we need to taper?

Our theoretical model ..., x−1, x0, x1, ... consists of a doubly inﬁnite time series. We could think of our data, yt as the following transformation of the model

yt = ht xt

where ht = 1 for t = 1, ..., n and zero otherwise. This has repercussions on the expectation of the periodogram of our data.

Z 0.5 E[Iy (ωj )] = Wn(ωj − ω)fx (ω)dω −0.5

2 where Wn(ω) = |Hn(ω)| and Hn(ω) is the Fourier transform of the sequence ht .

Speciﬁcally, n 1 X H (ω) = √ h e−2πiωt n n t t=1

When we put in the ht above, we obtain a spectral window of

sin2(n2πω) Wn(ω) = . sin2(πω)

We set Wn(0) = n.

Fejer window, n=480 400 200 spectrum 0

−0.10 −0.05 0.00 0.05 0.10

freq

Fejer window (log), n=480 −20 spectrum −60

−0.10 −0.05 0.00 0.05 0.10

freq

Fejer window, n=480, L=9 Fejer window(log), n=480, L=9 0 50 −20 30 −40 spectrum spectrum 10 −60 0

−0.10 0.00 0.05 0.10 −0.10 0.00 0.05 0.10

freq freq

Full Tapering Window, n=480, L=9 Full Tapering Window(log), n=480, L=9 20 15 −5 10 −15 spectrum spectrum 5 −25 0

−0.10 0.00 0.05 0.10 −0.10 0.00 0.05 0.10

freq freq

Full Tapering, n=480, transformation in time domain 0.8 0.4 0.0 0 100 200 300 400

time

Full Tapering Window, n=480, L=9 15 5 spectrum 0

−0.10 −0.05 0.00 0.05 0.10

freq

Full Tapering Window(log), n=480, L=9 0 −10 spectrum −25

−0.10 −0.05 0.00 0.05 0.10

freq

50% Tapering, n=480, transformation in time domain 0.8 0.4 0.0 0 100 200 300 400

time

50% Tapering Window, n=480, L=9 30 10 spectrum 0

−0.10 −0.05 0.00 0.05 0.10

freq

50% Tapering Window(log), n=480, L=9 −5 spectrum −15

−0.10 −0.05 0.00 0.05 0.10

freq

I Smoothing introduces bias, but reduces variance.

I Smoothing tries to solve the problem of too many “parameters”.

I Tapering decreases bias and introducese variance.

I Tapering attempts to diminish the inﬂuence of sidelobes that are introduced via the spectral window.

0 20 40 60 80 100

Time

Series: sun Raw Periodogram 12000 8000 spectrum 6000 4000 2000 0

0.0 0.1 0.2 0.3 0.4 0.5

frequency bandwidth = 0.00289

Series: sun Smoothed Periodogram 10000 8000 6000 spectrum 4000 2000 0

0.0 0.1 0.2 0.3 0.4 0.5

frequency bandwidth = 0.00764

Series: sun Smoothed Periodogram 8000 6000 4000 spectrum 2000 0

0.0 0.1 0.2 0.3 0.4 0.5

frequency bandwidth = 0.0126

Series: sun Smoothed Periodogram 10000 8000 6000 spectrum 4000 2000 0

0.0 0.1 0.2 0.3 0.4 0.5

frequency bandwidth = 0.00764

Series: sun Smoothed Periodogram 12000 10000 8000 6000 spectrum 4000 2000 0

0.0 0.1 0.2 0.3 0.4 0.5

frequency bandwidth = 0.00764

John Fricks Time Series II – Frequency Domain Methods Outline Introduction Stationarity The Periodogram and the Spectral Density Smoothing and Tapering Example Extensions and References Smoothed Periodogram with ARMA Spectral Density The smoothed periodogram of the sun spot data with the spectral density of the AR(3) model overlayed. 12000 10000 8000 var 6000 4000 2000 0

0.0 0.1 0.2 0.3 0.4 0.5

freq

I What can be done for non-stationary data?

I One approach is to decompose our time series as a sum of a non-constant (deterministic) trend plus a stationary “noise” term: xt = µt + yt

I What if our data instead appears as a stationary model locally, but globally the model appears to shift? One approach is to divide the data into shorter sections (perhaps overlapping) and

I This approach is developed in Shumway and Stoﬀer. One essentially looks at how the spectral density changes over time.

John Fricks Time Series II – Frequency Domain Methods Outline Introduction Stationarity The Periodogram and the Spectral Density Smoothing and Tapering Example Extensions and References Wavelets

I We have been using Fourier components as a basis to represent stationary processes and seasonal trends.

I Since we are dealing with ﬁnite data, we must use a ﬁnite number of terms, and perhaps one could use an alternative basis.

I Wavlets are one option to accomplish this goal. They are particularly well suited to the same situation as dynamic Fourier analysis.

John Fricks Time Series II – Frequency Domain Methods Outline Introduction Stationarity The Periodogram and the Spectral Density Smoothing and Tapering Example Extensions and References References

I Robert Shumway and David Stoﬀer. Time Series Analysis and Its Applications. Springer NY, 2006.

I Peter Brockwell and Richard Davis. Time Series: Theory and Methods, Second Ed. Springer NY, 1991.

I David Brillinger. Time Series: Data Analysis and Theory. SIAM, 2001.

I Donald Percival and Andrew Walden. Spectral Analysis for Physical Applications. Cambridge University Press, 1993.

I Donald Percival and Andrew Walden. Wavelet Methods for Time Series Analysis. Cambridge University Press, 2000.

I St´ephaneMallat. A Wavelet Tour of Signal Processing. Academic Press, 1998.

John Fricks Time Series II – Frequency Domain Methods