Lesson 9: Autoregressive-Moving Average (ARMA) models

Umberto Triacca

Dipartimento di Ingegneria e Scienze dell’Informazione e Matematica Universit`adell’Aquila, [email protected]

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Introduction

We have seen that in the class of stationary, zero mean, Gaussian processes the probabilistic structure of a stochastic process is completly characterized by the autocovariance function.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autocovariance function

Stationary, zero mean, Gaussian process

'$ DGP γx (k)

7 &%          ? x1, ..., xT 

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models   Introduction

However, in general, to know the autocovariance function means to know a sequence composed by an infinite number of elements.

We have to estimate a infinite number of parameters

γx (0), γx (1), γx (2), ...,

from observed data.

This mission is impossible

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Introduction

We introduce a very important class of stochastic processes, which autocovariance functions depend on a finite number of unknown parameters:

the class of the AutoregRessive Moving Average (ARMA) processes.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

Definition. The process {xt ; t ∈ Z} is an autoregressive moving average process of order (p, q), denoted with

xt ∼ ARMA(p, q),

if

xt − φ1xt−1 − ... − φpxt−p = ut + θ1ut−1 + ... + θqut−q ∀t ∈ Z,

2 where ut ∼ WN(0, σu), and φ1, ..., φp, θ1, ..., θq are p + q constants and the polynomials

p φ(z) = 1 − φ1z − ... − φpz

and q θ(z) = 1 + θ1z... + θqz have no common factors. Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

For q = 0 the process reduces to an autoregressive process of order p, denoted with xt ∼ AR(p),

xt − φ1xt−1 − ... − φpxt−p = ut ∀t ∈ Z, For p = 0 to a moving average process of order q, denoted with xt ∼ MA(q)

xt = ut + θ1ut−1 + ... + θqut−q ∀t ∈ Z,

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models An example of Autoregressive-Moving Average (ARMA) process

The process {xt ; t ∈ Z} defined by

xt = 0.3xt−1 + ut + 0.7ut−1 ∀t ∈ Z,

2 where ut ∼ WN(0, σu), is an ARMA(1,1) process. Here φ(z) = 1 − 0.3z and θ(z) = 1 + 0.7z.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models An example of Autoregressive-Moving Average (ARMA) process

A realizzation of the ARMA(1,1) process xt = 0.3xt−1 + ut + 0.7ut−1 is presented in the following figure.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models An example of Autoregressive (AR) process

The process {xt ; t ∈ Z} defined by

xt = 0.7xt−1 − 0.5xt−1 + ut ∀t ∈ Z,

2 where ut ∼ WN(0, σu), is an AR(2) process. Here φ(z) = 1 − 0.7z + 0.5z 2

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models An example of Autoregressive (AR) process

A realizzation of the AR(2) process xt = 0.7xt−1 − 0.5xt−2 + ut is presented in the following figure.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models An example of Moving Average (MA) process

The process {xt ; t ∈ Z} defined by

xt = ut + 0.7ut−1 ∀t ∈ Z,

2 where ut ∼ WN(0, σu), is an MA(1) process. Here θ(z) = 1 + 0.7z

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models An example of Moving Average (MA) process

A realizzation of the MA(1) process xt = ut + 0.7ut−1 is presented in the following figure.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models An example of over-parameterization

Consider the process {xt ; t ∈ Z} defined by

xt = xt−1 − 0.21xt−2 + ut − 0.7ut−1 ∀t ∈ Z,

2 where ut ∼ WN(0, σu).

This process looks like an ARMA(2,1) process but it is not an ARMA(2,1) process.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models An example of over-parameterization

Here

φ(z) = 1 − z + 0.21z 2 = (1 − 0.7z)(1 − 0.3z)

and θ(z) = 1 − 0.7z We note that both polynomials have a common factor, namely 1 − 0.7z. Discarding the common factor in each leaves

φ∗(z) = 1 − 0.3z

and θ∗(z) = 1. Thus the process is an AR(1) process, defined by xt = 0.3xt−1 + ut

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Causal Autoregressive-Moving Average (ARMA) models

Definition. An ARMA(p, q) process {xt ; t ∈ Z} is causal (strictly, a causal function of {ut ; t ∈ Z}) if there exists constants ψ0, ψ1, ... such that

∞ X |ψj | < ∞ j=0

and ∞ X xt = ψj ut−j ∀t. j=0

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

Here, it is important to clarify the meaning of equality

∞ X xt = ψj ut−j ∀t j=0

It means that

 n !2 X lim E  xt − ψj ut−j  = 0. n→∞ j=0

The equality is defined in terms of a limit in the quadratic mean.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

The following two theorems provide, respectively, a characterization of the of causality and stationarity of an ARMA(p, q) process.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

Theorem. An ARMA(p, q) process {xt ; t ∈ Z} is causal if and only if

p φ(z) = 1 − φ1z − ... − φpz 6= 0 for all |z| ≤ 1.

Theorem. An ARMA(p, q) process {xt ; t ∈ Z} is stationary if and only if

p φ(z) = 1 − φ1z − ... − φpz 6= 0 for all |z| = 1.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

The causality and the stationarity of an ARMA process depend entirely on the autoregressive parameters and not on the moving-average ones.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

Further, we note that if an ARMA(p, q) process is causal, then is stationary, but stationarity does not imply causality.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

Consider, for example, the following AR(1) process:

xt = 3xt−1 + ut

2 where ut WN(0, σu). We have that

φ(z) = 1 − 3z 6= 0 for all |z| = 1.

and hence the process is stationary, but non causal since

φ(z) = 1 − 3z = 0 for z = 1/3.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

An important result: There is a one-to-one correspondence between the parameters of a causal ARMA(p,q) process and the autocovariance function.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

It is important to underline that if we consider the set of autocorrelation functions there is not a one-to-one correspondence between the parameters of a causal ARMA(p,q) process and the autocorrelation function.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

Consider the following two MA(1) processes.

xt = ut + θut−1

2 where ut ∼ WN(0, σu), with |θ| < 1 and 1 y = u + u t t θ t−1

2 where ut ∼ WN(0, σu). Since θ 1/θ = , 1 + θ2 1 + (1/θ)2 we have that both processes share the same autocorrelation function. Thus it cannot be used to distinguish between the two parametrizations.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

This example shows that an MA(1)-process is not uniquely determined by its autocorrelation function. There is an identification problem with the MA(1) models.

In general, (if all roots of θ(z) = 0 are real) there can be 2q different MA(q) processes with the same autocorrelation function.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

Definition. An ARMA(p, q) process {xt ; t ∈ Z} is invertible (strictly, an invertible function of {ut ; t ∈ Z}) if there exists constants π0, π1, ... such that

∞ X |πj | < ∞ j=0

and ∞ X ut = πj xt−j ∀t. j=0

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

The following theorem provides a necessary and sufficient condition for the invertibility.

Theorem. An ARMA(p, q) process {xt ; t ∈ Z} is invertible if and only if

q θ(z) = 1 + θ1z + ... + θqz 6= 0 for all |z| ≤ 1.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

We note that an AR(p) process is always invertible, even if it is non-stationary, while an MA(q) process is always stationary, even if it is non-invertible.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

The invertibility can be used in order to ensure the identifiability of MA processes.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Autoregressive-Moving Average (ARMA) models

In general, (if all roots of θ(z) = 0 are real) there can be 2q different MA(q) processes with the same autocorrelation function, but only one of these is invertible.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Conclusion

In the class of the mean-zero causal and invertible Gaussian ARMA processes there is a one-to-one correspondence between the family of the finite dimensional distributions of the process and the finite parametric representation of process.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Conclusion

In the class of the mean-zero causal and invertible Gaussian ARMA processes the probabilistic properties of the process are completely characterized by the finite set of parameters

 2 φ1, φ2, ..., φp, θ1, θ2, ..., θq, σu

Now, we have to estimate a finite number (p + q + 1) of parameters from observed data.

This mission is possible.

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models Conclusion

Zero-mean causal invertible Gaussian ARMA process

'$DGP 2 {φ1, φ2, ..., φp, θ1, θ2, ..., θq, σu}

7 &%          ? x1, ..., xT 

Umberto Triacca Lesson 9: Autoregressive-Moving Average (ARMA) models