Examples of Stationary Processes 1) Strong Sense White Noise: A

Examples of Stationary Processes

1) Strong Sense White Noise: A process ǫt is strong sense white noise if ǫt is iid with mean 0 and ﬁnite variance σ2.

2) Weak Sense (or second order or wide sense) White Noise: ǫt is second order stationary with

E(ǫt) = 0 and σ2 s = t Cov(ǫt,ǫs)=  0 s 6= t  2 In this course: ǫt denotes white noise; σ denotes variance of ǫt. Use subscripts for variances of other things.

16 Example Graphics:

White noise: iid N(0, 1) data IID N(0,1)

0 200 400 600 800 1000

White noise: Xt = ǫt ··· ǫt+9 Wide Sense White Noise

0 200 400 600 800 1000

17 2) Moving Averages: if ǫt is white noise then Xt = (ǫt + ǫt−1)/2 is stationary. (If you use second order white noise you get second order stationary. If the white noise is iid you get strict stationarity.)

Example proof: E(Xt)= E(ǫt)+E(ǫt−1) /2= 0 which is constant as required. Moreover: Cov(Xt, Xs) is

Var(ǫt)+Var(ǫt−1) 4 s = t 1  Cov(ǫt + ǫt−1,ǫt+1 + ǫt) s = t + 1 4 1 4Cov(ǫt + ǫt−1,ǫt+2 + ǫt+1) s = t + 2 .  .   Most of these covariances are 0. For instance

Cov(ǫt + ǫt−1,ǫt+2 + ǫt+1)= Cov(ǫt,ǫt+2) + Cov(ǫt,ǫt+1) + Cov(ǫt−1,ǫt+2) + Cov(ǫt−1,ǫt+1) = 0 because the ǫs are uncorrelated by assumption.

18 The only non-zero covariances occur for s = t 2 and s = t ± 1. Since Cov(ǫt,ǫt)= σ we get 2 σ s = t  2    2 Cov(Xt, Xs)= σ |s − t| = 1  4

 0 otherwise   Notice that this depends only on |s − t| so that the process is stationary.

The proof that X is strictly stationary when the ǫs are iid is in your homework; it is quite diﬀerent.

19 Example Graphics:

Xt = (ǫt + ǫt−1)/2 MA(1)Process

0 200 400 600 800 1000

Xt = ǫt + 6ǫt−1 + 15ǫt−2 + 20ǫt−3 +15ǫt−4 + 6ǫt−5 + ǫt−6 MA(6) Process

0 200 400 600 800 1000

20 The trajectory of X can be made quite smooth (compared to that of white noise) by averaging over many ǫs.

3) Autoregressive Processes:

AR(1) process X: process satisfying equations:

Xt = µ + ρ(Xt−1 − µ)+ ǫt (1) where ǫ is white noise. If Xt is second order stationary with E(Xt) = θ, say, then take ex- pected values of (1) to get

θ = µ + ρ(θ − µ) which we solve to get

θ(1 − ρ)= µ(1 − ρ) . Thus either ρ = 1 (later – X not stationary) or θ = µ. Calculate variances:

Var(Xt) = Var(µ + ρ(Xt−1 − µ)+ ǫt) = Var(ǫt) + 2ρCov(Xt−1,ǫt) 2 + ρ Var(Xt−1)

21 Now assume that the meaning of (1) is that ǫt is uncorrelated with Xt−1, Xt−2, ··· .

Strictly stationary case: imagining somehow Xt−1 is built up out of past values of ǫs which are independent of ǫt.

Weakly stationary case: imagining that Xt−1 is actually a linear function of these past values.

Either case: Cov(Xt−1,ǫt) = 0.

2 If X is stationary: Var(Xt) = Var(Xt−1) ≡ σX so 2 2 2 2 σX = σ + ρ σX whose solution is σ2 σ2 = X 1 − ρ2

22 Notice that this variance is negative or unde- ﬁned unless |ρ| < 1. There is no stationary process satisfying (1) for |ρ|≥ 1.

Now for |ρ| < 1 how is Xt determined from the ǫs? (We want to solve the equations (1) to get an explicit formula for Xt.) The case µ =0 is notationally simpler. We get

Xt = ǫt + ρXt−1 = ǫt + ρ(ǫt−1 + ρXt−2) . k−1 = ǫt + ρǫt−1 + ··· + ρ ǫt−k+1 k + ρ Xt−k Since |ρ| < 1 it seems reasonable to suppose k that ρ Xt−k → 0 and for a stationary series X this is true in the appropriate mathematical sense. This leads to taking the limit as k →∞ to get ∞ j Xt = ρ ǫt−j . jX=0

23 Claim: if ǫ is a weakly stationary series then ∞ j Xt = j=0 ρ ǫt−j converges (technically it con- vergesP in mean square) and is a second order stationary solution to the equation (1).

If ǫ is a strictly stationary process then under some weak assumptions about how heavy the ∞ j tails of ǫ are Xt = j=0 ρ ǫt−j converges almost surely and is a stronglyP stationary solution of (1).

In fact; if ...,a−1, a0, a1, a2,... are constants 2 such that aj < ∞ and ǫ is weak sense white noise (respectivelyP strong sense white noise with finite variance) then ∞ Xt = ajǫt−j j=X−∞ is weakly stationary (respectively strongly stationary with finite variance). In this case we call X a linear filter of ǫ.

24 Example Graphics:

AR(1)Process: Rho=0.99

0 200 400 600 800 1000

AR(1) Process: Rho=0.5

0 200 400 600 800 1000

25 Motivation of the jargon “ﬁlter” comes from physics.

Consider an electric circuit with a resistance R in series with a capacitance C.

Apply “input” voltage U(t) across the two el- ements.

Measure voltage drop across capacitor.

Call this voltage drop “output” voltage; denote output voltage by Xt.

26 The relevant physical rules are these:

1. The total voltage drop around the circuit is 0. This drop is −U(t) plus the voltage drop across the resistor plus X(t). (The negative sign is a convention; the input voltage is not a “drop”.)

2. Voltage drop across resistor is Ri(t) where i is current ﬂowing in circuit.

3. If the capacitor starts oﬀ with no charge on its plates then the voltage drop across its plates at time t is t i(s) ds X(t)= 0 R C

These rules give t i(s) ds U(t)= Ri(t)+ 0 R C

27 Diﬀerentiate the deﬁnition of X to get

X′(t)= i(t)/C so that

U(t)= RCX′(t)+ X(t) . Multiply by et/RC/RC to see that

et/RCU(t) ′ = et/RCX(t) RC whose solution, remembering X(0) = 0, is ob- tained by integrating from 0 to s to get 1 s es/RCX(s)= et/RCU(t) dt RC Z0 leading to 1 s X(s)= e(t−s)/RCU(t) dt RC Z0 1 s = e−u/RCU(s − u) du RC Z0 This formula is the integral equivalent of our definition of filter and shows X = filter(U).

28 Defn: If {ǫt} is a white noise series and µ and b0,...,bp are constants then

Xt = µ + b0ǫt + b1ǫt−1 + ··· + bpǫt−p is a moving average of order p; write MA(p).

Defn: A process X is an autoregression of order p (written AR(p)) if p Xt = ajXt−j + ǫt. X1

Defn: Process X is an ARMA(p, q) (mixed autoregressive of order p and moving average of order q) if for some white noise ǫ: φ(B)X = ψ(B)ǫ p j φ(B)= I − ajB X1 and p j ψ(B)= I − bjB X1

Problems: existence, stationarity, estimation, etc. 29 Other Stationary Processes:

Periodic processes: Suppose Z1 and Z2 are independent N(0, σ2) random variables and that ω is a constant. Then

Xt = Z1 cos(ωt)+ Z2 sin(ωt) has mean 0 and

2 Cov(Xt, Xt+h)= σ [cos(ωt) cos(ω(t + h)) + sin(ωt) sin(ω(t + h))] = σ2 cos(ωh) Since X is Gaussian we ﬁnd that X is second order and strictly stationary. In fact (see your homework) You can write

Xt = R sin(ωt + Φ) where R and Φ are suitable random variables so that the trajectory of X is just a sine wave.

30 Poisson shot noise processes:

Poisson process is a process N(A) indexed by subsets A of R such that each N(A) has a Pois- son distribution with parameter λlength(A) and if A1,...Ap are any non-overlapping subsets of R then N(A1),...,N(Ap) are independent. We often use N(t) for N([0, t]).

Shot noise process: X(t) = 1 at those t where there is a jump in N and 0 elsewhere; X is stationary.

If g a function deﬁned on [0, ∞) and decreasing suﬃciently quickly to 0 (like say g(x) = e−x) then the process Y (t)= g(t − τ)1(X(τ) = 1)1(τ ≤ t) X is stationary.

Y jumps every time t passes a jump in Poisson process; otherwise follows trajectory of sum of several copies of g (shifted around in time). We commonly write ∞ Y (t)= g(t − τ)dN(τ) Z0

31 ARCH Processes: (Autoregressive Conditional Heteroscedastic)

Defn: Mean 0 process X is ARCH(p) if

Var(Xt+1|Xt, Xt−1, ··· ) ∼ N(0, Ht) where p 2 Ht = α0 + αiXt+1−i X1

GARCH Processes: (Generalized Autoregres- sive Conditional Heteroscedastic)

Defn: The process X is GARCH(p, q) if X has mean 0 and

Var(Xt+1|Xt, Xt−1, ··· ) ∼ N(0, Ht) where p q 2 Ht = α0 + αiXt+1−i + βjHt−j X1 X1

Used to model series with patches of high and low variability. 32 Markov Chains

Defn: A transition kernel is a function P (A,x) which is, for each x in a set X (the state space), a probability on X .

Defn: A sequence Xt is Markov (with stationary transitions P ) if

P (Xt+1 ∈ A|Xt, Xt−1, ··· )= P (A, Xt)

That is, the conditional distribution of Xt+1 given all history to time t depends only on value of Xt.

Fact: under some conditions as t →∞ Xt, Xt+1,... becomes stationary.

Fact: under similar conditions can give X0 a distribution (called stationary initial distribution) so that X is a strictly stationary process.