Chapter 2 Fundamental Concepts
Chapter 2 Fundamental Concepts
This chapter describes the fundamental concepts in the theory of time series models. In particular we introduce the concepts of stochastic process, mean and covariance function, stationary process, and autocorrelation function.
2.1 Time Series and Stochastic Processes
± ± ± The sequence of random variables {Zt: t = 0, 1, 2, 3,...} is called a stochastic process. It is known that the complete probabilistic structure of such a process is determined by the set of distributions of all finite collections of the Z’s. Fortunately, we will not have to deal explicitly with these multivariate distributions. Much of the information in these joint distributions can be described in terms of means, variances, and covariances. Consequently, we concentrate our efforts on these first and second moments. (If the joint distributions of the Z’s are multivariate normal distributions, then the first and second moments completely determine all the joint distributions.)
2.2 Means, Variances, and Covariances
± ± ± For a stochastic process {Zt: t = 0, 1, 2, 3,...} the mean function is defined by µ () ± , ± , t =forEZt t = 0, 1 2 ... (2.2.1) µ µ that is, t is just the expected value of the process at time t. In general t can be different at each time point t. γ The autocovariance function, t,s, is defined as γ (), ± , ± , ts, =forCov Zt Zs t = 0, 1 2... (2.2.2) − µ − µ − µ µ where Cov(Zt, Zs) = E[(Zt t)(Zs s)] = E(ZtZs) t s. ρ The autocorrelation function, t,s, is given by ρ (), ± , ± , ts, =forCorr Zt Zs t = 0, 1 2 ... (2.2.3) where
1 Chapter 2 Fundamental Concepts
(), γ Cov Zt Zs ts, Corr() Z , Z ==------(2.2.4) t s () () γ γ Var Zt Var Zs tt, ss, We review the basic properties of expectation, variance, covariance, and correlation in “Appendix: Expectation, Variance, Covariance, and Correlation” on page 16. Recall that both covariance and correlation are measures of the (linear) dependence between random variables but that the unitless correlation is somewhat easier to interpret. The following important properties follow from known results and our definitions: γ = Var() Z ρ tt, t tt, = 1 γ = γ ρ ρ ts, st, ts, = st, (2.2.5) γ ≤ γ γ ρ ≤ 1 ts, tt, ss, ts, ρ ± Values of t,s near 1 indicate strong (linear) dependence, whereas values near zero ρ indicate weak (linear) dependence. If t,s = 0, we say that Zt and Zs are uncorrelated. To investigate the covariance properties of various time series models, the following result will be used repeatedly: If c1, c2,..., cm and d1, d2,..., dn are constants and t1, t2,..., tm and s1, s2,..., sn are time points, then m n m n Cov∑ c Z , ∑ d Z = ∑ ∑ c d Cov() Z , Z (2.2.6) i ti j sj i j ti sj i = 1 j = 1 i = 1 j = 1
The proof of Equation (2.2.6), though tedious, is a straightforward application of the linear properties of expectation. As a special case, we obtain the following well-known result n n n i – 1 Var∑ c Z = ∑ c2Var() Z + 2∑ ∑ c c Cov() Z , Z (2.2.7) i ti i ti i j ti tj i = 1 i = 1 i = 2 j = 1
The Random Walk
Let a1, a2,... be a sequence of independent, identically distributed random variables each σ2 with zero mean and variance a . The observed time series, {Zt: t = 1, 2,...}, is constructed as follows:
2 2.2 Means, Variances, and Covariances
Z1 = a1 Z = a + a …2 1 2 (2.2.8) … Zt = a1 +++a2 at
Alternatively, we can write Zt = Zt – 1 + at (2.2.9) If the a’s are interpreted as the sizes of the “steps” taken (forward or backward) along a number line, then Zt is the position of the “random walker” at time t. From Equation (2.2.8) we obtain the mean function: µ () ()… () ()… () t ==EZt Ea1 +++a2 at =Ea1 +++Ea2 Eat = 00++… +0 so that µ t = 0 for all t (2.2.10) We also have () ()… () ()… () Var Zt ==Var a1 +++a2 at Var a1 +++Var a2 Var at = σ2 σ2 … σ2 a +++a a t terms σ2 = t a so that () σ2 Var Zt = t a (2.2.11) Notice that the process variance increases linearly with time. To investigate the covariance function, suppose that 1 ≤ t ≤ s. Then we have γ (), ()… , … … ts, ==Cov Zt Zs Cov a1 +++a2 at a1 ++++a2 at at + 1 ++as From Equation (2.2.6) we have s t γ (), ts, = ∑ ∑ Cov ai aj i = 1 j = 1
σ2 However, these covariances are zero unless i = j in which case they equal Var(ai) = a . γ σ2 There are exactly t of these so that t,s = t.a
3 Chapter 2 Fundamental Concepts
γ γ Since t,s = s,t, this specifies the autocovariance function for all time points t and s and we can write γ σ2 ≤≤ ts, =fort a 1ts (2.2.12) The autocorrelation function for the random walk is now easily obtained as γ ts, t ρ =------=- for 1 ≤≤ts (2.2.13) ts, γ γ s tt, ss, The following numerical values help us understand the behavior of the random walk. 1 8 ρ ==---0.707 ρ ==--- 0.943 12, 2 89, 9 24 1 ρ ==------0.980 ρ ==------0.200 24, 25 25 125, 25 The values of Z at neighboring time points are more and more strongly and positively correlated as time goes by. On the other hand, the values of Z at distant time points are less and less correlated. A simulated random walk is shown in Exhibit (2.1) where the a’s were selected from a standard normal distribution. Note that even though the theoretical mean function is zero for all time points, the facts that the variance increases over time and that the correlation between process values nearby in time is nearly 1 indicate that we should expect long excursions of the process away from the mean level of zero. The simple random walk process provides a good model (at least to a first approximation) for phenomena as diverse as the movement of common stock prices, and the position of small particles suspended in a fluid (so-called Brownian motion).
4 2.2 Means, Variances, and Covariances
Exhibit 2.1 A Random Walk
8
3 RandWalk
-2
Index 10 20 30 40 50 60
A Moving Average
As a second example, suppose that {Zt} is constructed as at + at – 1 Z = ------(2.2.14) t 2 where (as always throughout this book), the a’s are assumed to be independent and σ2 identically distributed with zero mean and variance a . Here () () at + at – 1 Eat + Eat – 1 µ ==EZ() E------=------t t 2 2 = 0 and () () at + at – 1 Var at + Var at – 1 Var() Z ==Var------t 2 4 σ2 = 0.5 a Also
5 Chapter 2 Fundamental Concepts
at + at – 1 at – 1 + at – 2 Cov() Z , Z = Cov------, ------t t – 1 2 2 (), (), (), Cov at at – 1 ++Cov at at – 2 Cov at – 1 at – 1 = ------4 (), Cov at – 1 at – 2 + ------4 (), Cov at – 1 at – 1 =------(as all the other covariances are zero) 4 σ2 = 0.25 a or γ σ2 tt, – 1 =0.25 a for all t (2.2.15) Furthermore,
at + at – 1 at – 2 + at – 3 Cov() Z , Z = Cov------, ------t t – 2 2 2 =0 since the as are independent
Similarly, Cov(Zt, Zt−k) = 0 for k > 1, so we may write 0.5σ2 for ts–0= a γ ts, = σ2 0.25 a for ts–1= 0 for ts–1>
For the autocorrelation function we have 1 for ts–0= ρ , = 0.5 for ts–1= (2.2.16) ts 0 for ts–1>
σ2 σ2 since 0.25a /0.5a =0.5. ρ ρ ρ ρ Notice that 2,1 = 3,2 = 4,3 = 9,8 = 0.5. Values of Z precisely one time unit apart have ρ ρ exactly the same correlation no matter where they occur in time. Furthermore, 3,1 = 4,2 ρ ρ = t,t−2 and, more generally, t,t−k is the same for all values of t. This leads us to the important concept of stationarity.
6 2.3 Stationarity
2.3 Stationarity
To make statistical inferences about the structure of a stochastic process on the basis of an observed record of that process, we must usually make some simplifying (and presumably reasonable) assumptions about that structure. The most important such assumption is that of stationarity. The basic idea of stationarity is that the probability laws that govern the behavior of the process do not change over time. In a sense, the process is in statistical equilibrium. Specifically, a process {Zt} is said to be strictly stationary if the joint distribution of Z ,,,Z … Z is the same as the joint t1 t2 tn ,,,… distribution of Z Z Z for all choices of time points t1, t2,..., tn and all t1 – k t2 – k tn – k choices of time lag k.
Thus, when n = 1 the (univariate) distribution of Zt is the same as that of Zt−k for all t and k; in other words, the Z’s are (marginally) identically distributed. It then follows that E(Zt) = E(Zt−k) for all t and k so that the mean function is constant for all time. Additionally, Va r (Zt) = Va r (Zt−k) for all t and k so that the variance is also constant in time.
Setting n = 2 in the stationarity definition we see that the bivariate distribution of Zt and Zs must be the same as that of Zt−k and Zs−k from which it follows that Cov(Zt , Zs) = Cov(Zt−k , Zs−k) for all t, s, and k. Putting k = s and then k = t, we obtain γ (), ts, = Cov Zts– Z0 (), = Cov Z0 Zst– (), = Cov Z0 Z ts– γ = 0, ts– that is, the covariance between Zt and Zs depends on time only through the time difference |t−s| and not otherwise on the actual times t and s. Thus for a stationary process we can simplify our notation and write γ (), ρ (), k =andCov Zt Ztk– k = Corr Zt Ztk– (2.3.1) Note also that γ ρ k k = γ----- 0 The general properties given in Equation (2.2.5) now become γ () ρ 0 = Var Zt 0 = 1 γ γ ρ ρ k = –k k = –k (2.3.2) γγ≤ ρ ≤ 0 k 1
7 Chapter 2 Fundamental Concepts
If a process is strictly stationary and has finite variance, then the covariance function must depend only on the time lag. A definition that is similar to that of strict stationarity but is mathematically weaker is the following: A stochastic process {Zt} is said to be weakly (or second-order) stationary if 1. The mean function is constant over time, and γ γ 2. tt, – k=0, k for all time t and lag k In this book the term stationary when used alone will always refer to this weaker form of stationarity. However, if the joint distributions for the process are all multivariate normal distributions, it can be shown that the two definitions coincide.
White Noise A very important example of a stationary process is the so-called white noise process which is defined as a sequence of independent, identically distributed random variables {at}. Its importance stems not from the fact that it is an interesting model itself but from the fact that many useful processes can be constructed from white noise. The fact that {at} is strictly stationary is easy to see since Pr() a ≤ x ,,,a ≤…x a ≤ x t1 1 t2 2 tn n =Pr() a ≤ x Pr() a ≤ x …Pr() a ≤ x (by independence) t1 1 t2 2 tn n = Pr() a ≤ x Pr()… a ≤ x Pr() a ≤ x t1 – k 1 t2 – k 2 tn – k n (identical distributions) =Pr() a ≤ x ,,,a ≤…x a ≤ x (by independence) t1 – k 1 t2 – k 2 tn – k n µ as required. Also t = E(at) is constant and Var() a for k = 0 γ t k = 0 for k ≠ 0
Alternatively, we can write 1 for k = 0 ρ = (2.3.3) k 0 for k ≠ 0
The term white noise arises from the fact that a frequency analysis of the model (not considered in this book) shows that, in analogy with white light, all frequencies enter equally. We usually assume that the white noise process has mean zero and denote σ2 Var(at) by a .
8 2.3 Stationarity
The moving average example, page 8, where Zt = (at + at−1)/2, is another example of a stationary process constructed from white noise. In our new notation, we have for the moving average process that 1 for k = 0 ρ = 0.5 for k = ±1 k 0 for k ≥ 2
Random Cosine Wave
As a somewhat different example,† consider the process defined as follows: t Z =forcos 2π ------+ Φ t = 012, ± , ± , … t 12 where Φ is selected (once) from a uniform distribution on the interval from 0 to 1. A sample from such a process will appear highly deterministic, since Zt will repeat itself identically every 12 time units and look like a perfect (discrete time) cosine curve. However, its maximum will not occur at t = 0 but will be determined by the random phase Φ. Still, the statistical properties of this process can be computed as follows: () πt Φ EZ = Ecos 2 ------+ t 12 1 t = cos 2π ------+ φ dφ ∫ 12 0 1 1 t = ------sin 2π ------+ φ 2π 12 φ = 0 1 t t = ------sin2π------+ 2π – sin 2π------2π 12 12 =0 (since the sines must agree) µ So t = 0 for all t. Also
† This example contains optional material that is not needed in order to understand the remainder of this book.
9 Chapter 2 Fundamental Concepts
γ πt Φ πs Φ , = Ecos 2 ------+ cos 2 ------+ ts 12 12
1 t s = cos 2π ------+ φ cos 2π ------+ φ dφ ∫ 12 12 0 1 1 ts– ts+ = --- cos2π ------+ cos 2π ------2+ φ dφ 2∫ 12 12 0 1 1 πts– 1 πts+ φ = ---cos 2 ------+ ------sin 2 ------2+ 212 4π 12 φ = 0
1 ts– = --- cos 2π ------2 12 So the process is stationary with autocorrelation function k ρ =forcos 2π------k = 012, ± , ± , … (2.3.4) k 12
This example suggests that it will be difficult to assess whether or not stationarity is a reasonable assumption for a given time series on the basis of the time plot of the observed data.
The random walk of page 2, where Zt = a1 + a2 +...+ at, is also constructed from white σ2 noise but is not stationary. For example, Var(Zt) = t a is not constant; furthermore, the γ σ2 ≤ ≤ covariance function ts, = t a for 0 t s does not depend only on time lag. However, suppose that instead of analyzing {Zt} directly, we consider the differences of ∇ ∇ − successive Z-values, denoted Zt. Then Zt = Zt Zt−1 = at, so the differenced series, {∇ Zt}, is stationary. This represents a simple example of a technique found to be extremely useful in many applications. Clearly, many real time series cannot be reasonably modeled by stationary processes, since they are not in statistical equilibrium but are evolving over time. However, we can frequently transform nonstationary series into stationary series by simple techniques such as differencing. Such techniques will be vigorously pursued in the remaining chapters.
10 2.4 Exercises
2.4 Exercises
2.1 Suppose E(X) = 2, Var(X) = 9, E(Y) = 0, Va r (Y) =4, and Corr(X,Y) = 0.25. Find: (a) Var(X + Y) (b) Cov(X, X + Y) (c) Corr(X + Y, X − Y) 2.2 If X and Y are dependent but Var(X) = Var(Y), find Cov(X + Y, X − Y) µ σ2 2.3 Let X have a distribution with mean and variance and let Zt = X for all t. (a) Show that {Zt} is strictly and weakly stationary. (b) Find the autocovariance function for {Zt}. (c) Sketch a “typical” time plot of Zt.
2.4 Let {at} be a zero mean white noise processes. Suppose that the observed process is θ θ Zt = at + at−1 where is either 3 or 1/3. θ θ (a) Find the autocorrelation function for {Zt} both when = 3 and when =1/3. (b) You should have discovered that the time series is stationary regardless of the value of θ and that the autocorrelation functions are the same for θ = 3 and θ = 1/3. For simplicity, suppose that the process mean is known to be zero and the variance of Zt is known to be 1. You observe the series {Zt} for t = 1, 2,..., n and ρ suppose that you can produce good estimates of the autocorrelations k. Do you think that you could determine which value of θ is correct (3 or 1/3) based on ρ the estimate of k? Why or why not?
2.5 Suppose Zt = 5 + 2t + Xt where {Xt} is a zero mean stationary series with γ autocovariance function k. (a) Find the mean function for {Zt}. (b) Find the autocovariance function for {Zt}. (c) Is {Zt} stationary? (Why or why not?)
Xt for t odd 2.6 Let {Xt} be a stationary time series and define Zt = Xt + 3 for t even (), (a) Show that Cov Zt Ztk– is free of t for all lags k. (b) Is {Zt} stationary?
11 Chapter 2 Fundamental Concepts
γ 2.7 Suppose that {Zt} is stationary with autocovariance function k. ∇ − (a) Show that Wt = Zt = Zt Zt−1 is stationary by finding the mean and autocovariance function for {Wt}. ∇2 ∇ − − + (b) Show that Ut = Zt = [Zt Zt−1] = Zt 2Zt−1 Zt−2 is stationary. (You need not find the mean and autocovariance function for {Ut}.) γ 2.8 Suppose that {Zt} is stationary with autocovariance function k. Show that for any fixed positive integer n and any constants c1, c2,..., cn , the process {Wt} defined by Wt = c1Zt + c2Zt−1 + ... + cnZt−n−1 is stationary. (Note that Exercise (2.7) is a special case of this result.) β β 2.9 Suppose Zt = 0 + 1t + Xt where {Xt} is a zero mean stationary series with γ β β autocovariance function k and 0 and 1 are constants. ∇ − (a) Show that {Zt} is not stationary but that Wt = Zt = Zt Zt−1 is stationary. µ (b) In general, show that if Zt = t + Xt where {Xt} is a zero mean stationary series µ ∇m ∇ ∇m−1 and t is a polynomial in t of degree d, then Zt = ( Zt) is stationary for m ≥ d and nonstationary for 0 ≤ m < d.
2.10 Let {Xt} be a zero-mean, unit-variance stationary process with autocorrelation ρ µ σ function k. Suppose that t is a nonconstant function and that t is a positive-valued nonconstant function. The observed series is formed as µ σ Zt = t + tXt (a) Find the mean and covariance function for the {Zt} process. (b) Show that autocorrelation function for the {Zt} process depends only on time lag. Is the {Zt} process stationary? (c) Is it possible to have a time series with a constant mean and with Corr(Zt,Zt−k) free of t but with {Zt} not stationary? γ 2.11 Suppose Cov(Xt,Xt−k) = k is free of t but that E(Zt) = 3t. (a) Is {Xt} stationary? − (b) Let Zt = 7 3t + Xt. Is {Zt} stationary? − 2.12 Suppose that Zt = at at−12. Show that {Zt} is stationary and that, for k > 0, its autocorrelation function is nonzero only for lag k = 12. − θ 2 2.13 Let Zt = at (at−1) . (a) Find the autocorrelation function for {Zt}. (b) Is {Zt} stationary?
12 2.4 Exercises
2.14 Evaluate the mean and covariance function for each of the following processes. In each case determine whether or not the process is stationary. θ (a) Zt = 0 + tat ∇ (b) Wt = Zt where Zt is as given in part (a). (c) Zt = at at−1 2.15 Suppose that X is a random variable with zero mean. Define a time series by − t Zt =( 1) X. (a) Find the mean function for {Zt}. (b) Find the covariance function for {Zt}. (c) Is {Zt} stationary?
2.16 Suppose Zt = A + Xt where {Xt} is stationary and A is random but independent of {Xt}. Find the mean and covariance function for {Zt} in terms of the mean and autocovariance function for {Xt} and the mean and variance of A.
γ 1 n 2.17 Let {Zt} be stationary with autocovariance function k. Let Z = ---∑ Z . Show n t = 1 t that γ n – 1 0 2 k Var() Z = ----- + --- 1 – --- γ n n ∑ n k k = 1
n – 1 1 k = --- 1 – ----- γ n ∑ n k kn= –1+ γ 2.18 Let {Zt} be stationary with autocovariance function k. Define the sample variance n 1 as S2 = ------()Z – Z 2 . n – 1 ∑ t t = 1 n n ()µ 2 ()2 ()µ 2 (a) First show that ∑ Zt – = ∑ Zt – Z + nZ– . t = 1 t = 1 (b) Use part (a) to show that n – 1 n n 2 k ES()2 ==------γ – ------Var() Z γ – ------1 – --- γ . n – 1 0 n – 1 0 n – 1 ∑ n k k = 1 (Use the results of Exercise (2.17) for the last expression.) γ 2 γ (c) If {Zt} is white noise with variance 0, show that E(S ) = 0.
13 Chapter 2 Fundamental Concepts
θ θ 2.19 Let Z1 = 0 + a1 and then for t > 1 define Zt recursively by Zt = 0 + Zt−1 + at. Here θ 0 is a constant. The process {Zt} is called a random walk with drift. θ (a) Show that Zt may be rewritten as Zt = t 0 + at + at−1 +...+a1. (b) Find the mean function for Zt. (c) Find the autocovariance function for Zt.
2.20 Consider the standard random walk model where Zt = Zt−1 + at with Z1 = a1. µ µ (a) Use the above representation of Zt to show that t = t−1 for t > 1 with initial µ µ condition 1 = E(a1) =0. Hence show that t = 0 for all t. σ2 σ2 (b) Similarly, show that Var(Zt) = Var(Zt−1) + a , for t > 1 with Var(Z1) = a , and, σ2 hence Var(Zt) = t.a ≤ ≤ (c) For 0 t s, use Zs = Zt + at+1 + at+2 +...+ as to show that Cov(Zt, Zs) = Var(Zt) σ2 and, hence, that Cov(Zt, Zs) = min(t, s) a .
2.21 A random walk with random starting value. Let Zt = Z0 + at + at−1 +...+a1 for t > 0 µ σ2 where Z0 has a distribution with mean 0 and variance 0 . Suppose further that Z0, a1, ..., at are independent. µ (a) Show that E(Zt) = 0 for all t. σ2 σ2 (b) Show that Var(Zt) = t a +.0 σ2 σ2 (c) Show that Cov(Zt, Zs) = min(t, s) a + 0 . σ2 σ2 t a + 0 (d) Show that Corr() Z , Z =for------0≤≤ts . t s σ2 σ2 s a + 0
2.22 Let {at} be a zero-mean white noise process and let c be a constant with |c| < 1. Define Zt recursively by Zt = cZt−1 + at with Z1 = a1. (a) Show that E(Zt) = 0. σ2 2 4 2t−2 (b) Show that Var(Zt) = a (1 + c +c +...+ c ). Is {Zt} stationary? (c) Show that Var() Z (), t – 1 Corr Zt Zt – 1 = c ------()- Var Zt and, in general, Var() Z (), k tk– > Corr Zt Ztk– =forc ------() k 0 Var Zt
14 2.4 Exercises
(d) For large t, argue that σ2 a Var() Z ≈ ------and Corr() Z , Z ≈ ck for k > 0 t 1 – c2 t tk–
so that {Zt} could be called asymptotically stationary. a1 (e) Suppose now that we alter the initial condition and put Z1 = ------. Show 1 – c2 that now {Zt} is stationary.
2.23 Two processes {Zt} and {Yt} are said to be independent if for any time points t1, ,,,… t2,..., tm and s1, s2,..., sn, the random variables {Z Z Z } are independent t1 t2 tm ,,,… of the random variables {Z Z Z }. Show that if {Zt} and {Yt} are s1 s2 sn independent stationary processes, then Wt = Zt + Yt is stationary.
2.24 Let {Xt} be a time series in which we are interested. However, because the measurement process itself is not perfect, we actually observe Zt = Xt + at. We assume that {Xt} and {at} are independent processes. We call Xt the signal and at the measurement noise or error process. ρ If {Xt} is stationary with autocorrelation function k, show that {Zt} is also stationary with ρ k Corr() Z , Z =for------k ≥ 1 t tk– σ2 ⁄ σ2 1 + a X
σ2 ⁄ σ2 We call X a the signal-to-noise ratio, or SNR. Note that the larger the SNR, the closer the autocorrelation function of the observed process {Zt} is to the autocorrelation function of the desired signal {Xt}.
k β []()π ()π β 2.25 Suppose Zt = 0 + ∑ Ai cos 2 fit + Bi sin 2 fit where 0, f1, f2,..., fk are i = 1
constants and A1, A2,..., Ak, B1, B2,..., Bk are independent random variables with σ2 zero means and variances Va r (Ai) = Var(Bi) = i . Show that {Zt} is stationary and find its covariance function.
15 Chapter 2 Fundamental Concepts
1 2.26 Define the function Γ = ---EZ[]()– Z 2 . In geostatistics, Γ is called the ts, 2 t s t,s semivariogram. Γ γ γ (a) Show that for a stationary process ts, = 0 – ts– . Γ (b) A process is said to be intrinsically stationary if t,s depends only on the time difference |t−s|. Show that the random walk process is intrinsically stationary. 2.27 For a fixed, positive integer r and constant φ, consider the time series defined by φ φ2 φr Zt = at + at−1 + at−2 +...+ at−r . (a) Show that this process is stationary for any value of φ. (b) Find the autocorrelation function. 2.28 (Mathematical statistics required) Suppose that []π()Φ , ± , ± , … Zt =forAcos 2 ft + t = 012 where A and Φ are independent random variables and f is a constant—the frequency. The phase Φ is assumed to be uniformly distributed on (0,1), and the amplitude A 2 ⁄ has a Rayleigh distribution with pdf fa()= ae–a 2 for a > 0. (a) Show that {Zt} is stationary. (b) Show that for each time point t, Zt has a normal distribution. (It can also be shown that all of the finite dimensional distributions are multivariate normal so that the process is strictly stationary.) (Hint: Let XA= cos[]2π()ft + Φ and YA= sin[]2π()ft + Φ and find the joint distribution of X and Y.)
2.5 Appendix: Expectation, Variance, Covariance, and Correlation
In this appendix we define expectation for continuous random variables. However, all of the properties described hold for all types of random variables. Let X have probability density function f(x) and let the pair (X,Y) have joint probability density function f(x,y). ∞ The expected value of X is defined as EX()= ∫ xf() x dx . –∞ ∞ (If ∫ xf() x dx < ∞ ; otherwise E(X) is undefined.) E(X) is also called the expectation –∞ µ µ of X or the mean of X and is often denoted or X.
16 2.5 Appendix: Expectation, Variance, Covariance, and Correlation
Properties of Expectation ∞ If h(x) is a function such that ∫ hx()fx()dx < ∞ , then –∞ ∞ EhX()() = ∫ hx()fx()dx (2.A.1) –∞ ∞ ∞ If ∫ ∫ hxy(,)fxy(), dyxd < ∞ , then –∞ –∞ ∞ ∞ EhXY[](), = ∫ ∫ hxy(), fxy(), dyxd (2.A.2) –∞ –∞ As a corollary to Equation () we have the important result EaX()++ bY c = aE() X ++bE() Y c (2.A.3)
∞ ∞ Also EXY()= ∫ ∫ xyf() x, y dyxd (2.A.4) –∞ –∞ The variance of a random variable X is defined as
Var() X = EXEX{}[]– ()2 (2.A.5)
2 σ2 σ2 . (provided E(X ) exists.) The variance of X is often denoted or X
Properties of Variance
Var() X ≥ 0 (2.A.6)
Var() a+ bX = b2Var() X (2.A.7)
If X and Y are independent, then Var() X+ Y = Var() X + Var() Y (2.A.8)
Var() X = EX()2 – []EX()2 (2.A.9) The positive square root of the variance of X is called the standard deviation of X and is σ σ − µ σ often denoted or X. The random variable (X X)/ X is called the standardized version of X. The mean and standard deviation of a standardized variable are always zero and one, respectively. (), []()µ ()µ The covariance of X and Y is defined as Cov X Y = EX– X Y – Y .
17 Chapter 2 Fundamental Concepts
Properties of Covariance
If X and Y are independent Cov() X, Y = 0 . (2.A.10)
Cov() a+ bX, cdY+ = bdCov() X, Y (2.A.11)
Var() X+ Y = Var() X ++Var() Y 2Cov() X, Y (2.A.12)
Cov() X+ Y, Z = Cov() X, Z + Cov() Y, Z (2.A.13)
Cov() X, X = Var() X (2.A.14)
Cov() X, Y = Cov() Y, X (2.A.15) The correlation coefficient of X and Y, denoted Corr(X,Y) or ρ, is defined as Cov() X, Y Corr() X, Y = ------Var() X Var() Y Alternatively, if X* is a standardized X and Y* is a standardized Y, then ρ = E(X*Y*).
Properties of Correlation
–1 ≤≤Corr() X, Y 1 (2.A.16)
Corr() a+ bX, cdY+ = sign() bd Corr() X, Y
1 if bd > 0 (2.A.17) where sign() bd = 0 if bd = 0 –1 if bd < 0
Corr(X,Y) = ±1 if, and only if there are constants a and b such that Pr(Y = a + bX) = 1. (2.A.18)
18