c

1997 by Jay H. Lee, Jin Hoon Choi, and Kwang Soon Lee

Chapter 5

STOCHASTIC PROCESSES

A sto chastic pro cess refers to a family of random variables indexed by a

parameter set. This parameter set can be continuous or discrete. Since we

are interested in discrete systems, we will limit our discussion to pro cesses

with a discrete parameter set. Hence, a sto chastic pro cess in our context is

a time sequence of random variables.

5.1 BASIC PROBABILITY CONCEPTS

5.1.1 DISTRIBUTION FUNCTION

Let xk  be a sequence. Then, xk ;


;xk  form an `-dimensional

1 `

random variable. Then, one can de ne the nite dimensional distribution

function and the density function as b efore. For instance, the distribution

function F  ;


; ; xk ;


;xk , is de ned as:

1 ` 1 `

F  ;


; ; xk ;


;xk  = Prfxk    ;


;xk    g 5.1

1 ` 1 ` 1 1 ` `

The density function is also de ned similarly as b efore.

We note that the ab ove de nitions also apply to vector time sequences if 102

c

1997 by Jay H. Lee, Jin Hoon Choi, and Kwang Soon Lee

xk  and  's are taken as vectors and each integral is de ned over the

i i

space that  o ccupies.

i

5.1.2 MEAN AND COVARIANCE

Mean value of the sto chastic variable xk  is

Z

1

xk  = E fxk g = dF ; xk  5.2

1

Its covariance is de ned as

T

R k ;k  = E f[xk  xk ][xk  xk ] g

x 1 2 1 1 2 2

R R

1 1

T

[ xk ][ xk ] dF  ; ; xk ;xk  =

1 1 2 2 1 2 1 2

1 1

5.3

The cross-covariance of two sto chastic pro cesses xk  and y k  are de ned as

T

R k ;k  = E f[xk  xk ][y k  yk ] g

xy 1 2 1 1 2 2

R R

1 1

T

[ xk ][ yk ] dF  ; ; xk ;yk  =

1 1 2 2 1 2 1 2

1 1

5.4

Gaussian processes refer to the pro cesses of which any nite-dimensional

distribution function is normal. Gaussian pro cesses are completely

characterized by the mean and covariance.

5.1.3 STATIONARY STOCHASTIC PROCESSES

Throughout this book we will de ne stationary sto chastic pro cesses as those

with time-invariant distribution function. Weakly stationary or stationary

in a wide sense pro cesses are pro cesses whose rst two moments are 103

c

1997 by Jay H. Lee, Jin Hoon Choi, and Kwang Soon Lee

time-invariant. Hence, for a weakly stationary pro cess xk ,

E fxk g = x 8k

5.5

T

E f[xk  x][xk   x] g = R   8k

x

In other words, if xk  is stationary, it has a constant mean value and its

covariance dep ends only on the time di erence  . For Gaussian pro cesses,

weakly stationary pro cesses are also stationary.

For scalar xk , R 0 can be interpreted as the variance of the signal and

R   R  

reveals its time correlation. The normalized covariance ranges from

R 0 R 0

0 to 1 and indicates the time correlation of the signal. The value of 1

indicates a complete correlation and the value of 0 indicates no correlation.

Note that many signals have b oth deterministic and sto chastic comp onents.

In some applications, it is very useful to treat these signals in the same

framework. One can do this by de ning

P

1

N

x = lim xk 

N !1

k =1

N

5.6

P

1

N T

R   = lim [xk  x ][xk   x]

x N !1

k =1

N

Note that in the ab ove, b oth deterministic and sto chastic parts are

averaged out. The signals for which the ab ove limits converge are called

\quasi-stationary" signals. The ab ove de nitions are consistent with the

previous de nitions since,in the purely sto chastic case, a particular

realization of a stationary sto chastic pro cess with given mean x and

covariance R   should satisfy the ab ove relationships.

x

5.1.4 SPECTRA OF STATIONARY STOCHASTIC

PROCESSES

Throughout this chapter, continuous time is rescaled so that each discrete

time interval represents one continuous time unit. If the sample interval T

s 104

c

1997 by Jay H. Lee, Jin Hoon Choi, and Kwang Soon Lee

is not one continuous time unit, the frequency in discrete time needs to be

1

scaled with the factor of .

T

s

Sp ectral density of a stationary pro cess xk  is de ned as the Fourier

transform of its covariance function:

1

X

1

j !

 ! = R  e 5.7

x x

2

 =1

Area under the curve represents the power of the signal for the particular

frequency range. For example, the power of xk  in the frequency range

! ;!  is calculated by the integral

1 2

Z

! =!

2

2
 ! d!

x

! =!

1

Peaks in the signal sp ectrum indicate the presence of p erio dic comp onents

in the signal at the resp ective frequency.

The inverse Fourier transform can be used to calculate R   from the

x

sp ectrum  !  as well

x

Z



j !

 ! e d! 5.8 R   =

x x



With  = 0, the ab ove b ecomes

Z



T

E fxk xk  g = R 0 =  ! d! 5.9

x x



which indicates that the total area under the sp ectral density is equal to the

variance of the signal. This is known as the Parseval's relationship.

Example: Show plots of various covariances, sp ectra and realizations!

**Exercise: Plot the sp ectra of 1 white noise, 2 sinusoids, and 3white

noise ltered through a low-pass lter. 105

c

1997 by Jay H. Lee, Jin Hoon Choi, and Kwang Soon Lee

5.1.5 DISCRETE-TIME WHITE NOISE

A particular typ e of a sto chastic pro cess called white noise will be used

extensively throughout this b o ok. xk  is called a white noise or white

sequence if

P xk jx` = P xk  for ` < k 5.10

for all k . In other words, the sequence has no time correlation and hence all

the elements are mutually indep endent. In such a situation, knowing the

realization of x` in no way helps in estimating xk .

A stationary white noise sequence has the following prop erties:

E fxk g = x 8k

8

>

<

5.11

R if  = 0

x

T

E fxk  xxk   x g =

>

:

0 if  6= 0

Hence, the covariance of a white noise is de ned by a single matrix.

The sp ectrum of white noise xk  is constant for the entire frequency range

since from 5.7

1

 !  = R 5.12

x x

2

The name \white noise" actually originated from its similarity with white

light in sp ectral prop erties.

5.1.6 COLORED NOISE

A sto chastic pro cess generated by ltering white noise through a dynamic

system is called \colored noise."

Imp ortant: 106

c

1997 by Jay H. Lee, Jin Hoon Choi, and Kwang Soon Lee

A stationary sto chastic pro cess with any given mean and

covariance function can be generated by passing a white noise

through an appropriate dynamical system.

To see this, consider



dk = H q "k + d 5.13

where "k  is a white noise of identity covariance and H q  is a stable /

stably invertible transfer function matrix. Using simple algebra Ljung

-REFERENCE, one can show that

j! T j!

 !  = H e H e  5.14

d



The sp ectral factorization theorem REFERENCE - Astrom and

Wittenmark, 1984 says that one can always nd H q  that satis es 5.14

for an arbitrary  and has no p ole or zero outside the unit disk. In other

d

words, the rst and second order moments of any stationary signal can be

matched by the ab ove mo del.

This result is very useful in mo deling disturbances whose covariance

functions are known or xed. Note that a stationary Gaussian pro cess is

completely sp eci ed by its mean and covariance. Such a pro cess can be

mo delled by ltering a zero-mean Gaussian white sequence through

appropriate dynamics determined by its sp ectrum plus adding a bias at the

output if the mean is not zero. 107

c

1997 by Jay H. Lee, Jin Hoon Choi, and Kwang Soon Lee

5.1.7 INTEGRATED WHITE NOISE AND

NONSTATIONARY PROCESSES

Some pro cesses exhibit mean-shifts whose magnitude and o ccurence are

random. Consider the following mo del:

y k  = y k 1 + "k 

where "k  is a white sequence. Such a sequence is called integrated white

noise or sometimes random walk. Particular realizations under di erent

distribution of "k  are shown b elow:

P(ζ)

y(k)

90% 10%

More generally, many interesting signals will exhibit stationary b ehavior

combined with randomly o ccuring mean-shifts. Such signals can be mo deled

as 108

c

1997 by Jay H. Lee, Jin Hoon Choi, and Kwang Soon Lee

ε ( ) 2 k ~ − Hq()1

ε ( ) 1 k 1 ++ 1q− −1 y(k)

ε( ) k 1 −1 − −1 Hq()

1q y(k)

As shown ab ove, the combined e ects can be expressed as an integrated

1

white noise colored with a lter H q .

Note that while y k  is nonstationary, the di erenced signal
y k  is

stationary.

∆ ε( ) y(k) ε( ) y(k) k 1 −1 k −1 −1 () ()

1q− Hq Hq

5.1.8 STOCHASTIC DIFFERENCE EQUATION

Generally, a sto chastic pro cess can be mo deled through the following

sto chastic di erence equation.

xk +1 = Axk + B"k 

5.15

y k  = Cxk + D"k 

where "k  is a white vector sequence of zero mean and covariance R .

"

Note that

k

E fxk g = AE fxk 1g = A E fx0g

5.16

T T T T

E fxk x k g = AE fxk 1x k 1gA + BR B

" 109

c

1997 by Jay H. Lee, Jin Hoon Choi, and Kwang Soon Lee

If all the eigenvalues of A are strictly inside the unit disk, the ab ove

approaches a stationary pro cess as k ! 1 since

lim E fxk g = 0

k !1

5.17

T

lim E fxk x k g = R

k !1 x

where R is a solution to the Lyapunov equation

x

T T

R = AR A + BR B 5.18

x x "

Since y k = Cxk + D"k ,

E fy k g = CEfxk g + DE f"k g = 0

T T T T T T T

E fy k y k g = CEfxk x k gC + DE f"k " k gD = CR C + DR D

x "

5.19

The auto-correlation function of y k  b ecomes

8

>

T T

<

CR C + DR D for  = 0

x "

T

R   = E fy k +  y k g =

y

>

 T  1 T

:

CA R C + CA BR D for  > 0

x "

5.20

The sp ectrum of w is obtained by taking the Fourier transform of R  

y

and can be shown to be

T

j! 1 j! 1

C e I A B + D 5.21  !  = C e I A B + D R

y "

In the case that A contains eigenvalues on or outside the unit circle, the

pro cess is nonstationary as its covariance keeps increasing see Eqn. 5.16.

However, it is common to include integrators in A to mo del mean-shifting

random-walk-like b ehavior. If all the outputs exhibit this b ehavior, one

can use

xk +1 = Axk + B"k 

5.22

y k  = Cxk + D"k 

Note that, with a stable A, while
y k  is a stationary pro cess, y k  110

c

1997 by Jay H. Lee, Jin Hoon Choi, and Kwang Soon Lee

includes an integrator and therefore is nonstationary.

Stationary process Nonstationary(Meanshifting) process

ε(κ) y(k) ε(κ) ∆y(k) 1 y(k) x(k+1)=Ax(k)+B ε(k) x(k+1)=Ax(k)+B ε(k) y(k)=Cx(k)+D ε(k) y(k)=Cx(k)+D ε(k) 1-q -1

stable system stable system integrator

5.2 STOCHASTIC SYSTEM MODELS

Mo dels used for control will often include b oth deterministic and sto chastic

inputs. The deterministic inputs corresp ond to known signals like

manipulated variables. The sto chastic signals cover whatever remaining

parts that cannot be predicted a priori. They include the e ect of

disturbances, other pro cess variations and instrumentation errors.

5.2.1 STATE-SPACE MODEL

The following sto chastic di erence equation may be used to characterize a

sto chastic disturbance:

xk +1 = Axk + Buk + " k 

1

5.23

y k  = Cxk + " k 

2

" k  and " k  are white noise sequences that represent the e ects of

1 2

disturbances, measurement error, etc. They may or may not be correlated.

 If the ab ove mo del is derived from fundamental principles, "k  may be

a signal used to generate physical disturbance states which are 111

c

1997 by Jay H. Lee, Jin Hoon Choi, and Kwang Soon Lee

included in the state x or merely arti cal signals added to represent

random errors in the state equation. " k  may be measurement noise

2

or signals representing errors in the output equations.

 If the mo del is derived on an empirical basis, " k  and " k  together

1 2

represent the combined e ects of the pro cess / measurement

randomness. In other words, the output is viewed as a comp osite of

two signals y k  = y k +y k , one of which is the output of the

d s

deterministic system

xk +1 = Axk + Buk 

5.24

y k  = Cxk 

d

and the other is the random comp onent

xk +1 = Axk + " k 

1

5.25

y k  = Cxk + " k 

s 2

With such a mo del available, one problem treated in statistics is to predict

future states, xk + i;i  0 given collected output measurements

y k ;


;y1. This is called state estimation and will be discussed in the

next chapter.

The other problem is building such a mo del. Given data

y i;ui;i = 1;


;N, the following two metho ds are available.

 One can use the so called subspace identi cation metho ds.

 One can build a time series mo del or more generally a transfer function

mo del of the form

1 1

y k  = Gq uk + H q "k 

Then, one can p erform a state-space realization of the ab ove to obtain

the state-space mo del. 112

c

1997 by Jay H. Lee, Jin Hoon Choi, and Kwang Soon Lee

Details of these metho ds will be discussed in the identi cation chapter.

For systems with mean-shifting nonstationary disturbances in all output

channels, it may be more convenient to express the mo del in terms of the

di erenced inputs and outputs:

xk +1 = Axk + B
uk + " k 

1

5.26

y k  = Cxk + " k 

2

If the undi erenced y is desired as the output of the system, one can simply

rewrite the ab ove as

3 3 2 3 2 3 2 3 2 3 2 2

0 I B xk  A 0 xk +1

7 7 6 7 6 7 6 7 6 7 6 6

5 5 4 5 4 5 4 5 4 5 4 4

" k  " k +
uk + + =

2 1

I C CB y k  CA I y k +1

2 3

 

xk 

6 7

4 5

y k  =

0 I

y k 

5.27 113

c

1997 by Jay H. Lee, Jin Hoon Choi, and Kwang Soon Lee

Physio-Chemical I/D Experiment Principles Historical Data Base

Nonlinear ODE's I/O Data

zf = f(z f,uf,w f) u(1), ,u(N)

yf = f(z f) y(1), ,y(N)

Linearization Parameteric + ID(PEM) Discretization

Linear Difference Eqns: Discrete Transfer Functions

z(k+1)=Az(k)+B u(k)+B w(k) -1 -1 u w y(k)=G(q ,θ)u(k)+H(q ,θ )e(k) y(k)=Cz(k) Subspace I D

Augmentation Stochastic States Realization z x w

ε x(k+1) = Ax(k) + Bu(k) + 1(k) ε

y(k) = Cx(k) + 2(k)

5.2.2 INPUT-OUTPUT MODELS

One can also use input-output mo dels. A general form is

y k  = Gq uk + H q "k  5.28 114

c

1997 by Jay H. Lee, Jin Hoon Choi, and Kwang Soon Lee

Within the ab ove general structure, di erent parameterizations exist. For

instance, a p opular mo del is the following ARMAX AR for

Auto-Regressive, MA for Moving-Average and X for eXtra input pro cess:

1 n 1 1 m

y k  = I + A q +


+ A q  B q +


+ B q uk 

1 n 1 m

1 n 1 1 n

+I + A q +


+ A q  I + C q +


+ C q "k 

1 n 1 n

5.29

Note that the ab ove is equivalent to the following linear time-series

equation:

y k  = A y k 1 A y k 2


A y k n

1 2 n

5.30

+B uk 1 +


+ B uk m

1 m

+"k +C "k 1 +


+ C "k n

1 n

In most practical applications, matrices A 's and C 's are restricted to be

i i

diagonal, which results in a MISO rather than a MIMO structure. In such

a case, sto chastic comp onents for di erent output channels are restricted to

be mutual ly independent.

For systems with integrating typ e disturbances in all output channels, a

more appropriate mo del form is

1

y k = Gq uk + H q "k  5.31

1

1 q

The ab ove can be easily rewritten as

y k = Gq 
uk + H q "k  5.32 115