RS – 4 – Jointly Distributed RV (a)

Chapter 4 Jointly distributed Random variables

Continuous Multivariate distributions

Continuous Random Variables

1 RS – 4 – Jointly Distributed RV (a)

Joint Probability Density Function (pdf)

Definition: Joint Probability density function Two random variable are said to have joint probability density function f(x,y) if

1. 0 f  xy , 

 2. f x , y dxdy  1  

3. P  X , Y A  f x , y dxdy A

Marginal and Condition Density

Definition: Marginal Density Let X and Y denote two RVs with joint pdf f(x,y), then the marginal density of X is  f xfxydy , X      f yfxydx , and the marginal density of Y is Y    

Definition: Conditional Density Let X and Y denote two RVs with joint pdf f(x,y) and marginal

densities fX(x), fY(y), then the conditional density of Y given X = x and the conditional density of X given Y = y are given by f  xy,  f  xy,  fyxYX  fxyXY  f X x f Y y

2 RS – 4 – Jointly Distributed RV (a)

Joint MGF Definition: MGF of (X,Y) Let X and Y be two RVs with joint pdf f(x,y) then the MGF of X  Y: m XY (t1,t2 )  E[exp(t1 X  t2Y )]   exp(t1 X  t2Y ) f (x, y)dxdy R 2 Theorem: The MGF of a pair of independent RVs is the product of the MGF of the corresponding marginal distributions. That is,

mXY(t1,t2) = mX(t1) mY(t2). Proof: m XY (t1 , t 2 )   exp( t1 X  t 2Y ) f (x, y)dxdy   exp( t1 X ) exp( t 2Y ) f (x) f ( y)dxdy   exp( t1 X ) f (x)dx  exp( t 2Y ) f ( y)dy  m X (t1 )mY (t 2 )

Marginal MGF

Definition: MGF of the marginal distribution of X (andY)

Let mXY(t1,t2) be the MGF of (X,Y), then the MGF of the marginal distributions of X and Y are, respectively, mXY(t1,0) and mXY(0,t2).

Proof:    m X (t)   exp(tX ) f X (x)dx  exp(tX )[ f XY (x, y)dy]dx    

  exp(tX ) f XY (x, y)dydx  m XY (t,0) Similar derivation for Y.

3 RS – 4 – Jointly Distributed RV (a)

The bivariate Normal distribution

Sir Francis Galton (1822 –1911, England)

The bivariate normal distribution

Let the joint distribution be given by: 1 1  Qx12, x fxx, e2 12 2 2112   where 2 2 xxxx 112  11 2 2 2 2 1122 Qxx,   12 1   2

This distribution is called the bivariate Normal distribution.

The parameters are 1, 2 , 1, 2 and  The properties of this distribution were studied by Francis Galton and discovered its relation to the regression, term Galton coined.

4 RS – 4 – Jointly Distributed RV (a)

The bivariate normal distribution Surface Plots of the bivariate Normal distribution

The bivariate normal distribution Note: We can have a more compact joint using linear algebra:

2 1  1  x   x   x   x     f (x , x )  exp ( 1 1 )2  2( 1 1 )( 2 2 )  ( 2 2 )2   1 2 2 2(1  2 )       21 2 (1  )   1 1 2 2   1  1   exp (x μ)'1(x μ) 2(|  |)1/2  2 

(1) Determine the inverse and determinant of Σ (the covariance matrix)

2 2 1 21 2 2 2 2 2 12 2 2 2    2  | |1 2 12 1 2 (1 2 2 ) 1 2 (1  ) 12 2  1 2 2 1 1  2 21    2  | | 12 1 

5 RS – 4 – Jointly Distributed RV (a)

The bivariate normal distribution

(2) Write a quadratic form for Q(x1,x2):

2 2 xxxx 112  11 2 2 2 2 1122 Qxx,   12 1   2

1 Q(x1, x2 )  (x μ)' (x μ) 2 1   2  21 x1  1   (x1  1) (x2  2 )  2    |  | 12 1 x2  2 

1 2 2  x1  1   (x1  1) 2 (x2  2 )12 (x1  1) 21 (x2  2 )1   |  | x2  2  1  ((x   )2 2  2 (x   )(x   )  (x   )2 2 ) |  | 1 1 2 21 1 1 2 2 2 2 1 2 2 2 2 ((x1  1)  2  2 21(x1  1)(x2  2 )  (x2  2 ) 1 )  2 2 2 1  2 (1  )

The Bivariate Normal Distribution

f(x,y)

12 12  y 12 y y

   x x x Contour Plots of the Bivariate Normal Distribution

yyy 12 12  12  

2 2 2

x xx 1 1 1 Scatter Plots of data from the Bivariate Normal Distribution

yyy 12  12   12 

2 2 2

x xx 1 1 1

6 RS – 4 – Jointly Distributed RV (a)

The bivariate normal distribution: MGF

Let MGF of a bivariate normal is given by: 1 m (t ,t )  exp[t   t   (t 2 2  t 2 2  2 t t   )] XY 1 2 1 X 2 Y 2 1 X 2 Y XY 1 2 X Y

Note: When ρXY = 0 –i.e., X and Y are independent. The MGF is: 1 m (t ,t )  exp[t   t   (t 2 2  t 2 2 )] XY 1 2 1 X 2 Y 2 1 X 2 Y

Marginal distributions for the Bivariate Normal

Recall the definition of marginal distributions for continuous RV:   f xfxxdx , and f xfxxdx , 11  12  2 22  12  1   In the case of the bivariate normal distribution the marginal

distribution of xi is Normal with mean i and standard deviation i. Proof:

The marginal distributions of x2 is   1 1  Qx12, x f xfxxdx ,   e2 dx 22 12 1 2  1  2112    where 22 xxxx 112  11 2 2 2 2 1122 Qxx,   12 1   2

7 RS – 4 – Jointly Distributed RV (a)

Marginal distributions for the Bivariate Normal

Now: 22 xxxx 112  11 2 2 2 2 1122 Qxx,   12 1   2

2 22 xa11 x aa cxc222 1 2 bbbb xx2    11222    x 1122   1  21   21

2  2 xx 1 222 22 + 22 21 22 1212111 

Marginal distributions for the Bivariate Normal

22 2 Hence bb 1 or 1 1

a  x   122 Also b2  22  1121

1  1  12x   2   1  2  and 1 ax12   2

8 RS – 4 – Jointly Distributed RV (a)

Marginal distributions for the Bivariate Normal

Finally 2 a 2  2 xx c 1 222  22 b 2 22 21 22  1212111 

2  2 xxa 2 c 1 2 22  22 22 21 22b 2 1212111  

2  2 xx 1 222  22 22 21 22  1212111  2    1 x   12    2   2  1 

Marginal distributions for the Bivariate Normal

and 2 1  2 112 cxx12 122 2  22  22  1 1  22 2   1  122x      2   2 1 1 2 2   2 1 x22 22 1 1  2 

2 x    22  2

9 RS – 4 – Jointly Distributed RV (a)

Marginal distributions for the Bivariate Normal

Summarizing 22 xxxx 112 11 2 2 2 2 1122 Qxx,   12 1  2 2 xa1  c b

2 where b 1 1

1 ax12   2 2 x   and c  22  2

Marginal distributions for the Bivariate Normal

 f xfxxdx , Thus 22  12  1   1 1  Qx12, x  e2 dx 2  1 2112   

2 1 xa1   c 1 2 b  edx 2  1 2112   

2  c 2  1 xa1  21 be    e2 b dx 2  2 b 1 2112   

2 1 x   1  22  e 2  2 22

10 RS – 4 – Jointly Distributed RV (a)

Marginal distributions for the Bivariate Normal

• Thus the marginal distribution of x2 is Normal with mean 2 and standard deviation 2.

• Similarly, the marginal distribution of x1 is Normal with mean 1 and standard deviation 1.

Note: This derivation is much easier using MGFs. Use the MGF of a bivariate normal. To get the MGF of the marginal of

X, set t2=0. 1 m (t ,t )  exp[t   t   (t 2 2  t 2 2  2 t t   )] XY 1 2 1 X 2 Y 2 1 X 2 Y XY 1 2 X Y 1 m (t ,0)  exp[t   (t 2 2 )]  m (t ) XY 1 1 X 2 1 X X 1

Marginal distributions: Bivariate Normal

Bivariate Normal Distribution with marginal distributions

11 RS – 4 – Jointly Distributed RV (a)

Conditional distributions for the Bivariate Normal

Recall the definition of conditional distributions for continuous RVs:

f  xx12,  f  xx12,  fxx12 12 and fxx21 21 f22x f11x In the case of the bivariate normal distribution the conditional

distribution of xi given xj is Normal with mean and standard deviation:  i 2 ij ijjx   and  ij i 1  j

Conditional distributions: Bivariate Normal

1 Proof: |  Qx12, x e 2 2 f (x1, x2 ) 2112   f   2 1|2 1 x   22 f (x2 ) 1  e 2 2 22

2 2 2 11x  11xa122 x  Qx, x | 22 c |  12 22b  ee22 2  2  22 2111  21   where 2  x   2 a    1 (x  ) c  22 b 1 1 1 2 2  2  2

12 RS – 4 – Jointly Distributed RV (a)

Conditional distributions: Bivariate Normal

2 1xa1    1 2b Hence fxx12 12 e 2 b

Then, the conditional distribution of x2 given x1 is Normal with mean and standard deviation:

1 2 ax12 122   and b 12 1 1   2

Conditional distributions: Bivariate Normal

• Bivariate Normal Distribution with conditional distribution

• Using matrix notation, the conditional moments are given by: 1 1|2  1   12  22 ( x2   2 ) 1  1|2   11   12  22  21

13 RS – 4 – Jointly Distributed RV (a)

Conditional distributions: Bivariate Normal

Major axis of ellipses x2

( 1, 2)

 2  2|1   2   (x1  1 ) 1

Regression to the mean (μ1= μ2= μ) :

2|1    (x1  ); 0    1

   x1     ( 12 ) 2 (x   )    12 (x   ) Note: 2|1 2 1 1 2 2 1 1 12 1 1

Conditional distributions: KF Application

• The Kalman filter (KF) uses the observed data to learn about the unobservable state variables, which describe the state of the model.

• KF models dynamically what we measure, zt, and the state, yt. In the simple, linear model we have:

yt = Ayt-1 + wt (state or transition equation)

zt = Hyt + vt (measurement equation)

wt, vt: error terms, with zero mean and variance Q and R, respectively.

• Based on time t-1 information, the KF generates predictions for yt: yt|t-1 = A yt-1|t-1 + B ut T Pt|t-1 = A Pt-1 A + Q (conditional variance of yt ) • It also generates an update, once the information t is known: y = y + P HT (F )-1 e t|t t|t-1 t|t-1 t|t-1 t|t-1 28 T -1 Pt|t = Pt|t-1 – Pt|t-1 H (Ft|t-1) HPt|t-1

14 RS – 4 – Jointly Distributed RV (a)

Conditional distributions: KF Application

• We define the forecast error for the observed zt and its variance as: et|t-1 = zt –zt|t-1 = zt – Hyt|t-1 T Ft|t-1= H Pt|t-1 H + R

Then, we write the joint of distribution of (yt , et)|It :  y | I  y   P P H'  t t1  ~ N t|t1 , t|t1 t|t1      HP F  et | It1   0   t|t1 t|t1  • Recall a property of the multivariate normal distribution:

1 1|2  1   12  22 ( x 2   2 ) 1  1|2   11   12  22  21 Then, from the joint, we can easily derive the KF update: T -1 yt|t = yt|t-1 + Pt|t-1 H (Ft|t-1) et|t-1 T -1 Pt|t = Pt|t-1 – Pt|t-1 H (Ft|t-1) HPt|t-1 29

15