RS – 4 – Jointly Distributed RV (a) Chapter 4 Jointly distributed Random variables Continuous Multivariate distributions Continuous Random Variables 1 RS – 4 – Jointly Distributed RV (a) Joint Probability Density Function (pdf) Definition: Joint Probability density function Two random variable are said to have joint probability density function f(x,y) if 1. 0 f xy , 2. f x, y dxdy 1 3. P X, Y A f x , y dxdy A Marginal and Condition Density Definition: Marginal Density Let X and Y denote two RVs with joint pdf f(x,y), then the marginal density of X is f xfxydy , X f yfxydx , and the marginal density of Y is Y Definition: Conditional Density Let X and Y denote two RVs with joint pdf f(x,y) and marginal densities fX(x), fY(y), then the conditional density of Y given X = x and the conditional density of X given Y = y are given by f xy, f xy, fyxYX fxyXY f X x f Y y 2 RS – 4 – Jointly Distributed RV (a) Joint MGF Definition: MGF of (X,Y) Let X and Y be two RVs with joint pdf f(x,y) then the MGF of X Y: m XY (t1,t2 ) E[exp(t1 X t2Y )] exp(t1 X t2Y ) f (x, y)dxdy R 2 Theorem: The MGF of a pair of independent RVs is the product of the MGF of the corresponding marginal distributions. That is, mXY(t1,t2) = mX(t1) mY(t2). Proof: m XY (t1 , t 2 ) exp( t1 X t 2Y ) f (x, y)dxdy exp( t1 X ) exp( t 2Y ) f (x) f ( y)dxdy exp( t1 X ) f (x)dx exp( t 2Y ) f ( y)dy m X (t1 )mY (t 2 ) Marginal MGF Definition: MGF of the marginal distribution of X (andY) Let mXY(t1,t2) be the MGF of (X,Y), then the MGF of the marginal distributions of X and Y are, respectively, mXY(t1,0) and mXY(0,t2). Proof: m X (t) exp(tX ) f X (x)dx exp(tX )[ f XY (x, y)dy]dx exp(tX ) f XY (x, y)dydx m XY (t,0) Similar derivation for Y. 3 RS – 4 – Jointly Distributed RV (a) The bivariate Normal distribution Sir Francis Galton (1822 –1911, England) The bivariate normal distribution Let the joint distribution be given by: 1 1 Qx12, x fxx, e2 12 2 2112 where 2 2 xxxx 112 11 2 2 2 2 1122 Qxx, 12 1 2 This distribution is called the bivariate Normal distribution. The parameters are 1, 2 , 1, 2 and The properties of this distribution were studied by Francis Galton and discovered its relation to the regression, term Galton coined. 4 RS – 4 – Jointly Distributed RV (a) The bivariate normal distribution Surface Plots of the bivariate Normal distribution The bivariate normal distribution Note: We can have a more compact joint using linear algebra: 2 1 1 x x x x f (x , x ) exp ( 1 1 )2 2( 1 1 )( 2 2 ) ( 2 2 )2 1 2 2 2(1 2 ) 21 2 (1 ) 1 1 2 2 1 1 exp (x μ)'1(x μ) 2(| |)1/2 2 (1) Determine the inverse and determinant of Σ (the covariance matrix) 2 2 1 21 2 2 2 2 2 12 2 2 2 2 | |1 2 12 1 2 (1 2 2 ) 1 2 (1 ) 12 2 1 2 2 1 1 2 21 2 | | 12 1 5 RS – 4 – Jointly Distributed RV (a) The bivariate normal distribution (2) Write a quadratic form for Q(x1,x2): 2 2 xxxx 112 11 2 2 2 2 1122 Qxx, 12 1 2 1 Q(x1, x2 ) (x μ)' (x μ) 2 1 2 21 x1 1 (x1 1) (x2 2 ) 2 | | 12 1 x2 2 1 2 2 x1 1 (x1 1) 2 (x2 2 )12 (x1 1) 21 (x2 2 )1 | | x2 2 1 ((x )2 2 2 (x )(x ) (x )2 2 ) | | 1 1 2 21 1 1 2 2 2 2 1 2 2 2 2 ((x1 1) 2 2 21(x1 1)(x2 2 ) (x2 2 ) 1 ) 2 2 2 1 2 (1 ) The Bivariate Normal Distribution f(x,y) 12 12 y 12 y y x x x Contour Plots of the Bivariate Normal Distribution yyy 12 12 12 2 2 2 x xx 1 1 1 Scatter Plots of data from the Bivariate Normal Distribution yyy 12 12 12 2 2 2 x xx 1 1 1 6 RS – 4 – Jointly Distributed RV (a) The bivariate normal distribution: MGF Let MGF of a bivariate normal is given by: 1 m (t ,t ) exp[t t (t 2 2 t 2 2 2 t t )] XY 1 2 1 X 2 Y 2 1 X 2 Y XY 1 2 X Y Note: When ρXY = 0 –i.e., X and Y are independent. The MGF is: 1 m (t ,t ) exp[t t (t 2 2 t 2 2 )] XY 1 2 1 X 2 Y 2 1 X 2 Y Marginal distributions for the Bivariate Normal Recall the definition of marginal distributions for continuous RV: f xfxxdx , and f xfxxdx , 11 12 2 22 12 1 In the case of the bivariate normal distribution the marginal distribution of xi is Normal with mean i and standard deviation i. Proof: The marginal distributions of x2 is 1 1 Qx12, x f xfxxdx , e2 dx 22 12 1 2 1 2112 where 22 xxxx 112 11 2 2 2 2 1122 Qxx, 12 1 2 7 RS – 4 – Jointly Distributed RV (a) Marginal distributions for the Bivariate Normal Now: 22 xxxx 112 11 2 2 2 2 1122 Qxx, 12 1 2 2 22 xa11 x aa cxc222 1 2 bbbb xx2 11222 x 1122 1 21 21 2 2 xx 1 222 22 + 22 21 22 1212111 Marginal distributions for the Bivariate Normal 22 2 Hence bb 1 or 1 1 a x 122 Also b2 22 1121 1 1 12x 2 1 2 and 1 ax12 2 8 RS – 4 – Jointly Distributed RV (a) Marginal distributions for the Bivariate Normal Finally 2 a 2 2 xx c 1 222 22 b 2 22 21 22 1212111 2 2 xxa 2 c 1 2 22 22 22 21 22b 2 1212111 2 2 xx 1 222 22 22 21 22 1212111 2 1 x 12 2 2 1 Marginal distributions for the Bivariate Normal and 2 1 2 112 cxx12 122 2 22 22 1 1 22 2 1 122x 2 2 1 1 2 2 2 1 x22 22 1 1 2 2 x 22 2 9 RS – 4 – Jointly Distributed RV (a) Marginal distributions for the Bivariate Normal Summarizing 22 xxxx 112 11 2 2 2 2 1122 Qxx, 12 1 2 2 xa1 c b 2 where b 1 1 1 ax12 2 2 x and c 22 2 Marginal distributions for the Bivariate Normal f xfxxdx , Thus 22 12 1 1 1 Qx12, x e2 dx 2 1 2112 2 1 xa1 c 1 2 b edx 2 1 2112 2 c 2 1 xa1 21 be e2 b dx 2 2 b 1 2112 2 1 x 1 22 e 2 2 22 10 RS – 4 – Jointly Distributed RV (a) Marginal distributions for the Bivariate Normal • Thus the marginal distribution of x2 is Normal with mean 2 and standard deviation 2. • Similarly, the marginal distribution of x1 is Normal with mean 1 and standard deviation 1. Note: This derivation is much easier using MGFs. Use the MGF of a bivariate normal. To get the MGF of the marginal of X, set t2=0. 1 m (t ,t ) exp[t t (t 2 2 t 2 2 2 t t )] XY 1 2 1 X 2 Y 2 1 X 2 Y XY 1 2 X Y 1 m (t ,0) exp[t (t 2 2 )] m (t ) XY 1 1 X 2 1 X X 1 Marginal distributions: Bivariate Normal Bivariate Normal Distribution with marginal distributions 11 RS – 4 – Jointly Distributed RV (a) Conditional distributions for the Bivariate Normal Recall the definition of conditional distributions for continuous RVs: f xx12, f xx12, fxx12 12 and fxx21 21 f22x f11x In the case of the bivariate normal distribution the conditional distribution of xi given xj is Normal with mean and standard deviation: i 2 ij ijjx and ij i 1 j Conditional distributions: Bivariate Normal 1 Proof: | Qx12, x e 2 2 f (x1, x2 ) 2112 f 2 1|2 1 x 22 f (x2 ) 1 e 2 2 22 2 2 2 11x 11xa122 x Qx, x | 22 c | 12 22b ee22 2 2 22 2111 21 where 2 x 2 a 1 (x ) c 22 b 1 1 1 2 2 2 2 12 RS – 4 – Jointly Distributed RV (a) Conditional distributions: Bivariate Normal 2 1xa1 1 2b Hence fxx12 12 e 2 b Then, the conditional distribution of x2 given x1 is Normal with mean and standard deviation: 1 2 ax12 122 and b 12 1 1 2 Conditional distributions: Bivariate Normal • Bivariate Normal Distribution with conditional distribution • Using matrix notation, the conditional moments are given by: 1 1|2 1 12 22 ( x2 2 ) 1 1|2 11 12 22 21 13 RS – 4 – Jointly Distributed RV (a) Conditional distributions: Bivariate Normal Major axis of ellipses x2 ( 1, 2) 2 2|1 2 (x1 1 ) 1 Regression to the mean (μ1= μ2= μ) : 2|1 (x1 ); 0 1 x1 ( 12 ) 2 (x ) 12 (x ) Note: 2|1 2 1 1 2 2 1 1 12 1 1 Conditional distributions: KF Application • The Kalman filter (KF) uses the observed data to learn about the unobservable state variables, which describe the state of the model.