EE401 (Semester 1) Jitkomut Songsiri 5. Random Vectors

probabilities • characteristic function • cross correlation, cross covariance • Gaussian random vectors • functions of random vectors •

5-1 Random vectors we denote X a random vector

X is a function that maps each outcome ζ to a vector of real numbers an n-dimensional random variable has n components:

X1 X X = 2  .  X   n   also called a multivariate or multiple random variable

Random Vectors 5-2 Probabilities

Joint CDF

F (x) , FX(x , x ,...,x )= P (X x ,X x ,...,X x ) 1 2 n 1 ≤ 1 2 ≤ 2 n ≤ n

Joint PMF

p(x) , pX(x1, x2,...,xn)= P (X1 = x1,X2 = x2,...,Xn = xn)

Joint PDF

∂n f(x) , fX(x1, x2,...,xn)= F (x) ∂x1 ...∂xn

Random Vectors 5-3 Marginal PMF

pXj (xj)= P (Xj = xj)= ...... pX(x1, x2,...,xn) x1 xj 1 xj+1 xn −

Marginal PDF

∞ ∞ fX (xj)= ... fX(x1, x2,...,xn) dx1 ...dxj 1dxj+1 ...dxn j − −∞ −∞

Conditional PDF: the PDF of Xn given X1,...,Xn 1 is −

fX(x1,...,xn) f(xn x1,...,xn 1)= − | fX1,...,Xn 1(x1,...,xn 1) − −

Random Vectors 5-4 Characteristic Function

the characteristic function of an n-dimensional RV is defined by

j(ω1X1+ +ωnXn) Φ(ω)=Φ(ω1,...,ωn) = E[e ··· ]

T = ejω xf(x)dx x where ω1 x1 ω x ω = 2 , x = 2  .   .  ω  x   n  n    

Φ(ω) is the n-dimensional Fourier transform of f(x)

Random Vectors 5-5 Independence

the random variables X1,...,Xn are independent if

the joint pdf (or pmf) is equal to the product of their marginal’s

Discrete pX(x ,...,x )= p (x ) p (x ) 1 n X1 1 Xn n Continuous fX(x ,...,x )= f (x ) f (x ) 1 n X1 1 Xn n we can specify an RV by the characteristic function in place of the pdf,

X1,...,Xn are independent if

Φ(ω) = Φ (ω ) Φ (ω ) 1 1 n n

Random Vectors 5-6 Example: White noise signal in communication

the n samples X1,...Xn of a noise signal have the joint pdf:

(x2+ +x2 )/2 e− 1 ··· n fX(x ,...,x )= for all x ,...,x 1 n (2π)n/2 1 n

the joint pdf is the n-product of one-dimensional Gaussian pdf’s

thus, X1,...,Xn are independent Gaussian random variables

Random Vectors 5-7 Expected Values the expected value of a function

g(X)= g(X1,...,Xn) of a vector random variable X is defined by

E[g(X)] = g(x)f(x)dx Continuous x E[g(X)] = g(x)p(x) Discrete x Mean vector

X1 E[X1] X E[X ] µ = E[X]= E 2 , 2  .   .  X  E[X ]  n  n     

Random Vectors 5-8 Correlation and Covariance matrices

Correlation matrix has the second moments of X as its entries:

E[X1X1] E[X1X2] E[X1Xn] E[X X ] E[X X ] E[X X ] R , E[XXT ] = 2 1 2 2 2 n  . . ... .  E[X X ] E[X X ] E[X X ]  n 1 n 2 n n    with Rij = E[XiXj]

Covariance matrix has the second-order central moments as its entries:

C , E[(X µ)(X µ)T ] − − with C = cov(X ,X ) = E[(X µ )(X µ )] ij i j i − i j − j

Random Vectors 5-9 Symmetric matrix

n n T A R × is called symmetric if A = A ∈ Facts: if A is symmetric

all eigenvalues of A are real • all eigenvectors of A are orthogonal • A admits a decomposition • A = UDU T

where U T U = UU T = I (U is unitary) and D is diagonal

(of course, the diagonals of D are eigenvalues of A)

Random Vectors 5-10 Unitary matrix

n n a matrix U R × is called unitary if ∈ U T U = UU T = I

1 1 cos θ sin θ example: 1 − , − √2 1 1 sin θ cos θ Facts:

a real unitary matrix is also called orthogonal • 1 T a unitary matrix is always invertible and U − = U • columns vectors of U are mutually orthogonal • norm is preserved under a unitary transformation: • y = Ux = y = x ⇒

Random Vectors 5-11 Positive definite matrix a symmetric matrix A is positive semidefinite, written as A 0 if xT Ax 0, x Rn ≥ ∀ ∈ and positive definite, written as A 0 if ≻ xT Ax > 0, for all nonzero x Rn ∈

Facts: A 0 if and only if

all eigenvalues of A are non-negative • all principle minors of A are non-negative •

Random Vectors 5-12 1 1 example: A = − 0 because 1 2 − 1 1 x xT Ax = x x 1 1 2 1− 2 x − 2 = x2 + 2x2 2x x 1 2 − 1 2 =(x x )2 + x2 0 1 − 2 2 ≥ or we can check from

eigenvalues of A are 0.38 and 2.61 (real and positive) • 1 1 the principle minors are 1 and − = 1 (all positive) • 1 2 − note: A 0 does not mean all entries of A are positive!

Random Vectors 5-13 Properties of correlation and covariance matrices let X be a (real) n-dimensional random vector with mean µ

Facts:

R and C are n n symmetric matrices • × R and C are positive semidefinite • If X ,...,X are independent, then C is diagonal • 1 n the diagonals of C are given by the variances of X • k if X has zero mean, then R = C • C = R µµT • −

Random Vectors 5-14 Cross Correlation and Cross Covariance

let X, Y be vector random variables with means µX,µY respectively

Cross Correlation

cor(X, Y)= E[XYT ] if cor(X, Y) = 0 then X and Y are said to be orthogonal

Cross Covariance

cov(X, Y) = E[(X µ )(Y µ )T ] − X − Y = cor(X, Y) µ µT − X Y if cov(X, Y) = 0 then X and Y are said to be uncorrelated

Random Vectors 5-15 Affine transformation let Y be an affine transformation of X:

Y = AX + b where A and b are deterministic matrices

µ = Aµ + b • Y X

µY = E[AX + b]= AE[X]+ E[b]= AµX + b

C = AC AT • Y X C = E[(Y µ )(Y µ )T ]= E[(A(X µ ))(A(X µ ))T ] Y − Y − Y − X − X = AE[(X µ )(X µ )T ]AT = AC AT − X − X X

Random Vectors 5-16 Diagonalization of covariance matrix suppose a random vector Y is obtained via a linear transformation of X

XY A

the covariance matrices of X, Y are C , C respectively • X Y A may represent linear filter, system gain, etc. • the covariance of Y is C = AC AT • Y X

Problem: choose A such that CY becomes ’diagonal’ in other words, the variables Y1,...,Yn are required to be uncorrelated

Random Vectors 5-17 since CX is symmetric, it has the decomposition:

T CX = UDU where

D is diagonal and its entries are eigenvalues of C • X U is unitary and the columns of U are eigenvectors of C • X diagonalization: pick A = U T to obtain

T T T T T CY = ACXA = AUDU A = U UDU U = D as desired

Random Vectors 5-18 one can write X in terms of Y as

Y1 n T Y2 X = UU X = UY = U1 U2 Un   = YkUk . k=1 Yn     this equation is called Karhunen-Lo´eve expansion

X can be expressed as a weighted sum of the eigenvectors U • k the weighting coefficients are uncorrelated random variables Y • k

Random Vectors 5-19 4 2 example: X has the covariance matrix 2 4 design a transformation Y = AX s.t. the covariance of Y is diagonal the eigenvalues of CX and the corresponding eigenvectors are

1 1 λ = 6, u = , λ = 2, u = 1 1 1 2 2 1 − u and u are orthogonal, so if we normalize u so that u = 1 then 1 2 k k u1 u2 U = √2 √2 is unitary

T therefore, CX = UDU where

1/√2 1/√2 6 0 U = ,D = 1/√2 1/√2 0 2 −

T thus, if we choose A = U then CY = D which is diagonal as desired

Random Vectors 5-20 Whitening transformation we wish to find a transformation Y = AX such that

CY = I

a white noise property: the covariance is the identity matrix • all components in Y are all uncorrelated • the variances of Y are normalized to 1 • k T from CY = ACXA and use the eigenvalue decomposition in CX

T T 1/2 1/2 T T CY = AUDU A = AUD D U A d