ECE 650 – Lecture #7 Random Vectors: 2nd Moment Theory, continued
[Ref: R. Scholtz Lecture Notes, USC]
D. van Alphen
D. van Alphen 1 Lecture Overview
• Review of 2nd-Moment Theory for Real R Vectors
• Causal Linear Transformations (for Coloring Noise)
• Cholesky Decomposition
• Expanding the Covariance Matrix: Spectral Resolution
• Mean-squared Length of Random (Column) Vectors
• Directional Preference of Random Vectors – Scholtz Peanut – Directional Preference of White Noise
• Whitening Process for Colored Noise
D. van Alphen 2 2nd-Moment Theory & Transforms: Review
• Recall: default random vectors are real column vectors T • Correlation matrix for R Vector X: RX = E[XX ] T • Covariance Matrix for R Vector X: CX = E[(X-hX)(X-hX) ] T = RX - hXhX • If Y = HX, then: E[Y] = H E[X] T T and RY = H RX H , CY = H CX H T • CY = E L E (E: e-vector columns, L: e-values on diag.)
• To generate colored noise vector Z with mean vector hZ, and covariance CZ, starting with elementary white noise vector W: – Perform linear transform: Z = HW + C 1/2 • where C = hZ, and where H = E L ;
D. van Alphen 3 Linear Transformations & Causality
• Consider the vector Y = HX; i.e.,
Y1 h11 h12 h1n X1 Y h h h X Y 2 21 22 2n 2 Ym hm1 hm2 hmn Xn where the subscripts imply temporal order; i.e.,
input X1 comes before input X2
output Y1 output Y2
• Recall for a causal system, output Yi can only depend on current and previous input values (Xj, j i).
D. van Alphen 4 Linear Transformations & Causality
Y1 = h11 X1 + h12 X2 + … + h1n Xn 0 for a causal system
Y2 = h21 X1 + h22 X2 + h23 X3 … + h1n Xn 0 for a causal system . . .
Yn = hn1 X1 + hn2 X2 + … + hnn Xn Assuming m = n, so H is square No 0’s necessary for a causal system
D. van Alphen 5 Linear Transformations & Causality
h11 0 0 0 0 h h 0 0 21 22 H h31 h32 h33 0 0 hn1 hn2 hn3 hnn Claim: For a causal system, H needs to be lower triangular (all 0’s above the diagonal). Repeating the example from Lecture 6: say we want to simulate a 0-mean vector Y with covariance matrix:
1 .5 .5 Assume that we have access to CY .5 1 .5 elementary white vector W; find causal H such that Y = HW. .5 .5 1 D. van Alphen 6 Linear Transformations & Causality, continued
T • We still need covariance matrix factorization: CY = H H • But now H must be lower triangular.
• One approach: brute force factorization, with algebra:
1 .5 .5 h11 0 0 h11 h21 h31 C .5 1 .5 h h 0 0 h h Y 21 22 22 32 .5 .5 1 h31 h32 h33 0 0 h33
Using the row-1 equations from CY: 2 h11 = 1 h11 = 1 (using the positive solution)
and h11 h21 = -.5 h21 = -.5
and h11 h31 = -.5 h31 = -.5
D. van Alphen 7 Linear Transformations & Causality, continued
• Repeating: 1 .5 .5 h11 0 0 h11 h21 h31 C .5 1 .5 h h 0 0 h h Y 21 22 22 32 .5 .5 1 h31 h32 h33 0 0 h33
Using the row-2 equations from CY:
h11 h21 = -.5 (redundant) 2 2 and h21 + h22 =1 h22 = sqrt(3)/2 (using pos. solution)
and h21 h31 + h22 h32 = - 0.5 h32 = -sqrt(3)/2
Using row-3 equations from CY (to obtain h33): 2 2 2 h31 + h32 + h33 = 1 h33 = 0
D. van Alphen 8 Linear Transformations & Causality, continued
• Repeating: 1 .5 .5 h11 0 0 h11 h21 h31 C .5 1 .5 h h 0 0 h h Y 21 22 22 32 .5 .5 1 h31 h32 h33 0 0 h33 Thus, 1 0 0 lower triangular H 1/ 2 3 / 2 0 causal 1/ 2 3 / 2 0
T Using MATLAB to verify HH = CY: >> H=[1 0 0; -.5 sqrt(3)/2 0; -.5 sqrt(3)/6 sqrt(6)/3];
>> H*H‘ (answer came back as CY)
D. van Alphen 9 Cholesky Decomposition (Another Way to Factor Covariance Matrices)
• This technique (Cholesky Decomposition) only works for positive definite* covariance matrices – Pos. Def. ______
• MATLAB command to factor covariance matrix C using Cholesky Decomposition: chol(C)
• chol(X) uses only the diagonal and upper triangular part of X. – The lower triangular is assumed to be the (complex conjugate) transpose of the upper. • O.K. for covariance matrices, since they are symmetric (for real R. Vectors) * Recall that covariance matrices must be non-negative definite (NND),
but not necessarily positive definite. (In the previous example, CY had a zero-eigenvalue, and was hence not positive definite.) D. van Alphen 10 Cholesky Decomposition, continued
• If C is positive definite, then R = chol(C) produces an upper triangular R so that R‘ * R = C. (R’ will be lower triangular) 1 1/ 2 0 • Example: Let C = 1/ 2 1 1/ 2 0 1/ 2 1 • MATLAB Code: >> C = [1 .5 0; .5 1 .5; 0 .5 1]; >> H*H' >> R = chol(C) ans = R = 1.0000 0.5000 0 1.0000 0.5000 0 0.5000 1.0000 0.5000 0 0.8660 0.5774 0 0.5000 1.0000 0 0 0.8165 >> H = R'; = C, D. van Alphen 11 Spectral Resolution of the Covariance Matrix
• Let C be a covariance matrix for some real R Vector, with
eigenvectors ei and corresponding eigenvalues li. e T 1 T • Then C = (EL)E = [l1 e1 l2 e2 … ln en] e T n n T (the spectral C = li ei ei i1 resolution of C)
Spectrum of C: set of distinct eigenvalues, {li}
T • Each component matrix ei ei is a projection matrix*.
*A projection matrix P has 2 properties: P2 = P, and P = PT (Symmetric). (The product PX “projects” X onto the column space of P).
D. van Alphen 12 Spectral Resolution (or Decomposition) Example 7 2 • Consider covariance matrix C = . 2 4 2 1 • Eigenvalues: l = 8, l = 3; eigenvalues e 5 , e 5 1 2 1 1 2 2 2 5 5 2 1 4 /5 2 /5 l e e 8 5 8 1 1 1 1 5 5 2 /5 1/5 5 1 1 2 1/5 2 /5 l e e 3 5 3 2 2 2 2 5 5 2 /5 4 /5 5 4 /5 2 /5 1/5 2 /5 C l1 e1e1 l2 e2e2 8 3 2 /5 1/5 2 /5 4 /5 D. van Alphen 13 Mean-squared Length of Random Vectors
• Recall for deterministic vectors Y: |Y|2 = YT Y
3 T 3 2 e.g., for Y ; Y Y [3 4] 25 | Y | 4 4 Y1 E[| Y |2] E{YTY} E{ [Y Y Y ] } 1 2 n Yn n n 2 2 2 E[Y1 Y2 Yn ] Rii tr(RY) li i1 i1 2 n Conclusion: Mean-squared length of Y is: E{Y } tr(RY ) li i1
Linear Algebra Note: The trace of a matrix is the sum of its diagonal elements, which also equals the sum of its eigenvalues D. van Alphen 14 Directional Preference of R. Vectors
• Recall if X is a 3-D real vector, and UX, UY, and UZ are ortho- normal basis functions for R3, then
(X UX) UX
is the projection of vector X onto basis function UX, or the projection of X in the direction UX. X
UZ
UX
UY
(X UX) UX (coefficient) D. van Alphen 15 Directional Preference, continued
• Now let Y0 be the centered version of real random vector Y; – let b be any real unit-length vector: |b| = 1; then
T (Y0 b) b = b Y0 b
is the projection of Y0 in the direction of b.
• Thus, the mean-squared length of the projection of Y0 onto b is: T 2 T 2 2 T T E[ | b Y0 b| ] = E[ | b Y0 | |b| ] = E[ | b Y0 Y0 b | ] = bT R b = bT C b Y0 Y Use to find the MS length of a R Vector LA Factoid: aTb = in any direction bTa = dot(a,b) (specified by b) . van Alphen 16 Directional Preference, continued
• Repeating: For arbitrary real (column) random vector Y, the MS length of Y in the direction of unit-length vector b is:
T b CY b • Question: What choice of b gives the largest mean-squared projection?
– Claim: If emax is the eigenvector corresponding to the largest eigenvalue lmax (of CY), and emin is the eigenvector corresponding to the smallest eigenvalue lmin (of CY), then
T 2 T 2 T 2 lmin = E{ |emin Y0| } E{ |b Y0| } E{ |emax Y0| } = lmax
Preferred direction of R. Vector Y, given by the vector with
the largest mean-squared projection, is emax. D. van Alphen 17 3 2 CX 2 4 Directional Preference: Example
• Let Z be a zero-mean vector with covariance matrix: 3 2 CZ 2 4 • Using MATLAB to find e-values, e-vectors: >> CZ = [3 sqrt(2); sqrt(2) 4]; >> [V Lambda] = eig(CZ) V = -0.8165 0.5774 .5774 emax , lmax 5 0.5774 0.8165 .8165 Lambda = .8165 2.0000 0 emin , lmin 2 0 5.0000 .5774
D. van Alphen 18 3 2 CX 2 4 Directional Preference: Example
3 2 • Repeating: Z is zero-mean, with CZ 2 4 • Now generate some unit-length vectors b, at various angles q; then find the RMS length of Z in each of these directions. cosq – Unit-length vector, b, at angle q, can be found: b = . MATLAB function m-file: sinq function B = unit_length_bees(theta) % This function generates unit-length column vectors, b, at % angles specified (in degrees) by input column vector theta. % The column vectors b are stored in matrix B. B = zeros(2, length(theta)); for icount = 1: length(theta) B(:,icount) = [ cosd(theta(icount)); sind(theta(icount))]; end B, scatterplot(B') % scatterplot requires 2-column input D. van Alphen 19 Directional Preference Example, continued
Resulting Scatterplot for Finding the MS length of Z in unit-length b Vectors: the direction of 0 b = [1 0]’:
Scatter plot 3 2 1 1 b’ CZ b = [1 0 2 4 0 0.5 1 [3 2 3 0 0
Quadrature
-0.5 RMS length: sqrt(3)
-1 -1 -0.5 0 0.5 1 In-Phase
D. van Alphen 20 Directional Preference Example, continued - Scholtz Peanut -
• MATLAB Code: RMS-length in Directions, b function RMS = rms_length(theta, cov) Scatter plot % This function finds the RMS length 2 % of 2-dimensional column vectors with % covariance matrix = cov, in various % directions specified (in degrees) in 1 % input row vector theta.
B = unit_length_bees(theta) 0 scaled_B = zeros(2, length(B))
Quadrature RMS = zeros(1,length(theta)) for icount = 1:length(theta) -1 RMS(icount) = sqrt(B(:,icount)'*cov * B(:,icount)); scaled_B(:,icount) = RMS(icount)*B(:,icount); end -2 scaled_B, scatterplot(scaled_B') -2 -1 0 1 2 In-Phase
D. van Alphen 21 White Noise: No Directional Preference
• Claim: White noise has no directional preference.
• Verification: Let W be a white column vector, so the covariance matrix is: 2 CW = s I
– Directional preference of W is given by the eigenvector emax corresponding to the largest eigenvalue of CW.
2 – To find the eigenvalues* of CW = s I:
2 2 det(s I – li I) = 0 li = s (for all i)
lmax = lmin No directional preference * Also note: diagonal matrices always have their e-values on the diagonal. D. van Alphen 22 Scatterplot for White Noise Example (Uniform R. Variables) Scatterplot for R. Vector: • Generating 1000 2- dimensional R Vectors: X [X1 X2 • MATLAB Code: Scatter plot 1
>> X1 = unifrnd(-1,1,1000,1); 0.8 >> X2 = unifrnd(-1,1,1000,1) 0.6 >> X = [X1 X2]; 0.4 0.2
>> scatterplot(X) 0
Quadrature -0.2
-0.4 Random variables X1 and X2 are -0.6 each uniform on (-1,1); -0.8 -1 For scatterplot, X must be a -1 -0.5 0 0.5 1 In-Phase row vector; here we generate 1000 realizations of X. No directional preference D. van Alphen 23 Scatterplot for White Noise Example (Gaussian R. Variables) Scatterplot for R. Vector: • Generating 1000 2- dimensional R Vectors: X [X1 X2 Scatter plot 4
• MATLAB Code: 3
• X1 = normrnd(0,1,1000,1); 2
• X2 = normrnd(0,1,1000,1); 1 • X = [X1 X2] 0 • scatterplot(X) Quadrature -1 -2
Random variables X1 and X2 are -3 each N(0,1); -4 -4 -2 0 2 4 For scatterplot, X must be a In-Phase row vector; here we generate No directional preference 1000 realizations of X. D. van Alphen 24 Whitening Colored Noise (Going the Other Way)
• Question: How would we “whiten” a zero-mean noise vector N T 1/2 with covariance matrix CN = H H (where H = E L ) ?
Problem: Find the N G W = G N required Cov. CN, Elt. White: Cov. CN, transformation, G. hN = 0 hN = 0
T T T T T T • We know: CW = G CN G = G(HH )G = (GH)(H G ) = (GH)(GH) T • We want G such that: CW = (GH)(GH) = I Choose G = H-1 = (EL1/2)-1 = L-1/2 ET
(Whitening Filter)
D. van Alphen 25 Property Summary for Symmetric NND Matrices (e.g., Covariance Matrices for Real R. Vectors) [Mix] Let A be Symmetric, NND; then
1. The eigenvalues of A are non-negative real numbers, which
can be ordered: l1 l2 … ln
2. The eigenvectors ei of A can be chosen to be orthonormal, so:
T T ei ei = 1, ei ej = 0 (i j) 3. (Spectral Resolution) The original matrix A can be written as a weighted sum (with eigenvalue weights):
T T T A = l1 e1 e1 + l2 e2 e2 + … + ln en en T T T l1 e1 e1 + l2 e2 e2 + … + lk ek ek, k< n (used for data compression) D. van Alphen 26