CovarianceCovariance ofof twotwo randomrandom variablesvariables

• Height and wake-up time are uncorrelated, but height and weight are correlated. • Covariance Cov(X, Y) = 0 for X = height, Y = wake-up times Cov(X, Y) > 0 for X = height, Y = weight • Definition:

Cov(X ,Y ) % CXY % E!(X $ #x )(Y $ # y )"

Question: If Cov(X, Y) < 0 for two random variables X, Y , what would a scatterplot of samples from X, Y look like?

Question: If we add arbitrary constants to the random variables X, Y , how does the covariance change?

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 13 EstimatingEstimating covariancecovariance fromfrom samplessamples

Again, we assume that we do not know the +x1 x2 ! xn ( underlying probability distributions. But consider )y y y & we sample n times and estimate: * 1 2 ! n '

1 n Cov(X ,Y ) - ,(xi $ mx )(yi $ my ) “sample covariance” n i-1

Questions: What is Cov(X, X) ? Cov(X, X) = Var(X) How are Cov(X, Y) and Cov(Y, X) related? Cov(X, Y) = Cov(Y, X)

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 14 EstimatingEstimating covariancecovariance inin MatlabMatlab

• Sample mean: x - [x1 x2 x3 ! xn ] mx . m_x my . m_y y - [y1 y2 y3 ! yn ]

• Covariance + y1 $ my ( )y $ m & 1 ) 2 y & Cov(X ,Y ) - [x1 $ mx x2 $ mx ! xn $ mx ] n ) " & ) & *)yn $ my '& Method 1: >> v = (1/n)*(x-m_x)*(y-m_y)’ Method 2: >> w = x-m_x >> z = y-m_y >> v = (1/n)*w*z’

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 15 CoefficientCoefficient ofof CorrelationCorrelation

The coefficient of correlation is defined as:

E!(X $ #x )(Y $ # y )" Cov(X ,Y ) 0 xy % - / x/ y / x/ y

Properties: • -1 ! Cxy ! 1 • if Cxy = 0 : X and Y uncorrelated • if Cxy bigger/smaller zero : X and Y are positively/negatively correlated • Advantage: we can multiply X and Y with arbitrary factors and Cxy stays the same.

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 16 CorrelationCorrelation ofof twotwo randomrandom variablesvariables

Definition: Corr(X ,Y ) - E!XY "

If X and Y have zero mean, this is the same as the covariance. If in addition, X and Y have variance of one this is the same as the coefficient of correlation.

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 17 Correlation,Correlation, Covariance,Covariance, Corr.CoeffCorr.Coeff..

Corr(X ,Y ) - E!"XY Correlation

Covariance Cov(X ,Y ) - E!(X $ # x )(Y $ # y )"

E!(X $ # )(Y $ # )" Cov(X ,Y ) x y Coefficient of Corr. Cxy - - / x/ y / x/ y

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 18 CovarianceCovariance MatrixMatrix

Covariance: Cov(A, B) - E!(A $ # A )(B $ #B )"

T Now consider random vector: X - (x1, x2 ,!, xn ) We can compute covariance between two components, say Cov(x , x ) - c - E!(x $ # )(x $ # )" 2 5 25 2 x2 5 x5 between x2 and x5:

Doing this for all combinations 6c c c 3 gives us the elements of the 4 11 21 ! n1 1 c c covariance matrix: 4 12 22 1 7 % 4 1 X 4 " # " 1 4 1 4 1 5c1n ! cnn 2

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 19 PropertiesProperties ofof CovarianceCovariance MatrixMatrix

6 c c c 3 4 11 21 ! n1 1 4c12 c22 1 4 1 7 X % Cov(x , x ) - c - E!(x $ # )(x $ # )" 4 " # " 1 i j ij i xi j x j 4 1 4 1 5c1n ! cnn 2

It is symmetric, because c ij - c ji . 2 Its diagonal elements are the individual variances: c ii - / i - var(xi ) . If it is diagonal, the x are all i 6/ 2 0 0 3 4 1 ! 1 uncorrelated and we have: 2 4 0 / 2 1 7 % 4 1 X 4 " # " 1 4 1 4 2 1 5 0 ! / n 2

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 20 ExampleExample covariancecovariance matrixmatrix

people’s heights: time people woke up this X ~ N(67, 20) morning: Y ~ N(9, 1)

Question: what is the covariance matrix of V = (X Y)T ? X and Y should be uncorrelated: +X ( +20 0( V - ) & ) & *Y ' * 0 1'

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 21 EstimatingEstimating thethe covariancecovariance matrixmatrix fromfrom samplessamples (including(including MatlabMatlab code)code)

Sample n times and find mean of samples:

v1 v2 ! vn +x x x ( +mx ( V - 1 2 ! n m - ) & )m & *y1 y2 ! yn ' * y '

Find the covariance matrix:

+x1 $ mx y1 $ my ( ) & 1 +x1 $ mx x2 $ mx ! xn $ mx ( x2 $ mx y2 $ my Cov(V ) - ) & ) & ) & n *y1 $ my y2 $ my ! yn $ my ' " " ) & y $ m *)xn $ mx n y '& >> m = (1/n)*sum(v,2) >> z = v - repmat(m,1,n) >> v = (1/n)*z*z’

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 22 GaussianGaussian distributiondistribution inin DD dimensionsdimensions

Of course, the most important distribution can also be extended to higher dimensions. Recall that a 1-dimensional Gaussian is completely determined by its mean, #, and its variance, /82: (x$# )2 1 $ X ~ N!#: /82) p(x) - 2/ 2 2" / e

A D-dimensional Gaussian (multivariate Gaussian) is completely determined by its mean, #, and its covariance matrix, 79

1 $ 1 (x$µ)T 7$1 (x$µ) p(x) - 2 X ~ N!#:7) D / 2 1/ 2 e (2") 7

Question: what happens when D = 1 for the D-dimensional Gaussian?

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 23 TheThe GaussianGaussian inin DD dimensionsdimensions

Question: What does a set of equiprobable points look like for a 2-dim. Gaussian? What for a D-dim. Gaussian?

In 2D, it’s an ellipse. In D dimensions, it’s an ellipsoid.

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 24 EquiprobableEquiprobable contourscontours ofof aa GaussianGaussian

If a Gaussian random vector has covariance matrix that is diagonal (all of the variables are uncorrelated, then the axes of the ellipsoid are parallel to the coordinate axes.

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 25 EquiprobableEquiprobable contourscontours ofof aa GaussianGaussian

If a Gaussian random vector has covariance matrix that is not diagonal (some of the variables are correlated), then the axes of the ellipsoid are perpendicular to each other, but are not parallel to the coordinate axes.

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 26