ST 370 Probability and Statistics for Engineers Covariance and Correlation

The probability distribution of a random variable gives complete information about its behavior, but its mean and variance are useful summaries.

Similarly, the joint probability distribution of two random variables gives complete information about their joint behavior, but their means and variances do not summarize how they behave together.

We also need to know their covariance:

cov(X , Y ) = σXY = E [(X − µX )(Y − µY )] .

1 / 15 Joint Probability Distributions Covariance and Correlation ST 370 Probability and Statistics for Engineers

Example: Mobile response time x = Number of bars 1 2 3 Marginal y = Response time 4+ 0.15 0.10 0.05 0.30 3 0.02 0.10 0.05 0.17 2 0.02 0.03 0.20 0.25 1 0.01 0.02 0.25 0.28 Marginal 0.20 0.25 0.55

From the marginal distributions:

µX = 1 × 0.20 + 2 × 0.25 + 3 × 0.55 = 2.35,

µY = 1 × 0.28 + 2 × 0.25 + 3 × 0.17 + 4 × 0.30 = 2.49.

2 / 15 Joint Probability Distributions Covariance and Correlation ST 370 Probability and Statistics for Engineers

Also from the marginal distributions,

2 σX = 0.6275, 2 σY = 1.4099.

For the covariance, we need the joint distribution:

3 4 X X σXY = [(x − µX )(y − µY )] fXY (x, y) x=1 y=1 = −0.5815.

3 / 15 Joint Probability Distributions Covariance and Correlation ST 370 Probability and Statistics for Engineers

Sign of covariance Negative covariance, as here, means that X and Y tend to move in opposite directions: a stronger signal leads to shorter response times, and conversely.

Positive covariance would mean that they tend to move in the same direction; zero covariance would mean that X and Y are not linearly related.

4 / 15 Joint Probability Distributions Covariance and Correlation ST 370 Probability and Statistics for Engineers

Magnitude of covariance The magnitude of the covariance is harder to interpret; in particular, it has the units of X multiplied by the units of Y , here seconds2.

It is easier to interpret a dimensionless quantity, the correlation coefficient cov(X , Y ) σXY ρXY = p = . V (X )V (Y ) σX σY

The correlation coefficient has the same sign as the covariance, and always lies between −1 and +1; in the example, ρXY = −0.618228.

5 / 15 Joint Probability Distributions Covariance and Correlation ST 370 Probability and Statistics for Engineers

Independence If X and Y are independent, then

fXY (x, y) = fX (x) × fY (y), and X X E(XY ) = xyfXY (x, y) x y X X = xyfX (x)fY (y) x y X X = xfX (x) yfY (y) x y = E(X )E(Y ).

6 / 15 Joint Probability Distributions Covariance and Correlation ST 370 Probability and Statistics for Engineers

More generally,

E[(X − a)(Y − b)] = E(X − a)E(Y − b) and with a = µX and b = µY ,

cov(X , Y ) = E(X − µX )E(Y − µY ) = 0, and consequently also ρXY = 0.

That is, if X and Y are independent, they are also uncorrelated.

The opposite is not generally true: if X and Y are uncorrelated, they might or might not be independent.

7 / 15 Joint Probability Distributions Covariance and Correlation ST 370 Probability and Statistics for Engineers

Estimating covariance and correlation

The covariance σXY and correlation ρXY are characteristics of the joint probability distribution of X and Y , like µX , σX , and so on.

That is, they characterize the population of values of X and Y .

8 / 15 Joint Probability Distributions Covariance and Correlation ST 370 Probability and Statistics for Engineers

From a sample of values, we estimate µX and σX byx ¯ and sx , the sample mean and standard deviation.

By analogy with the sample variance

n 1 X s2 = (x − x¯)2, x n − 1 i i=1 the sample covariance is given by

n 1 X s = (x − x¯)(y − y¯). xy n − 1 i i i=1

9 / 15 Joint Probability Distributions Covariance and Correlation ST 370 Probability and Statistics for Engineers

The sample correlation coefficient is

sxy rxy = sx sy Pn (x − x¯)(y − y¯) = i=1 i i . pPn 2pPn 2 i=1(xi − x¯) i=1(yi − y¯)

Notice the similarity to the calculation of the regression coefficient

Pn (x − x¯)(y − y¯) s s βˆ = i=1 i i = xy = r × y . 1 Pn 2 2 xy i=1(xi − x¯) sx sx

10 / 15 Joint Probability Distributions Covariance and Correlation ST 370 Probability and Statistics for Engineers

But note the difference in context: In the regression context, we have a model

Y = β0 + β1x + ,

in which x is a fixed quantity, and Y is a random variable; In the correlation context, both X and Y are random variables.

The connection between correlation and regression is deeper than just the computational similarity, but they are not the same thing.

11 / 15 Joint Probability Distributions Covariance and Correlation ST 370 Probability and Statistics for Engineers Linear Functions of Random Variables

Given random variables X1, X2,..., Xp and constants c1, c2,..., cp the random variable Y given by

Y = c1X1 + c2X2 + ··· + cpXp is a linear combination of X1, X2,..., Xp.

The expected value of Y is

E(Y ) = c1E(X1) + c2E(X2) + ··· + cpE(Xp)

12 / 15 Joint Probability Distributions Linear Functions of Random Variables ST 370 Probability and Statistics for Engineers

The variance of Y involves both the variances and covariances of the X s.

If the X s are uncorrelated, and in particular if they are independent, then 2 2 2 V (Y ) = c1 V (X1) + c2 V (X2) + ··· + cp V (Xp).

13 / 15 Joint Probability Distributions Linear Functions of Random Variables ST 370 Probability and Statistics for Engineers

Special case: the average 1 ¯ If c1 = c2 = ··· = cp = p , then Y is just X , the average of X1, X2,..., Xp

If the X s all have the same expected value µ, then

E X¯
= µ and if they are uncorrelated and all have the same variance σ2, then

σ2 V X¯
= . p

14 / 15 Joint Probability Distributions Linear Functions of Random Variables ST 370 Probability and Statistics for Engineers

Note that σ σ ¯ = √ , X p which becomes small when p is large.

That means that when p is large, X¯ is likely to be close to µ, a result known as the weak law of large numbers.

15 / 15 Joint Probability Distributions Linear Functions of Random Variables