Confidence Intervals, Testing and ANOVA Summary 1 One–Sample

Conﬁdence Intervals, Testing and ANOVA Summary

1 One–Sample Tests

1.1 One Sample z–test: Mean (σ known)

Let X1, ··· ,Xn a r.s. from N(µ, σ) or n > 30. Let

H0 : µ = µ0.

The test statistic is x¯ − µ z = √ 0 ∼ N(0, 1) σ/ n for H . 0 σ A (1 − α)100% conﬁdence interval for µ isx ¯ ± z? √ . Sample size for n margin of error, m, is z?σ 2 n = . m

1 1.2 One Sample t–test: Mean (σ unknown)

Let X1, ··· ,Xn a random sample and let either the population is normal or

15 ≤ n < 40 and there are no outliers or strong skewness or

n ≥ 40.

Let H0 : µ = µ0. The test statistic is x¯ − µ t = √ 0 ∼ t(n − 1) s/ n for H . 0 s A (1 − α)100% conﬁdence interval for µ isx ¯ ± t?(n − 1)√ . n

1.3 Matched Pairs

Let (X1,Y1) ··· (Xn,Yn) be a r.s. and deﬁne Dj = Xj −Yj. Assume n > 30, or the Dj’s are normal (or pretty much so). Let

H0 : µD = d.

The test statistic is d¯− d t = √ ∼ t(n − 1) sD/ n

for H0. A (1 − α)100% CI for µD is s d¯± t?(n − 1)√D . n

2 2 Two Sample Tests

2.1 Two Sample z–test: Mean (σX and σY both known)

Let X1, ··· ,XnX and Y1, ··· ,YnY be independent r.s.’s. Assume nX > 30 and nY > 30, or that both r.s.’s are normal. Let

H0 : µX = µY

The test statistic is x¯ − y¯ z = ∼ N(0, 1) q σ2 σ2 X + Y nX nY

for H0. A (1 − α)100% conﬁdence interval for µX − µY is s σ2 σ2 x¯ − y¯ ± z? X + Y . nX nY

3 2.2 Two Sample t–test: Mean (σX and σY both unknown)

Let X1, ··· ,XnX and Y1, ··· ,YnY be independent r.s.’s. Assume nX > 30 and nY > 30, or that both r.s.’s are normal. Let

H0 : µX = µY

The test statistic is x¯ − y¯ t = ∼ t(df) q s2 s2 X + Y nX nY 2 s2 s2 X + Y nX nY for H0. Welch’s t Test lets df = 2 2 . The con- s2 s2 1 X + 1 Y nX −1 nX nY −1 nY servative Welch’s t Test is to let df be the largest integer that is less than or equal to the df of Welch’s Test. An even more conservative test is to let the df = min(nX − 1, nY − 1). A (1 − α)100% conﬁdence interval for µX − µY is s s2 s2 x¯ − y¯ ± t?(df) X + Y . nX nY

4 2.3 Two Sample t–test: Mean (σY = σY unknown)

Let X1, ··· ,XnX and Y1, ··· ,YnY be independent r.s.’s. Assume nX > 30 and nY > 30, or both r.s.’s are normal. Let

H0 : µX = µY

2 2 Deﬁne the pooled estimate of σX = σY to be

2 2 2 (nX − 1)sX + (nY − 1)sY sp = . nX + nY − 2 The test statistic is x¯ − y¯ t = ∼ t(nX + nY − 2) q 1 1 sp + nX nY

for H0. A (1 − α)100% CI for µX − µY is r ? 1 1 (¯x − y¯) ± t (nX + nY − 2)sp + . nX nY

Note: It is generally diﬃcult to verify that the two variances are equal, so it is safer not to make this assumption unless one is conﬁdent the variances are equal.

5 2.4 Two Sample f–test: Standard Deviation

Let X1, ··· ,XnX and Y1, ··· ,YnY be independent normal r.s.’s, where the ﬁrst r.s. is the one with the larger sample variance. Let

H0 : σX = σY

The test statistic is

2 sX f = 2 ∼ F (nX − 1, nY − 1) sY

? for H0. Use the right hand tail for critical values, f , for a two–sided test.

Warning: the above f–test is not robust with respect to the normality assumption.

6 3 Proportion Tests

3.1 One Sample Large Sample Population Proportion z–test

Let X1, ··· ,Xn be a r.s. from Xj ∼ BIN(1, p),

H0 : p = p0

and np0 ≥ 10 and n(1 − p0) ≥ 10, # heads (some books use 5 instead of 10 here). Then letp ˆ = and the test n statistic be pˆ − p0 z = p ∼ N(0, 1) p0(1 − p0)/n ¯ (X =p ˆ is assumed to be normal by CLT) for H0. When # heads and # tails are both ≥ 15, a (1−α)100% conﬁdence interval r pˆ(1 − pˆ) for p isp ˆ ± z? when α ≤ 0.1. n Sample size for margin of error, m, is

( ? 2 (z ) pˆ(1−pˆ) pˆ known n = m2 . (z?)2 4m2 pˆ unknown A plus four (1 − α)100% conﬁdence interval for p is obtained by using above procedure, but ﬁrst adding two heads and two tails to the random sample (increasing the sample size to n + 4). Use when sample size is ≥ 10 and α ≤ 0.1.

7 3.2 Two Sample Proportions z–test

Let X1, ··· ,XnX and Y1, ··· ,YnY be independent r.s. where Xj ∼ BIN(1, pX ) and Yk ∼ BIN(1, pY ). Let

H0 : pX = pY = p # heads where p is unknown. Letp ˆ = . Assume the number of heads and # tosses tails in each sample is at least 5. Deﬁne the pooled estimate of pX and pY to be n pˆ + n pˆ p¯ = X X Y Y nX + nY and the test statistic be

pˆX − pˆY z = q ∼ N(0, 1) p¯(1−p¯) + p¯(1−p¯) nX nY

for H0. A (1 − α)100% CI for pX − pY when the number of heads and tails is at least 10 for each sample is s ? pˆX (1 − pˆX ) pˆY (1 − pˆY ) (ˆpX − pˆY ) ± z + . nX nY

A plus four (1 − α)100% conﬁdence interval for pX − pY is obtained by using above procedure, but ﬁrst adding one head and one tail to each of the random samples (increasing each sample size by 2). Use when α ≤ 0.1..

8 4 Correlation

The linear correlation coeﬃcient for (x1, y1), ··· , (xn, yn) is

Pn Pn Pn n j=1 xjyj − j=1 xj j=1 yj r = . r 2r 2 Pn 2 Pn Pn 2 Pn n j=1 xj − j=1 xj n j=1 yj − j=1 yj

The test statistic for H0 : ρ = 0 is r n − 2 r ∼ t(n − 2). 1 − r2

for H0.

9 5 Chi–Squared Tests

5.1 Goodness of Fit

Let X1, ··· ,Xn be a categorical r.s. where there is a total of k categories and th P (X = j category) = pj. Let

H0 : p1 = a1, ··· , pk = ak

where the aj’s are given. Deﬁne

def th oj = # of j categories observed def th ej = naj = # of j categories expected under H0

and assume that ej ≥ 1 for all j’s and that no more than a ﬁfth of the expected counts are < 5. In this case, the test statistic is

k 2 X (oj − ej) ∼ χ2(k − 1) e j=1 j

2 under H0 and one rejects H0 for large χ values.

10 5.2 Chi–Squared Test for Independence

Given a two–way table, oij, of observed outcomes, with r possible row outcomes and c possible column outcomes, let

H0 : there is no relationship between column and row variables.

Deﬁne

def oij = cell ij total (ith row total)(jth column total) e def= = expected count in cell ij under H ij table total 0

and assume that eij ≥ 1 for all cells and that no more than a ﬁfth of the expected counts are < 5. In this case, the test statistic is

r r 2 X X (oij − eij) ∼ χ2((r − 1)(c − 1)) e i=1 j=1 ij

2 under H0 and one rejects H0 for large χ values.

6 Simple Regression

Given the bivariate random sample, (x1, y1) ··· , (xn, yn)

Statistical Model of Simple Linear Regression: Given a predictor, x, the response, y is y = β0 + β1x + x

where β0 + β1x is the mean response for x. The noise terms, the x’s, are assumed to be independent of each other and to be randomly sampled from N(0, σ).

11 Estimating β0, β1 and σ: The least–squares regression line, y = b0 + b1x is obtained by letting Pn Pn Pn s n( xjyj) − ( xj)( yj) b = r y = j=1 j=1 j=1 and b =y ¯−b x.¯ 1 Pn 2 Pn 2 0 1 sx n j=1 xj − ( j=1 xj)

where b0 is an unbiased estimator of β0 and b1 is an unbiased estimator of β1. The variance of the observed yi’s about the predictedy ˆi’s is

P 2 P 2 P P (y − yˆ ) y − b0 yj − b1 xjyj s2 def= j j = j , n − 2 n − 2 which is an unbiased estimator of σ2. The standard error of estimate (also called the residual standard error) is s, an estimator of σ.

Hypothesis Tests and Conﬁdence Intervals for β0 and β1: Let

s 2 def s def 1 x¯ SEb = and SEb = + n . 1 q n 0 P 2 P 2 n j=1(xj − x¯) j=1(xj − x¯)

SEb0 and SEb1 are the standard error of the intercept, β0, and the slope, β1, for the least–squares regression line.

b1 To test the hypothesis H0 : β1 = 0 use the test statistic t ∼ ∼ t(n−2). SEb1 ∗ A level (1−α)100% conﬁdence interval for the slope β1 is b1±t (n−2)×SEb1 .

b0−b To test the hypothesis H0 : β0 = b use the test statistic t ∼ ∼ SEb0 t(n − 2). A level (1 − α)100% conﬁdence interval for the intercept β0 is ∗ b0 ± t (n − 2) × SEb0 .

Accepting H0 : β1 = 0 is equivalent to accepting H0 : ρ = 0.

12 (1 − α)100% Conﬁdence Interval for a mean response, µy: A (1−α)100 % conﬁdence interval for the mean response, µy when x takes ? on the value x isµ ˆy ± m where the margin of error is s 1 (x? − x¯)2 m = t (n − 2) s + . α/2 Pn 2 n j=1(xj − x¯) | {z } SEµˆ

The standard error of the mean response is SEµˆ.

(1 − α)100% Prediction Interval for future observation y given x = x?: A (1 − α)100% Prediction Interval for y given x = x? isy ˆ ± m where ? yˆ = b0 + b1x and the margin of error is s 1 (x? − x¯)2 m = t (n − 2) s 1 + + . α/2 Pn 2 n j=1(xj − x¯) | {z } SEyˆ

Test for Correlation: Consider the hypotheses

H0 : ρ = 0 vs HA : ρ 6= 0

The test statistic is r t = ∼ t(n − 2) for H0. q 1−r2 n−2

13 n X 2 The following holds for sum of squares: (yj − y¯) = j=1 | {z } SSTOT n n X 2 X 2 (ˆyj − y¯) + (yj − yˆj) . The mean squares which equal the j=1 j=1 | {z } | {z } SSA SSE sum of squares divided by it’s corresponding degree of freedom, SS MS def= Mean Square of Model = A and A 1 SS MS def= Mean Square of Error = s2 = E . E n − 2 The coeﬃcient of determination is the portion of the variation in y explained by the regression equation

Pn 2 SS (ˆyj − y¯) r2 def= A = j=1 . Pn 2 SSTOT j=1(yj − y¯)

ANOVA F Test for Simple Linear Regression: Consider H0 : β1 = 0 versus HA : β1 6= 0.

def MSA If H0 holds, f = is from F (1, n − 2) and one uses a right–sided test. MSE

The following is an ANOVA Table for Simple Linear Regression:

Source SS df MS ANOVA F Statistic p–value Model SSA 1 MSA f P (F (1, n − 2) ≥ f) Error SSE n − 2 MSE Total SSTOT n − 1

14 7 Multivariate Regression

Given multivariate variate random sample

(1) (1) (1) (2) (2) (2) (n) (n) (n) (x1 , x2 , ··· , xk , y1), (x1 , x2 , ··· , xk , y2), ··· , (x1 , x2 , ··· , xk , yn)

Statistical Model of Multivariate Linear Regression: Given a k (i) (i) (i) dimensional multivariate predictor, (x1 , x2 , ··· , xk ), the response, yi, is

(i) (i) yi = β0 + β1x1 + ··· + βkxk + i

(i) (i) where β0 + β1x1 + ··· + βkxk is the mean response. The noise terms, the i’s are assumed to be independent of each other and to be randomly sampled from N(0, σ).

(1) (1) (n) (n) Given a multivariate normal sample, x1 , ··· , xk , y1 , ··· , x1 , ··· , xk , yn , the least–squares multiple regression equation,y ˆ = b0 + b1x1 + Pn 2 ··· + bkxk, is the linear equation that minimizes j=1 (ˆyj − yj) , where def (j) (j) yˆj = b0 + b1x1 + ··· + bkxk . There must be at least k + 2 data points to Pn 2 2 def j=1(yi−yˆi) 2 do obtain the estimators b0, bj’s and s = n−k−1 of β0, βj’s and σ , where

b0, the y–intercept, is the unbiased, least square estimator of β0.

bj, the coeﬃcient of xj, is the unbiased, least square estimator of βj. s2 is an unbiased estimator of σ2 and s is an estimator of σ.

Due to computational intensity, computers are used to obtain b0, bj’s and s2.

15 Hypothesis Tests and Conﬁdence Intervals for the βj’s: To test the hypothesis H0 : βj = 0 use the test statistic

bj t ∼ ∼ t(n − k − 1) for H0. SEbj

∗ A level (1 − α)100% conﬁdence interval for βj is bj ± t (n − k − 1)SEbj .

SEbj is the standard error of βj (obtained from computer calculations).

Accepting H0 : βj = 0 is accepting that there is no linear association between Xj and Y , ie that correlation between Xj and Y is zero.

16 ANOVA Tables for Multivariate Regression: n X 2 The following holds for sum of squares: (yj − y¯) = j=1 | {z } SSTOT n n X 2 X 2 (ˆyj − y¯) + (yj − yˆj) . The mean squares which equal the j=1 j=1 | {z } | {z } SSA SSE sum of squares divided by it’s corresponding degree of freedom: SS MS def= Mean Square of Model = A and A k SS MS def= Mean Square of Error = s2 = E . E n − k − 1 ANOVA F Test for Multivariate Regression: The test statistic for

H0 : β1 = β2 = ··· = βk = 0 versus HA : not H0 is f = MSA . The p–value of the above test is P (F ≥ f) where F ∼ MSE F (k, n − k − 1).

ANOVA Table:

Source df Sum of Squares Mean Square F p–value MSA Model k SSA MSA P (F (k, n − k − 1) ≥ f) MSE Error n − k − 1 SSE MSE Total n − 1 SSTOT

17 Multiple Correlation Coefficient: def The squared multiple correlation, R2 = SSA , measures the portion SSTOT of the total variation that is explained√ by the model. The multiple correlation coefficient is just R = R2. The n − 1 adjusted coefficient of determination = R2 def= 1− (1−R2) adj n − k − 1 is a more accurate R2 for large k.

18 8 One–Way ANOVA

k = # of levels th nj = sample size from j level population k X N = nj = total # of r.v.’s j=1 th x¯j = sample mean from j level population 2 th sj = sample var from j level population x¯ = sample mean from all level populations

k ni X X 2 SSTOT = (xij − x¯) = Sum of Squares total i=1 j=1 k X 2 SSA = nj(¯xj − x¯) = SS between levels of treatment A j=1 k X 2 SSE = (nj − 1)sj = SS within levels of treatment A j=1 SS MS = TOT = Mean Squares Total TOT N − 1 SS MS = A = Mean Squares Treatment A K − 1 SS MS = E = Mean Squares Error E N − k MS f = A MSE

SSTOT = SSA + SSE.

Source df SS MS F p MSA Between k − 1 SSA MSA P (F(k − 1,N − I) ≥ f) MSE Within N − k SSE MSE Total N − 1 SSTOT

19 9 Two–Way ANOVA (2 treatments)

I def= #levels for Treatment A J def= #levels for Treatment B def SSA = Sum of Squares of for Treatment A SS MS def= A = Mean Squares of Treatment A A I − 1 def SSB = Sum of Squares of for Treatment B SS MS def= B = Mean Squares of Treatment B B J − 1 def SSAB = Sum of Squares of Non–additive part SS MS def= AB = Mean Squares of Non–additive part AB (I − 1)(J − 1) def SSE = Sum of Squares within treatments SS MS def= E = Mean Squares within treatments E n − IJ def SSTOT = Total Sum of Squares

SSTOT = SSA + SSB + SSAB + SSE.

Source df SS MS F p MSA Treatment A I − 1 SSA MSA P (F(I − 1,N − IJ) ≥ observed F) MSE MSB Treatment B J − 1 SSB MSB P (F(J − 1,N − IJ) ≥ observed F) MSE MSAB Interaction (I − 1)(J − 1) SSAB MSAB P (F((J − 1)(I − 1),N − IJ) ≥ observed F) MSE Error N − IJ SSE MSE Total N − 1 SSTOT

20 10 Addendum

The rules for the minimum sample size to use a test are human convention and diﬀer somewhat from statistician to statistician and book to book.