The Pivotal Quantity Method, the P-Value, & Some Basic Tests

Review: The pivotal quantity method, the p-value, & some basic tests

We have learned three approaches to derive the tests:

- Likelihood Ratio Test - Roy’s Union Intersection Principle - Pivotal Quantity Method

Part 1. ***Now we review the Pivotal Quantity Method.

Example. Inference on 1 population mean, when the population is normal and the population variance is known  the Z-test.

Definition : the Pivotal Quantity (P.Q.) :

A pivotal quantity is a function of the sample and the parameter of interest. Furthermore, its distribution is entirely known.

 2 1. We start by looking at the point estimator of  . X ~ N(, ) n

* Is X a pivotal quantity for  ?

→ X is not because  is unknown.

 2 * function of X and  : X   ~ N(0, ) n

→ Yes, it is pivotal quantity.

* Another function of X and  :

X   Z  ~ N(0,1) is our P.Q.  n

Yes, it is a pivotal quantity.

So, Pivotal Quantity is not unique.

2. Now that we have found the pivotal quantity Z, we shall start the derivation for the test for µ using the pivotal quantity Z

Definition: The test statistic is the PQ with the value of the parameter of interest under the null

hypothesis ( H 0 ) inserted

(*in this case, it is 휇 = 휇):

H0 X  0 Z 0  ~ N(0,1) is our test statistic.  n

X  0 That is, given H :    in true  Z 0  ~ N(0,1) 0 0  n

3. * Derive the decision threshold for your test based on the Type I error rate the significance level  For the pair of hypotheses:

H :    H :    a 0 0 0 versus

It is intuitive that one should reject the null hypothesis, in support of the alternative hypothesis, when the sample mean is

larger than 0 . Equivalently, this means when the test statistic

Z0 is larger than certain positive value c - the question is what is

the exact value of c -- and that can be determined based on the significance level α—that is, how much Type I error we would allow ourselves to commit.

Setting:

P(Type I error) = P(reject H | H ) = 0 0 P(Z  c | H :    )   0 0 0

c  z We will see immediately that  from the pdf plot below.

∴ At the significance level α, we will reject H 0 in favor of H a if

Z 0  Z

Other Hypotheses

H :  0 0 (one-sided test or one-tailed test) Ha : 0

X   Test statistic : Z 0 ~ N (0,1) 0  / n

  P(Z 0  c | H 0 :   0 )  c  Z

H :  0 0 (Two-sided or Two-tailed test) H a : 0

X   Test statistic : Z 0 ~ N (0,1) 0  / n

  P(| Z 0 | c | H 0 )  P(Z 0  c | H 0 )  P(Z 0  c | H 0 )

 2  P(Z 0  c | H 0 )

  P(Z  c | H ) 2 0 0

c  Z 2

Reject H 0 if | Z 0 | Z 2

Part 2. We have just discussed the “rejection region” approach for decision making. There is another approach for decision making, it is “p-value” approach.

*Definition: p-value – it is the probability that we observe a test statistic value that is as extreme, or more extreme, than the one we observed, given that the null hypothesis is true.

H :  H :  H :  0 0 0 0 0 0 H a : 0 Ha : 0 H a : 0

H0 X  0 Observed value of test statistic Z 0  N(0,1)  n ~

p-value p-value p-value  P(| Z 0 || z0 || H 0 )

 P(Z 0  z0 | H 0 )  P(Z 0  z0 | H 0 )  2  P(Z 0 || z0 || H 0 ) (1) the area under (2) the area under (3) twice the area to N(0,1) pdf to the N(0,1) pdf to the the right of | z0 | right of z0 left of z0

H 0:  0 (1) H a : 0

H0:  0 (2) Ha : 0

H0:  0 (3) H a : 0

The way we make conclusions is the same for all hypotheses.

We reject 푯ퟎ in favor of 푯풂 iff p-value < α

Part 3. CLT -- the large sample scenario: Any population (*usually non-normal – as the exact tests should be used if the population is normal), however, the sample size is large (this usually refers to: n ≥ 30)

Theorem. The Central Limit Theorem

Let X 1 , X 2 ,, X n be a random sample from a population with

mean  and variance  2 , we have:

X   n N (0,1)  n 

* When n is large enough (n  30) ,

X    Z  N(0,1) (approximately) – by CLT and the S n ~

Slutsky’s Theorem

Therefore the pivotal quantities (P.Q.’s) for this scenario:

X X   Z~ NZ (0,1) or  ~  N (0,1)  n S n

Use the first P.Q. if σ is known, and the second when σ is unknown.

The derivation of the hypothesis tests (rejection region and the p-value) are almost the same as the derivation of the exact Z-test discussed above.

H :  H :  H :  0 0 0 0 0 0 H a : 0 Ha : 0 H a : 0

 X  0 Test Statistic Z 0  N(0,1) S n ~

Rejection region : we reject H 0 in favor of H a at the significance level  if

Z 0  Z Z0  Z | Z 0 | Z 2 p-value p-value p-value  P(| Z 0 || z0 || H 0 )

Example. Normal Population, but the population variance is unknown

100 years ago – people use Z-test

This is OK for n large (n  30)  per the CLT (Scenario 2)

This is NOT ok if the sample size is samll.

“A Student of Statistics”

– pen name of William Sealy Gosset (June 13, 1876–October 16, 1937)

“The Student’s t-test”

X   P.Q. T ~ t S/ n n1

(Exact t-distribution with n-1 degrees of freedom)

“A Student of Statistics” – pen name of William Sealy Gosset

(June 13, 1876–October 16, 1937)

http://en.wikipedia.org/wiki/William_Sealy_Gosset

“The Student’s t-distribution”

X   P.Q. T ~ t S/ n n1

(Exact t-distribution with n-1 degrees of freedom )

Wrong Test for a 2-sided alternative hypothesis

Reject H 0 if |푧| ≥ 푍/

Right Test for a 2-sided alternative hypothesis

Reject H if |푡 | ≥ 푡 0 ,/

(Because t distribution has heavier tails than normal distribution.)

Right Test

H 0:  0 * Test Statistic H a : 0

H0 X  0 T0  tn1 S n ~

* Reject region : Reject H 0 at  if the observed test statistic

value |푡| ≥ 푡,/

* p-value

p-value = shaded area * 2

Further Review:

1. Definition : t-distribution Z T  ~ t k W k

Z ~ N (0,1)

2 W ~  k (chi-square distribution with k degrees of freedom)

Z &W are independent.

2. Def 1 : chi-square distribution : from the definition of the gamma distribution: gamma(α = k/2, β = 2)

/ MGF: 푀(푡) = mean & varaince: 퐸(푊) = 푘; 푉푎푟(푊) =2푘

i.i.d . Z , Z ,, Z N (0,1) Def 2 : chi-square distribution : Let 1 2 k ~ ,

k 2 2 then W   Z i ~  k i1

Example. Inference on 2 population means, when both populations are normal and we have two independent samples. Furthermore the population variances are unknown but

2 2 2 equal (1  2   ) pooled-variance t-test.

iid Data: X,, X ~(, N   2 ) 1n1 1 1

iid Y,, Y ~(, N   2 ) 1n2 2 2

Goal: Compare 1 and 2

Pivotal Quantity Approach:

1) Point estimator:

2 2 1  2 1 1 2 ˆ1 ˆ 2XYN~(  1  2 ,  )( N  1 2 ,(  ))  n1 n 2 n 1 n 2

2) Pivotal quantity:

(X Y )(    ) Z 1 2 ~ N (0,1) not the PQ, since we 1 1    n1 n 2 don’t know  2

(n 1) S 2 (n 1) S 2  1 1 ~  2 , 2 2 ~  2 , and they are  2 n1 1  2 n2 1 2 2 independent ( S1 & S2 are independent because these two samples are independent to each other)

(n 1) S2 ( n  1) S 2 W 1 1  2 2 ~  2 2  2 n1 n 2 2

i... i d 2 2 2 Definition: WZZ1  2  Zk , when Zi ~ N (0,1) , then

2 W ~ k .

Definition:

2 t-distribution: Z~ N (0,1) , W ~ k , and Z & W are Z independent, then T  ~ t W k k



(X Y )( 1   2 ) 1 1    Z n1 n 2 (X Y )( 1   2 ) T  ~ tn n 2 W (n 1) S2 ( n  1) S 2 1 1 1 2 1 1 2 2 S  2 2 p n1 n 2  2   n 1 n 2

n1 n 2  2 .

2 2 2 (n1 1) S 1  ( n 2  1) S 2 where S p  is the pooled variance. n1 n 2  2

This is the PQ of the inference on the parameter of interest

(1  2 )

3) Confidence Interval for (1  2 )

1 Pt (  Tt  ) n n2, n  n 2, 1 22 1 2 2

(X Y )( 1   2 ) 1 Pt (   t  ) n n2, n  n 2, 1 221 1 1 2 2 S p  n1 n 2 1 1 1 1 1(P t  Sp  ()() X Y 1 2 t   S p  ) n n2, n  n 2, 1 22n1 n 2 1 2 2 n 1 n 2 1 1 1 1 1PXYt (   Sp  1 2 XYt  S p  ) n n2, n  n 2, 1 22 n1 n 2 1 2 2 n1 n 2

 This is the 100(1 )% C.I for (1  2 )

4) Test:

(XY )  c H0 Test statistic: T 0 ~ t 01 1 n1 n 2  2 S p  n1 n 2

H0: 1  2  c 0 a)  (The most common situation Ha :1  2  c 0

is c0  0  H0: 1  2 )

At the significance level α, we reject H0 in favor of H iff T t a 0n1 n 2  2,

If P value  P( T0 t 0 | H 0 )   , reject H0

H0: 1  2  c 0 b)  Ha :1  2  c 0

At significance level α, reject H0 in favor of H a

iff T t 0n1 n 2  2,

If P value  P( T0 t 0 | H 0 )   , reject H0

H0: 1  2  c 0 c)  Ha :1  2  c 0

At α=0.05, reject H0 in favor of H a iff

|T0 | t  n n 2, 1 2 2

If P value2 P ( T0  | t 0 || H 0 )   , reject H0

Inference on two population variances

* Both pop’s are normal, two independent samples

i.i.d. (n 1)S 2 Sample 1 : X , X ,, X ~ N( , 2 ) ⇒ 1 1 ~  2 1 2 n1 1 1 2 n1 1  1

i.i.d. (n 1)S 2 Sample 2 : Y ,Y ,,Y ~ N( , 2 ) ⇒ 2 2 ~  2 1 2 n2 2 2 2 n2 1  2

2  1 (*** Parameter of interest : 2 )  2

2 2 1. Point estimator : ˆ1  S1

2 2 ˆ 2  S 2

Def. F-distribution Let W ~  2 , W ~  2 , W ,W are independent. 1 k1 2 k2 1 2

W k Then, F  1 1 ~ F k1 ,k2 W2 k2

  P(F  F ) k1 ,k2 , ,lower

1 1  P(  ) F F k1 ,k2 , ,lower

1 ~ F F k2 ,k1

1 F  k1 ,k2 , ,lower F k2 ,k1 , ,upper

(n 1)S 2 S 2 1 1 (n 1) 1  2 1 S 2 F  1  2 ~ F 2 2 n1 1,n2 1 (n2 1)S 2  1 2 (n2 1) 2  2  2

2  1 2. 100(1 )% CI for 2  2

2 S1 S 2 1  P(F  2  F )  ,lower 2  ,upper 2 2  1 2  2

2  1 2 1  2 1  P(  2  ) F S1 F 2,lower 2,upper 2 S 2

2 2 S1 S1 S 2  2 S 2  P( 2  1  2 ) F  2 F  ,upper 2  ,lower 2 2

3. Test

2  1 H 0 : 2  1  2

2  1 H a : 2  1  2

2 S H0 Test Statistic F  1 ~ F 0 2 n1 1,n2 1 S2

At the significance level  , we reject H 0 if F0 is too large or too small.

F0  c1 , F0  c2

* conventional boundries / thresholds

c  F 1 n 1,n 1, ,upper 1 2 2

c  F 2 n 1,n 1, ,lower 1 2 2

Data Analysis Example.

Example A new method of making concrete blocks has been proposed. To test whether or not the new method increases the compressive strength, 5 sample blocks are made by each method.

New 14 15 13 15 16 Method

Old Method 13 15 13 12 14

a. Get a 95% for the mean difference of the 2 methods. b. At  = 0.05, can you conclude the new method is better? Provide p-value.

Solution

a. Assume both populations are normal.

2 2 First, we check whether  1   2

2 2 H 0 : 1   2

2 2 H a : 1   2

2 S1 Test Statistic : F0  2 S 2

n 2 (xi  x) Recall S 2  i1 n 1

 F0  1  F4,4,0.1,upper  4.11

2 2 It is reasonable to assume  1   2

Pooled-variance statistic (PQ)

(X 1  X 2 )  (1  2 ) t  ~ tn n 2 1 1 1 2 S p  n1 n2

95% CI for (1  2 ) is

1 1 (X  X )  t  S   1 2 n1 n2 2,0.025 p n1 n2

2 2 (n1 1)S1  (n2 1)S2 S p  n1  n2  2 b. Assume both populations are normal.

2 2 First, we check whether  1   2

2 2 By part (a), we found that it is reasonable to assume  1   2

H 0 : 1  2  0

H a : 1  2  0

H0 X 1  X 2  0 Test Statistic : T0  ~ tn n 2 1 1 1 2 S p  n1 n2

At  =0.05, we reject H if T  t  t . 0 0 n1 n2 2,0.05 8,0.05

But T0 ( 1.66)  t8,0.05 ( 1.86)

We cannot reject H 0 at  =0.05.

LRT Derivation of the Pooled Variance T-Test

Given that we have two independent random samples from two normal populations with equal but unknown variances. Now we derive the likelihood ratio test for:

H:μ =μ vs H :μ ≠μ

Let μ =μ =μ, then,

ω={−∞<μ =μ =μ<+∞,0≤σ < +∞}

Ω={−∞<μ,μ <+∞,0<σ < +∞}

L(ω) =L(μ,σ) =( ) exp [− ∑ (x −μ) + ∑y −μ ], and there are two parameters.

lnL(ω) =− ln(2πσ) − ∑ (x −μ) + ∑ y − μ , since it contains two parameters, we take the partial derivatives with μ and σrespectively and set the partial derivatives equal to 0. Solving them we have:

∑ x + ∑ y nx +ny μ = = n +n n +n

1 σ = [ (x −μ) + y −μ ] n +n

L(Ω) = L(μ ,μ ,σ) =( ) exp [− ∑ (x −μ ) + ∑y −μ ], and there are three parameters.

n +n lnL(Ω) =− ln(2πσ) 2 1 − (x −μ) + y −μ 2σ

We take the partial derivatives with μ,μ and σ respectively and set them all equal to 0. Then solutions are:

μ =x,

μ =y,

1 σ = [ (x −x) + y −y ] n +n

Next we take the likelihood ratio. After some simplifications, we have:

1 L(ω) 2πσ σ λ= = = LΩ σ 1 2πσ

∑ (x −x) + ∑y −y = nx +ny nx +ny ∑ x − + ∑ y − n +n n +n

t = [1+ ] n +n −2 where t is the test statistic in the pooled variance t-test.

∗ Therefore, λ ≤ λ is equivalent to |t|≥c. Thus at the significance level α, we reject the null hypothesis in favor of the alternative when |t| ≥ c = t,α/

In general, for LR test, we know that when the sample size goes to infinity, we have

ퟐ – ퟐ퐥퐧훌~훘 퐤 where k is the difference in the number of free parameters under the alternative and the null hypothesis respectively.

Note: In the pooled-variance t-test derivation we have in the previous page:

k = 3 – 2 = 1

Group Quiz:

Let 푋, 푋,⋯, 푋 denote a sample of size n from the p-variate normal distribution with mean vector 휇and covariance matrix where is known.

Suppose we want to test 퐻: 휇 = 휇 versus 퐻: 휇 ≠ 휇 at the significance level α. Please derive the test using: (a) The pivotal quantity method; (b) The Roy’s union-intersection principle; (c) The likelihood ratio test.