Median ofthe p Value Underthe Alternative Hypothesis

Bhaskar BHATTACHARYA and Desale HABTZGHI

ormoreextreme than, the value actually observed. Thus, if the p valueis lessthan the preferredsigniŽ cance level, then one rejects Dueto absence of anuniversally acceptable magnitude of the H0.Over theyears several authors have attempted to properly TypeI error invarious Ž elds, p valuesare used as a well- explain p values.Gibbons and Pratt (1975) provided interpre- recognizedtool in decision making in all areas of statistical tationand methodology of the p values.Recently, Schervish practice.The distribution of p valuesunder the null hypothe- (1996)treated the p valuesas signiŽ cance probabilities. Discus- sisis uniform. However, under the alternative hypothesis the sions on p valuescan be foundin Blythand Staudte (1997) and distributionof the p valuesis skewed. The expected p value inDollinger, Kulinskaya, and Staudte (1996). However, these (EPV) hasbeen proposed by authorsto beusedas a measureof articlesdo nottreat the p valuesas random. theperformance of thetest. This article proposes the median of The p valuesare based on the test statistics used and hence the p values(MPV) whichis more appropriate for thispurpose. random.The stochastic nature of the p valueswas investigated Weworkout many examples to calculate the MPV’ s directly byDempster and Schatzoff (1965) and Schatzoff (1966) who andalso compare the MPV withthe EPV .Weconsidertesting equalityof distributionsagainst stochastic ordering in themulti- introduced“ expectedsigniŽ cance value.” Recently, Sackrowitz nomialcase and compare the EPV’ s andMPV’ s bysimulation. andSamuel-Cahn (1999) investigated this concept further and Asecondsimulation study for generalcontinuous data is also renamedit as the expected p value(EPV). Underthe null hy- consideredfor twosamples with different test statistics for the pothesis,the p valueshave a uniformdistribution over (0, 1) for samehypotheses. In both cases MPV performs betterthan EPV . anysample size. Thus, under H0, EPV is 1/2 always, and there isno wayto distinguish p valuesderived from largestudies and KEY WORDS: Comparingtest statistics; Expected p value; thosefrom small-scalestudies. Also it would be impossibleun- Median p value;Power; Stochastic order. der H0 todifferentiatebetween studies well powered to detect apositedalternative hypothesis and the underpowered to detect thesame posited alternative value. Incontrast, the distribution of the p valuesunder the alter- 1.INTRODUCTION nativehypothesis is a functionof thesample size and the true parametervalue in the alternative hypothesis. As the p values Thetheory of hypothesistesting depends heavily on thepre- measureevidence against the null hypothesis, it is of interest to speciŽed valueof the signiŽ cance level. T oavoidthe nonunique- investigatethe behavior of the p valuesunder the alternative at nessof the decision of testing the same hypotheses using the varioussample sizes. W ereject H when p valueis smallwhich sametest statistic but different signiŽ cance levels, it is apop- 0 isexpected when H istrue.As notedby Hung,O’ Neill,Bauer, ularchoice to report the p value. The p valueis the smallest 1 andKohne (1997), the distribution of the p valuesunder the levelof signiŽcance at which an experimenterwould reject the alternativeis highlyskewed. The skewness increases with the nullhypothesis on thebasis of theobserved outcome. The user samplesize and the true parameter value under the alternative cancompare his/ herown signiŽ cance level with the p value and reecting the ability to detectthe alternativeby increasingpower makehis/ herown decision. The p valuesare particularly useful underthese situations. Hence it ismore appropriate to consider incaseswhen the null hypothesis is well deŽ ned but the alterna- themedian of the value(MPV) insteadof theEPV underthe tiveis not(e.g., composite) so thatT ypeII error considerations p areunclear. In this context to quote Fisher, “ Theactual value of p alternativeas a measureof thecenterof its distributionwhich is obtainablefrom thetable by interpolation indicates the strength themain focus of thisarticle. Applications of thedistribution of ofevidenceagainst the null hypothesis.” the p valuesunder the alternative in thearea of meta-analysisof Wewillconsider tests of theform “Reject H when T c,” severalstudies is consideredby Hunget al.(1997). Studying the 0 ¶ where T isareal-valuedtest statistic computed from datawhen p valueunder the alternative is alsobeneŽ cial over the power of atestand is explainedin thenext paragraph. testingthe null hypothesis H0 againstthe alternative H1. The value c isdetermined from theprespeciŽ ed sizerestrictions such Whenseveral test procedures are available for thesame test- that P (T c) = ¬ .Of course,when T isadiscreterandom ingsituation one compares them by meansof power.However, H0 ¶ variable,one needs to adoptrandomization so that all sizes are powercalculations depend on the chosen signiŽ cance levels, possible.If t istheobserved value of T andthedistribution of T andin discrete cases involves randomization. These steps can under H is given by F ( ), then the p valueis givenby 1 F (t) beavoidedby considering MPV’ s. Also as thepower functions 0 0 ¢ ¡ 0 whichis theprobabilityof Ž ndingthe test statistic as extremeas, dependon the chosen signiŽ cance levels, it isdifŽcult to compare themwhen different power functions use different signiŽ cance levels.On the otherhand, MPV’ s dependonly on thealternative BhaskarBhattacharya is Associate Professor,Department ofMathematics, andnot on the signiŽ cance level. The smaller the valueof MPV , SouthernIllinois University, Carbondale, IL 62901(E-mail: Bhaskar@math. siu.edu).Desale Habtzghiis aGraduate Student,Department ofStatistics, Uni- thestronger the test. The value of an MPV cantell us which versityof Georgia,Athens, GA 30602. alternativean attained p valuebest represents for agivensam-

202The American Statistician, August 2002, V ol.56, No. 3 c 2002American Statistical Association DOI: 10.1198/000313002146 ® 1 plesize. Also, it helpsto knowthe behavior of theMPV’ s for Since F (t) F (t); t,itisseen that F (F ¡ (:5)) :5 which ³ µ 0 8 0 ³ ¶ varyingsample sizes. impliesthat ¬ :5 andequality holds when H is true. The ¤ µ 0 InSection 2, we obtaina generalexpression for theMPV’ s. smallerthe MPV thebetter it istodetectthe alternative. Given Wealsoderive a computationallyfavorable form tobe used thestochastic nature of the p valueunder the alternative, it is laterin thearticle. In Section 3, we considerseveral examples to alsotrue that the MPVissmallerthan the EPV andhence clearly computethe MPV’ s directly.In Section 4, we performa simu- preferableover the EPV .TheMPV beingsmaller than the EPV lationstudy to compute the MPV’s inthecase of testingagainst producesa smallerindifference region in thesense that higher stochasticordering for multinomialdistributions. W ealsoper- poweris generated closer to the null hypothesis region using form asecondsimulation study for generaldistributions when MPVthanwith using EPV . testingthe same hypotheses with two samples using three test Itis alsopossible to expressthe MPVinanother way. Let T ¹ statisticsof t test,Mann– Whitney– Wilcoxon, and Kolmogorov– F ( ) and,independently, T F ( ).If theobserved value of ³ ¢ ¤ ¹ 0 ¢ Smirnov.In Section 5, we makeconcluding remarks. T is t, then the p valueis simply g(t) = P (T t T = t). The ¤ ¶ j MPV is 2. VALUE ASARANDOM VARIABLE AND ITS p med g(t) = MPV(³ ) = P (T ¤ med T ): (3) MEDIAN ¶ If H0 istrue, then T and T ¤ areidentically distributed, and hence

For thetest statistic T withdistribution F0( ) under H0, let theabove probability is .5 for anycontinuous distribution. For 1 ¢ discretedistributions, although the above expressions still hold, F³ ( ) beitsdistribution under H1. Also, let F0¡ ( ) betheinverse ¢ 1 ¢ function of F ( ), so that, F (F ¡ (® )) = ® , for any 0 < ® < 1. MPVwillbe slightly higher as P (T ¤ = med T ) > 0. For an 0 ¢ 0 0 Since the p valueis theprobability of observinga moreextreme UMPtest,the MPV willbe uniformlyminimal for all ³ values valuethan the observedtest statistic value, as a randomvariable inthe alternative as comparedto theMPV’ s ofanyother test of itcanbe expressedas the same H0 versus H1.For computationalpurposes the above form oftheMPV in(3) isvery useful. X = 1 F0(T ): ¡ 3. EXAMPLES As F (T ) U(0; 1) under H , so is X.Thepower of the test 0 ¹ 0 Thissection considers several examples to calculate the isrelatedto the p value as MPV’s directly.

­ = P³ (X ¬ ) µ Example 1. Let X1; X2; : : : ; Xn bearandomsample of size n 1 2 = 1 F³ (F0¡ (1 ¬ )): (1) from a N(· ; ¼ ) distribution,and we liketo test H : · · ¡ ¡ 0 µ 0 versus H : · > · where ¼ isknown. Using the test statistic Notethe above is also the distribution function of the p value 1 0 T = X anda particularvalue · in H itfollows from (2) that underthe alternative. As T isstochastically larger under the 1 1 alternativethan under the null hypothesis, it followsthat the p pn(· · ) MPV = © 0 ¡ 1 ; (4) valueunder the alternative is stochastically smaller than under ¼ thenull (Lehmann 1986). This explains why the distribution of µ ¶ the p valueis skewedto therightunder the alternative. Hence to where © isthecdf of thestandard normal distribution. It is well estimatethe center of thedistribution of the p value,the median knownthat for asize ¬ testto achieve power ­ ata speciŽed isa betterchoice than the mean. The median is any value of alternativevalue · 1 thesample size n satisŽ es

¬ = ¬ ¤ whichsatisŽ es 2 2 (z1 ¬ + z­ ) ¼ ¡ n = 2 ; (5) P (X < ¬ ¤ ) :5 and P (X > ¬ ¤ ) :5: (· 0 · 1) ³ µ ³ µ ¡

For continuousdistributions, the median is the value of ¬ = ¬ ¤ where z® is the ® thquantile of thestandard normal distribution. whichsatisŽ es Using(4) and(5), it follows that

P³ (X ¬ ¤ ) = :5: MPV = ©( z1 ¬ z­ ): (6) µ ¡ ¡ ¡ Itfollows from (1) bysimplemanipulations that InT able1, we haveprovided values of theMPV’ s in(6) using

1 somecommonly used values of ¬ and ­ .EachMPV valueis ¬ ¤ = 1 F (F ¡ (:5)): (2) ¡ 0 ³ smallerthan the corresponding EPV valueof Table1 ofSack- rowitzand Samuel-Cahn (1999). W ehavealso graphed the EPV Table1. MPV’s in the T estingfor the Normal Mean as a Func- andthe MPV inFigure 1 for variousvalues of · > 0 when tionof theSigni ŽcanceLevel ¬ and Power ­ 1 · 0 = 0 atsample sizes 10 and50. The MPV’ s decreasefrom the.5 value at much faster rate than the EPV’ s althoughfor ¬ =­ 0.400.50 0.60 0.70 0.80 0.90 0.95 boththe rate increases with the sample size. In Figure 2, we 0.010.0191 0.0100 0.0049 0.0022 0.0008 0.0002 0.0000 havegraphed the EPV andthe MPV for varioussample sizes at 0.050.0820 0.0500 0.0288 0.0150 0.0065 0.0017 0.0005 · 1 = :3 and at · 1 = :5.Itis observed that the MPV’ s decrease 0.100.1519 0.1000 0.0624 0.0355 0.0169 0.0052 0.0017 ata fasterrate than the EPV’ s atsmaller sample sizes and when 0.150.2168 0.1500 0.0986 0.0593 0.0302 0.0102 0.0036 closerto thenull hypothesis.

TheAmerican Statistician, August 2002, V ol.56, No. 3 203 to test H0 : · 1 = · 2 versus H1 : · 1 > · 2 where ¼ is known. Usingthe teststatistic T = X Y anda particularvalue · · ¡ 1 ¡ 2 in H1 itfollows that

mn · · MPV = © 2 ¡ 1 : (7) m + n ¼ µr ¶ When m = n,formula(6) isstill valid and hence T able1 isalso correctin thiscase. For unknown ¼ ,aconsistentestimator of ¼ maybe usedin (7) for moderatelylarge m; n,andthe formula in(7) becomesapproximately correct in this case.

Example 3. Suppose X1; X2; : : : ; Xn isa randomsample of 2 size n from a N(· ; ¼ ) distribution,and we liketo test H0 : ¼ ¼ versus H : ¼ > ¼ when · isunknown. Using the µ 0 1 0 teststatistic (n 1)S2=¼ 2 where S2 isthesample variance, it ¡ followsfrom (4) thatat thealternative point ¼ 1,

2 ¼ 1 1 MPV = 1 G G¡ (:5) ; (8) ¡ ¼ 2 µ 0 ¶ Figure1. EPV’s and MPV’s at various · 1 >0whentesting for the where G isthe cdfof achi-squaredistribution with n 1 degrees normalmean with · 0 =0forsample sizes n =10and n =50. ¡ offreedom.Note we neednot use the F -distributionas needed for thecomputation of theEPV inthis problem (Sackrowitz and If thevalue of ¼ isunknown,the sample standard deviation S Samuel-Cahn1999). (or anyother consistent estimator of ¼ )maybe usedto replace Thefollowing two examples are concerned with testing the itfor moderatelylarge .Theformula in (4) isapproximately n scaleand location parameters of theexponential distribution. correctin this case, consequently, T able1 isapproximately cor- rectfor theone-sample t testsituation. After observingan actual Example 4. Suppose T isexponentially distributed with pa- valuefor anapproximately normally distributed statistic, the t³ p rameter ³ (from pdf f(t) = ³ e¡ for t > 0) denoted by exp(³ ) expressionin (4) canbe usedto determine the · 1 for whichthe andwe liketo test H : ³ ³ versus H : ³ < ³ . It is seen 0 ¶ 0 1 0 givenvalue would be anMPV . that med T = (1=³ )ln 2.For aparticularvalue of ³ 1 < ³ 0, if T exp(³ ) and T exp(³ ),theMPV isgivenby ¤ ¹ 0 ¹ 1 Example 2. Let X1; X2; : : : ; Xn bearandomsample of size ³ =³ 2 P (T ¤ med T ) = 2¡ 0 1 : n from N(· 1; ¼ ),andindependently, let Y1; Y2; : : : ; Ym be ¶ 2 anotherrandom sample of size m from N(· 2; ¼ ), and we like For a size ¬ testwith power ­ , since ln ¬ = ln ­ = ³ 0=³ 1, it fol- 7 lowsthat for ¬ = :1 and ­ = :9, the MPV is 2:645 10¡ . The EPV is.0438in thiscase (Sackrowitz and Samuel-Cahn £ 1999). If arandomsample X1; X2; : : : ; Xn isavailable, the test may n be based on T = i= 1 Xi. Here T hasa gammadistribution withshape parameter n andscale parameter ³ ,we willdenote P its cdf by G ( ).Thenit follows that the MPV atalternative n;³ ¢ point ³ 1 is given by

1 1 P (T ¤ G¡ (:5)) = 1 Gn;³ (G¡ (:5)): ¶ n;³ 1 ¡ 0 n;³ 1

Example 5. Suppose X1; X2; : : : ; Xn isa randomsample of size n from anexponential distribution with parameters · ; ³ (with pdf f(x) = (1=³ )e (x · )=³ for x · ) de- ¡ ¡ ¶ noted by exp(· ; ³ ) andwe liketo test H : · · ver- 0 µ 0 sus H1 : · > · 0 where ³ isknown. The test statistic is T = min(X1; X2; : : : ; Xn) whosedistribution is exp(· ; ³ =n). Itis seenthat med T = · +(³ =n)ln 2.For aparticularalternative n(· 1 · 0)=³ · 1 > · 0,theMPV isgiven by :5e¡ ¡ . 4.SIMULA TION STUDIES Weperformtwo simulation studies to calculate and compare Figure2. EPV’s and MPV’s at various sample sizes when testing for theEPV’ s andMPV’ s. For thediscrete case we considerthe thenormal mean with · 0 = 0 and · 1 =.3 and · 1 = .5. binomialdistribution, with m trialsand probability of success

204 General Table2. EPV’s and MPV’s for T estingH 0 : p = q Against H1: p isStochastically Larger than q forDifferent Combinations of m ; n; p

m = 3 m = 6 EPV MPV EPV MPV p n = 50 n = 100 n = 50 n = 100 n = 50 n = 100 n = 50 n = 100

0.50 0.509 0.508 0.510 0.506 0.509 0.511 0.505 0.510 0.51 0.454 0.432 0.439 0.407 0.445 0.421 0.421 0.401 0.52 0.404 0.364 0.363 0.309 0.380 0.335 0.332 0.279 0.53 0.350 0.291 0.292 0.222 0.319 0.250 0.248 0.164 0.54 0.300 0.235 0.230 0.143 0.258 0.170 0.168 0.079 0.55 0.252 0.174 0.165 0.081 0.202 0.110 0.103 0.033 0.56 0.208 0.129 0.117 0.045 0.149 0.065 0.055 0.011 0.57 0.169 0.087 0.078 0.020 0.102 0.037 0.027 0.003 0.58 0.135 0.054 0.050 0.007 0.069 0.019 0.012 0.001 0.59 0.102 0.036 0.031 0.003 0.044 0.008 0.006 0.000 0.60 0.075 0.021 0.017 0.001 0.026 0.002 0.002 0.000 0.61 0.054 0.011 0.008 0.000 0.013 0.001 0.001 0.000 0.62 0.037 0.006 0.003 0.000 0.007 0.000 0.000 0.000 0.63 0.026 0.003 0.002 0.000 0.003 0.000 0.000 0.000 0.64 0.017 0.001 0.001 0.000 0.001 0.000 0.000 0.000 p, given by likelihoodratio test statistic is given by

m j m j m pj = p (1 p) ¡ ; j = 0; : : : ; m (9) j ¡ T = 2n p^i ln (p =qi) µ ¶ i i= 0 whichis symmetricwhen p = :5. When p > :5,thebinomial X distributionis skewed to the left, that is, it becomes stochas- where p is the ithcoordinate of p = p^E (q=p^ ); E (q=p^ ) i ^p jA ^p jA ticallylarger than the p = :5 case. Let q = (q0; : : : ; qm) isthe isotonic regression of q=p^ (allmultiplications and divi- and p = (p0; : : : ; pm) bethe vectors of binomial probabil- sionsof vectorsare done coordinatewise) onto the nonincreasing itiesobtained from (9) with p = :5 and p > :5, respec- cone = x = (x ; x ; : : : ; x ) : x x x A f 0 1 m 0 ¶ 1 ¶ ¢ ¢ ¢ ¶ mg tively.W econsidertesting H0 : p = q (i.e., pi = qi; i) withweights p^.Itis wellestablished (Robertson, Wright, and m 8 against H1 : p isstochastically larger than q (i.e., pi Dykstra1988) that under H ,asymptotically,the statistic T has i= j ¶ 0 m q ; j = 1; : : : ; m; and m p = m q = 1). The achi-barsquared distribution. i= j i 8 i= 0 i i= 0 iP P Table3. MPV’s for the One-Sided, P T wo-SampleP t ,Mann–Whitney– Wilcoxon (MWW), and Kolmogorov– Smirnov (KS) Tests forVarious Sample Sizes, Shift Parameters and Underlying Distributions

n = 10 n = 20 n = 50 t MWW KS t MWW KS t MWW KS Normal(0,1)

¢ 0 =0.0000 0.49430.5196 0.6745 0.49640.5032 0.6485 0.50660.4927 0.6181 ¢ 1 =0.3600 0.21150.2497 0.4072 0.12750.1368 0.1662 0.03890.0413 0.0888 ¢ 2 =0.5692 0.10840.1271 0.2039 0.03990.0458 0.0886 0.00270.0024 0.0100 ¢ 3 =0.8050 0.04460.0567 0.0843 0.00850.0086 0.0189 0.00010.0000 0.0007 Exponential

¢ 0 =0.0000 0.5022.5125 0.6858 0.49240.5078 0.6397 0.48640.5094 0.6031 ¢ 1 =0.3600 0.20300.1235 0.2157 0.12210.0505 0.0913 0.03090.0041 0.0017 ¢ 2 =0.5692 0.09050.0544 0.0851 0.03360.0104 0.0060 0.00130.0000 0.0000 ¢ 3 =0.8050 0.03120.0168 0.0281 0.00380.0012 0.0004 0.00000.0000 0.0000 Chi-square(10)

¢ 0 =0.0000 0.48630.5202 0.6815 0.51510.4878 0.6491 0.48910.5050 0.5990 ¢ 1 =1.6100 0.19550.2234 0.4172 0.13000.1129 0.1682 0.03490.0318 0.0533 ¢ 2 =2.5456 0.09470.1129 0.2001 0.03760.0299 0.0844 0.00230.0017 0.0054 ¢ 3 =3.6000 0.03410.0376 0.0764 0.00630.0049 0.0153 0.00000.0000 0.0010 Uniform(0,1)

¢ 0 =0.0000 0.49590.5196 0.6745 0.50130.5032 0.6485 0.50860.4927 0.6181 ¢ 1 =0.1039 0.21540.2497 0.4072 0.13050.1426 0.2897 0.03650.0442 0.0888 ¢ 2 =0.1643 0.11250.1271 0.2039 0.04310.0506 0.0886 0.00260.0036 0.0187 ¢ 3 =0.2324 0.04770.0647 0.2039 0.01040.0122 0.0416 0.00000.0001 0.0022 Doubleexponential

¢ 0 =0.0000 0.50160.5119 0.6830 0.50080.5122 0.6507 0.49470.5013 0.6074 ¢ 1 =0.5091 0.21190.1793 0.2138 0.12600.0928 0.1688 0.03290.0156 0.0183 ¢ 2 =0.8050 0.09840.0841 0.0842 0.03570.0197 0.0426 0.00190.0003 0.0004 ¢ 3 =1.1384 0.03700.0327 0.0286 0.00540.0033 0.0068 0.00000.0000 0.0000

TheAmerican Statistician, August 2002, V ol.56, No. 3 205 Wecreatea randomsample T1; : : : ; Tn distributedlike T , and ­ = :7.Theyalso have similar pattern as the corresponding EPV independently,another random sample T1¤ ; : : : ; Tn¤ distributed values.So their conclusions are also valid in ourcase.However, like T ¤ .Anunbiased estimator of EPV isgiven by ourKS valuesof MPVperformworse thanthe corresponding EPV valuesof Sackrowitzand Samuel-Cahn (1999) in all cases 1 n A = I(T ¤ T ); considered. E n i ¶ i i= 1 X 5.CONCLUSION andan unbiasedestimator of MPV isgivenby Thedistribution of the p valuesunder the alternative is a n 1 skeweddistribution to theright, and hence the medianof this dis- A = I(T ¤ med T ): M n i ¶ i tributionis advocated as a moreappropriate tool than its mean i= 1 X for determinationof the strength of a testfor aparticularal- Thevariance of A is EPV(1 EPV)=n and that of A is E ¡ M ternative.The alternatives closer to H0 aredetected easily with MPV(1 MPV)=n ¡ MPVthanwith EPV .TheMPV iseasily computed in mostcases We consider m = 3; 6 withsample sizes n = 50, 100 and anddoes not depend on thespeciŽed signiŽcance level of atest. replications10,000. The results are given in T able2. When p = Thus,it may be used as asinglenumber which can help to choose :5,bothof the EPV andMPV startslightly higher than .5 as amongdifferent test statistics when testing the samehypotheses. expectedfor discretedistributions. For p > :5,theMPV’ s are For approximatelynormally distributed statistics, T able1 canbe consistentlysmaller than the EPV’ s, both being very close to consultedto relatethe MPV valueto theusual signiŽ cance level zero when p > :65.Theeffects are more pronounced for larger andpower combinations. n.Notethe exact value of the EPV ortheMPV isdifŽ cult to calculatein this case. [Received January2001. Revised August2001.] Weconsidera secondsimulation study to compare the per- formanceof severaltests using the MPV’ s for ageneralcontin- uouscase. W eusethe same set up asSackrowitz and Samuel- REFERENCES Cahn(1999) but calculate the MPV’s instead.Thus we consider Blyth,C. R.,and Staudte, R. G.(1997),“ HypothesisEstimation and Acceptabil- thetwo-sample problem of testing H0 : F = G versus H1 : ityProŽ les for 2 2 ContingencyT ables,”Journalof theAmerican Statistical F isstochastically larger than G usingtwo independent random Association ,92,£694– 699. samplesfrom F and G respectively.The test statistics consid- Dempster, A.P.,andSchatzoff, M. (1965),“ ExpectedSigniŽ cance Levelas eredare the two-sample t test,the Mann– Whitney– Wilcoxon aSensibilityIndex for Test Statistics,” Journalof the American Statistical (MWW) testand theKolmogorov– Smirnov (KS) test.The com- Association ,60,420– 436. Dollinger,M. B.,Kulinskaya, E., Staudte, R. G.(1996),“ Whenis a p Value a parisonis madefor shiftalternatives so that G(x) = F (x + ¢), GoodMeasure ofEvidence,”in RobustStatistics, Data Analysis and Com- where F ischosen as various distributions. W econsider F as puterIntensive Methods ,ed.H. Reider,New York:Springer-V erlag,pp. 119– normal(0,1), exponential, chi-square with degrees of freedom 134. 10,uniform (0,1) and double exponential. W echosesample Gibbons,J. D.,and Pratt, J. W.(1975), “ P-values:Interpretation and Methodol- ogy,” TheAmerican Statistician , 29, 20–25. sizes10, 20, 50 and ¢0 = 0; ¢1 = 2:546¼ =p50; ¢2 = Hung,H. M.J.,O’ Neill,R. T.,Bauer, P .,andKohne, K. (1997),“ TheBehavior of 2:546¼ =p20; ¢3 = 2:546¼ =p10, where ¼ istheactual stan- the p Value Whenthe Alternative Hypothesis is True,” Biometrics, 53, 11–22. darddeviation of the underlying distribution F .Thischoice Lehmann,E. L.(1986), TestingStatistical Hypotheses ,(2nded.), New York: ismade so that when the underlying distribution is normal, a Wiley. testbased on the normal two-sample statistic and n observa- Robertson,T., Wright. F. T., and Dykstra, R. L.(1988), Order Restricted Statis- tionswould have size ¬ = :10 and power ­ = :7 (as opposed ticalInference ,New York:Wiley. to ­ = :9 ofSackrowitz and Samuel-Cahn 1999, table 4) for Sackrowitz,H., and Samuel-Cahn, E. (1999),“ P Values as RandomV ariables- ¢ = (2:546¼ =pn) andthus from Table1 we haveMPV = Expected p Values,” TheAmerican Statistician ,53,326– 331. Schatzoff,M. (1966),“ SensitivityComparisons Among Tests ofthe General .0355.Note that we haveused the same ¢i valuesfor all n. We consider10,000 replications. LinearHypotheses,” Journalof theAmerican Statistical Association , 61, 415– 435. TheMPV valuesin Table3 havesimilar magnitude as the EPV Schervish,M. J.(1996),“ P Values: WhatThey Are andWhat They Are Not,” valuesof Sackrowitzand Samuel-Cahn (1999, table 4) even at TheAmerican Statistician ,50,203– 206.

206 General