Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 niiultrsi n w-a rtrewyAOAfrteeatts weepsil)adfrthe for and possible) (where for test strategies exact permutation methods Resampling appropriate the tests; Randomization provide for Non-parametric; also ANOVA be we three-way Keywords it reference, whether or test, For residuals. two-way permutation of test. any permutation any the using for approximate under test essential in units wisdom an approximate exchangeable is conventional provide, terms correct or the alternative we of challenging individual or test choice guideline power, the the test in exact of that that with permutations, exact loss demonstrated showed an accordance restricted the also of in results cases, simulations terms than hypothesis, several including Our Simulation in power null ANOVA, In approach. procedures, terms. greater data). this two-way other raw using had interaction in than of of generally of more permutation strategies simulations model suffered as tests permutation Carlo reduced particular, (such Monte and different methods a of permutation hierarchies under of results approximate where nested residuals provide power strategy, The we of permutation models, and test. (ii) exact permutation mixed particular accuracy and an any constructing design; and level ANOVA for for random any the guideline used factors, in a be compare random provide term should involving we to individual designs strategy (i) form any complex two-fold: permutation some for for is which of possible, (ANOVA) especially paper just permutation unclear, analysis-of-variance this is hierarchies, units, of in It nested of purpose terms these. groups or individual of whole models combination of of mixed some permutation tests or permutations, residuals for restricted of possible include often These are designs. strategies permutation Several ukad e eln.Tl:6--7-59x55;Fx 493371;Emi:[email protected] E-mail: 64-9-373-7018; Fax: 5052; x 64-9-373-7599 Tel.: Zealand. New Auckland, 03 o.7() p 85–113 pp. 73(2), Simulation Vol. and 2003, Computation Statistical of Journal SN09-65pit SN16-13online 1563-5163 ISSN print; 0094-9655 ISSN errors distributed univariate of analysis normally the In of 1994). McArdle, and assumption Gaston 1961; Taylor, its relationship: variance of violation to is Scheffe ANOVA 1947; parametric robust (Cochran, Traditional psycho- quite sciences. biological, environmental the generally in and extensively ecological used tool medical, statistical logical, a is (ANOVA) variance of Analysis INTRODUCTION 1 O:10.1080 DOI: s oee,uraoal o ayknso aa( data of kinds many for unreasonable however, is, a etefrRsac nEooia mat fCatlCte,Mrn clg aoaois A11, Laboratories, Ecology Marine Cities, Coastal of Impacts Ecological on Research for Centre orsodn uhr rsn drs:Dprmn fSaitc,Uiest fAcln,PiaeBg92019, Bag Private Auckland, of University Statistics, of Department address: Present author. Corresponding * y e. 137762;Fx 137189;Emi:[email protected] E-mail: 31-31741-8094; Fax: 31-31747-6929; Tel.: EMTTO ET O MULTI-FACTORIAL FOR TESTS PERMUTATION nvriyo yny S,20,Australia; 2006, NSW, Sydney, of University NV;Eprmna ein ie n admfcos irrhcldsgs ie models; Mixed designs; Hierarchical factors; random and Fixed design; Experimental ANOVA; : = 0094965021000015558 eerhCnr,Bx10 70A,Wgnne,TeNetherlands The Wageningen, AC, 6700 100, Box Centre, Research AT .ANDERSON J. MARTI Rcie eray20;I nlfr 3Ags 2002) August 13 form final In 2001; February 1 (Received ´ 99 ndcradCcrn 97.Teasmto fnormality of assumption The 1967). Cochran, and Snedecor 1959; , NLSSO VARIANCE OF ANALYSIS a, n AOJ .TRBRAAK TER F. J. CAJO and * # b imti,Wgnne nvriyand University Wageningen Biometris, 03Tyo rni Ltd Francis & Taylor 2003 e.g. , clgclvralssoigamean– a showing variables ecological , b, y Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 NV ein nrdc w motn diin o taeis otepsiiiisavail- possibilities the However, to for designs. ANOVA strategies) obtained in (or results additions terms the important individual that two for and (Anderson introduce satisfied well designs be test ANOVA equally (Anderson simply apply exact circumstances could regression conceptually one multiple of variables, the set predictor to categorical widest closest and the the (Freedman comes model in 2001). reduced also Robinson, best a and and under the 1999) residuals performed of Legendre, Robinson, and permutation 1983) Anderson that 1999; Legendre, found Lane, regres- and was linear (Anderson multiple exists It in test term 2001). exact individual no an which of valid for test a sion, permutation approximate that an contended for suggested strategies permutations. (1995) (1998) using Edgington Manly done while be and cannot data, Gonzalez interaction and raw permutation for suggested (1997) the test (1992) Manly of to Braak model, example, ter permutation full For permutation, unrestricted a tests. by an under particular term interaction as residuals for an used of soon concern- of be opinion as test may of However, a divergence that a obtain observations. methods is permutation there of factor, appropriate sets one ing than multivariate more or includes design univariate experimental either for 2000), uaindsrbto o n-a NV ( ANOVA one-way for distribution permutation for mutation programs (Anderson, these for design. of NPMANOVA Motivation experimental facilities ANOVA the matrix. program complex extend distance any to computer in a desire tests our on the from based arose of analysis research multivariate this S and non-parametric and data for Braak important 2001) (ter ecological an CANOCO is of program testing ordination Permutation computer 2001). the Anderson, of 2001; feature Anderson, and McArdle 1999; aad 90 uetadShlz 96 Mielke 1976; al. Schultz, et and Hubert 1970; Valand, 1937; Pitman, 1935; (Fisher, achieved are BRAAK tests TER F. J. exact C. AND yet ANDERSON J. M. distributed, Scheffe normally 1993). be Clayton, and Breslow 1989; ( Nelder, explicitly and specified (GLMMs), be McCullagh models may mixed 1972; structure linear transformation error generalised suitable or non-normal a (GLMs) a finding models where by linear data generalised non-normal using of by problem or the avoid often can one data, 86 rpit.Tu,teehv enmn et rpsdfrtecmaio of comparison inap- the be for generally proposed will tests 1992) many Liang, been and have Zeger there 1986; Thus, Zeger, and propriate. Liang see analysis, tudinal A 1968). Hope, re-orderings, such of obser- subset the of random is (permutations) large hypothesis re-orderings a null possible the all (or under for by vations statistic value com- test its test exact accessible relevant calculating an more a by for of and constructed general, distribution faster In reference much 1997). Manly, the of 1995; permutation, advent Edgington, the 1992; (Crowley, with power years puter recent in attention renewed the fmliait aata eyo emtto fteosrainvcos( vectors observation the of permutation on rely that data multivariate of ( structure error of kind same trying the responses, multiple with are ( them there of fulfill where all situations model to many in to difficult Also, 1993). more Field, that and even permutation Johnson are under value. assumptions obtained observed butional statistic the than the extreme of more values or the to of equal proportion are the as calculated ic NV ihfie atr ssml pca aeo utpergeso,btwith but regression, multiple of case special a simply is factors fixed with ANOVA Since appropriate investigate to done been has work theoretical and empirical extensive Some hr sgnrlareetcnenn h prpit rcdr oaheea xc per- exact an achieve to procedure appropriate the concerning agreement general is There natraieapoc st s emtto et,weeerr r o sue to assumed not are errors where tests, permutation use to is approach alternative An emtto et r seilyueu n eeatfrmliait nlss hr distri- where analysis, multivariate for relevant and useful especially are tests Permutation priori a 92 lre 93 digo,19;Pla n Orlo and Pillar 1995; Edgington, 1993; Clarke, 1992; , ´ 99.B ‘xc’,w enta h yeIerro h eti xcl qa to equal exactly is test the of error I type the that mean we ‘‘exact’’, By 1959). , hsnsgicnelvlfrtets.Teueo emtto et a received has tests permutation of use The test. the for level significance chosen e.g. tal. et digo,19;Mny 97 Good, 1997; Manly, 1995; Edgington, , 96 Smith 1976; , e.g. ´ i 96 oe n Krzanowski, and Gower 1996; ci, e.g. ada 91 lo,1974; Olson, 1971; Mardia, , ˇ yuigGM si longi- in as GLMs using by , e.g. iae,19)frcanonical for 1998) milauer, edradWedderburn, and Nelder , tal. et 90 Excoffier 1990; , e.g. P priori a vlei then is -value atland Mantel , groups Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 o rvd n atclrdtiso hshr o atclrcsso unbalance. do of cases but particular designs, for unbalanced here for this tests of details permutation particular for any followed provide be not effects, should fixed Searle below consult for guideline ( designs) restrictions squares unbalanced mean summation for expected or usual model of ANOVA the formulations any alternative on in For rely terms for also be squares they we i.e. mean whether simulations) expected independent, for are considering effects (and In mean nested. we or designs, crossed orthogonal some By introduce 1964b). in and (1964a; terms designs of individual class for the test delimit permutation to exact need definitions. an first We ANOVA. constructing multi-factorial for a guideline a provide We TEST EXACT AN CONSTRUCTING 2 vectors. general, in response hold, multivariate Although will on models. results tests effects the permutation random designs, ANOVA analogous or univariate the mixed to being for here and as well attention hierarchies as our nested knowledge, restrict include our permu- we to to for date, ones simulations to extensive only provided recommendations most the designs ANOVA and the complex ANOVA. statements are in two-way These general tests of made. tation important models be them, several could ANOVA from in multi-way that, (including terms for so strategies individual chosen permutation for were possible possible) These several where of test, power exact and the knowledge. accuracy our level and to the literature, models the compare mixed in effects, form done. succinct random be a to for such are in possibility random, exists tests the currently or permutation hierarchies, including on fixed valid nested guideline, are light how general factors determines sheds whether factors, such guideline including other No This survey, with any design. or crossed in ANOVA experiment or possible, complex the nested where a of test, in design permutation the factor exact how an any of for construction situation the given for used guideline be a should give test-statistic we What (f) occur permutations? approximate? using to are tested restricted which test? be be and the terms the exact interaction permutations for of are may test should How tests a When (d) Which in factors? (c) (e) factor other nested hierarchy? a of indi- a of levels permuted: Should levels of within be as (a) should factor such units including: units, ranked Which larger issues, (b) higher some residuals? important or of units several form observation some clarify vidual or permuted, to be strategy. necessary data appropriate raw is most it or powerful design, ANOVA most the provide will residuals or data raw ihtevrosapoiaetssi em fterpwrfrdfeetknso designs. and of permutation combination kinds restricted which of design, different ANOVA use complex for (ii) a units, power in permutable their term of these choice particular of of (i) any compare both terms of for tests or unclear, in such other is tests how it or just Thus, approximate one however, using various unknown, constructed entirely the be virtually designs, with ANOVA can is many factor It In individual strategies. factors. an additional random for with test designs exact in an pertinent be may strategy second nywti eeso te atr)ad(i emtto fuisohrta h observations the than other units of permutation (ii) and factors) other ( ( of permutations levels restricted within (i) only regression: multiple for able e.g. elmtordsuso oeu-elctdotooa NV ein,floigNelder following designs, ANOVA orthogonal equi-replicated to discussion our limit We to done simulations Carlo Monte empirical of sets specific from results give we Second, First, issues. these of clarification a such precisely provide to is paper this of purpose The complex a in term any for distribution permutation appropriate an construct to order In httesmidvda ramn fet cosallvl fafie atreul zero. equals factor fixed a of levels all across effects treatment individual sum the that , emtn h nt nue yanse atri h eto ihrrne atr.The factor). ranked higher a of test the in factor nested a by induced units the permuting , UT-ATRA NLSSO AINE87 VARIANCE OF ANALYSIS MULTI-FACTORIAL tal. et 19) ecnie httesm general same the that consider We (1992). i.e. i.e. ihu umto restrictions summation without , loigpruain ooccur to permutations allowing , = r(i)pruainof permutation (iii) or Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 emttost cu ihntelvl ftrso ihrsalrodro ftesm order same the of or order the smaller either permuting tested of by being terms restricting achieved term of and the is levels F-ratio as the model the ANOVA within of occur an mean-square to denominator in permutations the term by any identified for units exchangeable test hypothesis. permutation null exact the An under distribution the same in the denominator have the units as exchangeable appears The mean-square whose 4 term identifies A F interaction 1c). the Fig. then units, levels, exchangeable three has factor second 8M .ADRO N .J .TRBRAAK TER F. J. C. AND ANDERSON J. M. 88 xhnebeudrtenl yohss(i.1) hr h eoiao ensur of square are mean mean (observations) denominator the units the is individual Where square that 1a). indicates (Fig. mean this hypothesis residual an null test, the the a Where under for 1. exchangeable denominator Figure the Consider in term. square particular any for ito h nt nue ytense em o xml,i iuel hr r eight are there lb Figure in example, For term. nested the by induced units the of sist yrsrcigtepruain ftoeuiswti eeso te factors. other of levels within units addressed con- is those by second and addressed of The is tested, permutations test. these being the permutation of the restricting term first for by The the units model. exchangeable ANOVA in appropriate linear variability invariance: the the to the sidering in equi-replicated contribute terms guarantee of may non-zero to context other that the aspects (ii) variation In different interest. of two of components per- factor identify (i) under the we constant on kept designs test ANOVA be the should orthogonal isolate tested to being idea is order not the data model in encapsulates the the mutation It in of exact. parameters is likelihood unknown test the all the hypothesis, that consequently null and the permutations, under these under that, invariant ensures guideline The units. able ( interaction that by identified cells denomi- of the blocks Where the replicates. two of of sist consisting unit an each B, of term nator nested the by induced units is it which within an term is of the that square of term mean order A denominator the etc. the main than order, in following: more second identified one the of are order mean is units’’ an we ‘‘Exchangeable term has nested. interaction term, term two-way a another a of in order, ‘‘order’’ nested first the of not By are does test. effects a this for that units’’ ( ‘‘exchangeable Note variances (‘‘i.i.d.’’). assumption error distributed of the homogeneity identically to of and tantamount assumption the is independent ( avoid exchangeability are design of errors experimental assumption that in The through units 1966). ensured to Kempthorne, be Scheffe treatments can 1955; Exchangeability of Kempthorne, hypothesis. allocation null the random under the units relevant of ability ntetr en etd h ai scntutdb eeec oteepce ensquares mean expected the to reference by variability constructed to is ratio contribute The that tested. variation being of term components the the in identifying by units exchangeable error similar of statement the of virtue with exchangeability by associated ( validity that errors hypothesis its consider the null We gains the interaction), distribution. tests under an same permutation distributions of test the for the have units terms in treatments of other effects the different given main in conditional: as is units (such hypothesis hypo- null distribution. model same null the the the designs, the have complex generally, in treatments more different More of in effect. case units the treatment with In associated no errors is the that there is that thesis is treatments, call generally rtoo h eto nte emi h oe dnie h xhnebeuisfrta test. that for units exchangeable the identifies model the in term another of test the of -ratio epoietefloiggnrlguideline: general following the provide We h osrcino the of construction The exchange- the of permutations restricted on based test permutation a yields guideline The the by meant is what and term a of ‘‘order’’ the for definitions provide we Next, exchange- assume still they normality, of assumption the avoid tests permutation Although h ulhptei o eto n atclrfco,woectgrcllvl emay we levels categorical whose factor, particular any of test a for hypothesis null The F rtoi h ensur f o xml,anse em hnteecagal nt con- units exchangeable the then term, nested a example, for of, square mean the is -ratio F rtoi h ensur f o xml,a neato em hnteuiscon- units the then term, interaction an example, for of, square mean the is -ratio . F ´ 99 rms easmdfrosrainlsuis( studies observational for assumed be must or 1959) , rtoisl rvdstencsayifraincnenn the concerning information necessary the provides itself -ratio e.g. ephre 1952). Kempthorne, , e.g. foefco a orlvl n a and levels four has factor one if , e.g. ok 97 ae,1996). Hayes, 1987; Boik, , 3 ¼ 2clswihaethe are which cells 12 e.g. ihr 1935; Fisher, , ð F 2 -ratio e.g. 4 Þ , Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 eoiao nodrt slt h opnn fvraino neetfrtets using test the for interest the of of variation square of the mean in component expected present the are the isolate tested in to being the appear not order must are in they that denominator square, variation mean of random). components expected is numerator’s If B test factor terms. (where b) and individual design A model; ANOVA model factor of one-way mixed within a crossed nested two-way in B a A factor in random of factor, with test fixed design, a) a nested tests: two-way A, a exact factor in for of factor strategies test higher-ranked c) permutation the of A, Diagram factor of 1 FIGURE ainwti eeso hs atr visti yesrn htteszso hi fet in effects without ANOVA their two-way of a sizes in the example, that For ensuring permutation. by under this permu- constant avoids Restricted remain factors test. model these under the of term levels the with’’) within (‘‘mixed to tation with units confounded exchangeable be the would this, non-zero, by and, test. tested permutation being the term in the used in be variability the to contribute oetrsi h oe a o otiuet aiblt ntetr fitrs u,if but, interest of term the in variability to contribute not may model the in terms Some F rto h eoiao ensur hsietfistecmoet fvrainthat variation of components the identifies thus square mean denominator The -ratio. UT-ATRA NLSSO AINE89 VARIANCE OF ANALYSIS MULTI-FACTORIAL Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 npriua,w ierslso miia iuain htcmaetevrospossible various power models. the empirical three-way compare greatest to the that provides extend two-way simulations them then tests. in empirical of we for which of terms which demonstrating results individual strategies, test, give permutation princi- exact of we the an demonstrate particular, tests These constructing In models. for in mixed available involved crossed and ples possible) fixed crossed where nested, for ANOVA exact, 1999). Legendre, and are and factors ximate (Anderson nuisance where ANOVA, outliers in for of contain outliers (1996) context not of the Cade do effects in thus and the occur and concerned Kennedy cannot categorical which and by regression, data, (Anderson raised multiple in raw results for problems of used variables comparable the permutation be highly that of give method also add the to of also can We expected permutation residuals 1999). be concerning full-model would Legendre, attention and although Robinson, our and situations, restrict residuals, (Anderson these generally reduced-model test here exact to we conceptually residuals reason, a under to Permutation this data closest (2001). For Robinson raw the 2001). and comes of Anderson model and reduced permutation (1999) the unrestricted Legendre and or Anderson see 1992) Braak, 1998). (ter Manly, Lane, and and model Freedman Gonzalez 1981; full 1997; White, (Manly, and the test: (Still test. approximate under model reasonable an a reduced and 1983), for provide a statistic approaches to under test different sufficient residuals the three not of for essentially clearly permutation values are in is possible unique there This 6 (B) give designs, value. only factor would complex leaves observed which For nested this the of is (A), a 3 which factor of A, of order) levels lower of one two (but test to ranked only the permutations higher are for a possible permutations there of few if levels too two example, of For being each test. there reasonable in a results test obtain units other exchangeable using and approaches restrictions alternative (For possibility. the only 2001). permutations: the Pesarin, possible exact is see alternative an statistics, data no construct observed leaves to interaction the effects possible of the with not main case using is the the interaction it of in an 1990), levels example, (Welch, for for ( cases test occurs, special appro- possible permutation an some This not but in used. is exists, Except be test test terms. may exact exact no test then an permutation permutations), for ximate possible required no leave strategy required the restrictions where situations In TESTS APPROXIMATE 3 consider to us leads This alternatives. per- detecting possible for enough have power tests. not approximate reasonable may enable or exist, see to always not mutations values, BRAAK may however, TER repeated F. test, J. permutation fixed C. exact AND several ANDERSON J. take M. context variables the (1982). predictor in Maritz permutation and where restricted Brown regression, of discussion multiple a For of (1995). Edgington of see levels psychology, within occur test. to exact restricted an be for should factor factor one other of the test a for permutations interaction, 90 nwa olw,w ecieadcmaetepsil emtto taeis(appro- strategies permutation possible the compare and describe we follows, what In regression, multiple of context general more the in approaches these of comparison a For of combination the when occurs feasible not is test exact an where situation Another An model. ANOVA an in term particular any for unique be will test permutation exact The in designs ANOVA complex of kinds many for permutation restricted of discussion a For F saitc srsrcigpruain within permutations restricting as -statistic, F rtoobtained -ratio i.e. hr the where , Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 in ihu etito.Ti ie aiblt mn l em ntemdl including model, the in ( terms A all factor among to variability test. mixes due approximate This an variability yield restriction. will without residuals tions these level Permuting appropriate the value. of observed mean each the subtracting from by A A of of effect ‘‘removing’’ the and estimating to hl eoesm fsurs ensursand squares mean we SS squares, Throughout, as normal. B) of (necessarily) factor are sums they that denote not shall but (i.i.d.), distributed identically and ihnthe within where model: linear following the with design ANOVA (hierarchical) nested a Consider DESIGN NESTED 4 by ul hl h usrp ‘’salidct h oa.Frteaoemdl eto atrBis B factor of statistic test a model, the above the For by total. the provided indicate ‘‘T’’shall subscript the while dual, oe taeiso emtto odtc aiblt u ofco .Dt eesimulated were Data B. factor to with due model variability above detect the to to permutation according of strategies posed 2001). Robinson, and Anderson 1998; Manly, and (Gonzalez i)rsrcigpruain ihnlvl fAmasta SS that means A of levels within permutations restricting (ii) F soitdwt observation with associated aito nybtenSS between only variation hs r etitdt cu ihnec ftelvl fA hsmasta hte or whether that strategy means This This B. factor A. to due of variability levels significant guideline. for the different the test not with of can we is consistent each effect, A is an within within has pre- occur A B the factor of ( to given not effect observations phrased: restricted the of be are can assumption, permutations B these no the factor make Thus, of we zero. effect which the from about of A, test of exact an sence for hypothesis B Factor, null Nested The the of Test 4.1 rrno.Fco ,bignse,wl,frpeetproe,awy ecniee random. fixed considered is be has A always B whether purposes, apply factor present will although for discussion will, same that nested, the Note being model B, this Factor for random. but factor, or fixed a as A sider oee,a hs‘mxn’ fSS of ‘‘mixing’’ this as However, emttos n ii SS (iii) and permutations, a e at set was r h ratio the rmanra itiuinwt enzr n rda nrae nsadr deviation. standard in increases gradual and zero mean with that distribution Note normal a from 33 B ijk A ð A natraieapoiaets ffco spoie ysml emtn l observa- all permuting simply by provided is B factor of test approximate alternative An iuain eedn odmntaetetp ro n eaiepwro h bv pro- above the of power relative and error I type the demonstrate to done were Simulations hspoie neatts eas i SS (i) because test exact an provides This o napoiaets,oecncluaeadprueterdcdmdlresiduals reduced-model the permute and calculate can one test, approximate an For : 3 a ¼ ¼ Þ ; j ð and i MS y Þ m notemdl o h emtto et easm nyta the that only assume we test, permutation the For model. the into , 100 ijk steukonpplto mean, population unknown the is F b A epciey n h ubro elctosprA obnto by combination AB per replications of number the and respectively, , B g b n = i h diieefc flvl fterno atrBwr hsnrandomly chosen were B factor random the of levels of effect additive The . htetetlvlo atrA yblsdby symbolised A, factor of level treatment th y MS ¼ ¼ ¼f i :: where , MS ifrn ausof values different 5 B B 2 ; ; MS ( e.g. B 5 ; = MS B r10 or y ephre 92 Scheffe 1952; Kempthorne, and i :: R UT-ATRA NLSSO AINE91 VARIANCE OF ANALYSIS MULTI-FACTORIAL stema fteosrain nlevel in observations the of mean the is naeae suafce,gvn esnbeapoiaetest approximate reasonable a giving unaffected, is average, on , B T g F y n SS and hr h feto atrAwsnon-zero: was A factor of effect the where , F ijk ¼ B B i.e. epciey lo h usrp ‘’ hl niaeteresi- the indicate ‘‘R’’ shall subscript the Also, respectively. , h ubro eesi atr n ilb designated be will B and A factors in levels of number The . y ¼ SS ijk b ,SS A MS R ¼ ees titoue oa of total a introduces it levels, R ne emtto migso aho SS of each on impinges permutation under a þ hc seatfrtets of test the for exact is which , m B A B ¼ SS ,wihi o edcntn ne permutation. under constant held not is which ), = ð þ A MS 4 Þ B A ; j ð b i þ i B R Þ þ ð ¼ eecoe ihnec ee ffco A. factor of level each within chosen were SS A B n eto atrAi rvddby provided is A factor of test a and Þ n h apesz ihnec el( cell each within size sample the and 5 T j ð A ð A i ´ Þ hs emtto ie (exchanges) mixes permutation Thus, . 99 Winer 1959; , ty osatars l permutations, all across constant stays Þ steefc fthe of effect the is j F ð y i Þ rto o atclrtr syfor (say term particular a for -ratios r oears h eeso ,but B, of levels the across done are ) þ e ijk A i and , A ean osatars all across constant remains i tal. et a ffco .Ti amounts This A. factor of e ijk F B steukonerror unknown the is b 1991). , j ¼ hlvlo atrB factor of level th fet,dntdby denoted effects, e saeindependent are ’s MS A i ¼f B = MS B n 33 econ- We . n SS and : R 3 . ; 33 : n 3 R ) ; , Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 2M .ADRO N .J .TRBRAAK TER F. J. C. AND ANDERSON J. M. of total a is, That 92 ewe SS between elctswti ahlvlo r ettgte saui n hs nt r permuted are units these and unit 1b). (Fig. a test as exact together the kept for are replicates, A individual B of permuting levels of than Rather across level B. each of levels within as test replicates the for units exchangeable the ahpruainlapoc sn 9 admpruain.Tets ttsi sdi all in used statistic test a set, The data permutations. random the 999 was simulations using approach radically permutational whose were each that distribution data (1) simulate to exponential used random an was exponentiating (iv) distribution ( latter of or non-normal The consisted distribution, the cubed. which then on normal were distribution distribution, standard values uniform lognormal a a a from the (ii) (iii) values distribution, in 10), normal investigated (1, standard be interval a to (i) distributions: designs ANOVA different this, in in for issues used distribution be fundamental normal could the most distributions used instance. the we first error allowed simulations, other in it Although effects A. as random factor of of levels level modeling each to peculiar and the of uigwoelvl fBa nt en htSS that means units as B of levels whole muting n ii SS (iii) and a ital dnia oe hnerr eenra Fg a a.I,weestetradi- the whereas I), Tab. 2a, (Fig. using normal done procedures were test errors all were when that tional showed power methods also identical simulations virtually These different 0.05. had of of level significance nominal hypothesis) the the null (as power the the of of tests. comparisons signed-ranks pair-wise rejections Wilcoxon’s simulations, of subsequent in number and this In parison. ul prahdtepwro h xc etwt nrae in increases with had test A) exact resi- the of of Permutation of levels I). power Tab. within 2b, the data (Fig. approached residuals (permuting of duals permutation test by followed exact power, the greatest the however, errors, cubed exponential rmzr.Teepce ensur o atrAcnan opnn fvrainattri- variation of component Therefore, a B. different contains not factor A is nested A factor varia- of for the the effect given square the to phrased: assumption, mean be butable no expected can make A The we factor which zero. of about from effect B, the of of levels test across exact bility an A for Factor, hypothesis Ranked null Higher The the of Test 4.2 I). (Tab. test exact the or residuals of normal-theory the and restriction without hsisei h aefrpruaintssa o h omlter etadwntbe won’t and test normal-theory the for as tests permutation for (Winer deter- here same a further and the one, discussed normal-theory important is the an is for issue situations case This such the a in be pool’’ constitutes would what to of not as mination or A, pool factor ‘‘to of decision The test test. one-way ( a residual doing the and with randomly) B factor a pooling with of significant not was hspoie neatts eas i SS (i) because test exact an provides This o ahseai,attlo 00st fsmltddt eepoue n etdwith tested and produced were data simulated of sets 1000 of total a scenario, each For four of each from chosen were errors random simulations, subsequent all for and this For yeIerr o l etpoeue i o ifrsgicnl rmoeaohro from or another one from significantly differ not did procedures test all for errors I Type oeta fi a endtrie rmapeiu etta aiblt u ofco B factor to due variability that test previous a from determined been had it if that Note F a ts a otpwru ntecs fuiomerr Tb ) ihlgomlor lognormal With I). (Tab. errors uniform of case the in powerful most was -test ¼ P eeso atrA hsi h enn fanse fet t eesaespecific are levels its effect: nested a of meaning the is This A. factor of levels 4 A vlefrtenormal-theory the for -value T e.g. n SS and ¼ al,1997). Manly, , SS ab R F ¼ B rtowt eoiao prpit o h ein o ahsimulated each For design. the for appropriate denominator a with -ratio þ hc seatfrtets of test the for exact is which , 0dfeetefcswr hsnrnol,wt v sindt each to assigned five with randomly, chosen were effects different 20 SS B P þ vleta a ufiinl ag,oewudhv h option the have would one large, sufficiently was that -value P tal. et SS vleta s‘sfcetylre’myntb straightforward. be not may large’’ ‘‘sufficiently is that -value A hs emtto ie ecags aito only variation (exchanges) mixes permutation Thus, . 1991). , i.e. F F goigfco n emtn l observations all permuting and B factor ignoring , ts,uigtetbe,wsas bandfrcom- for obtained also was tables, the using -test, ts a infiatyls oe hnpermutation than power less significantly had -test T F A ty osatars l emttos i)per- (ii) permutations, all across constant stays ¼ R MS ean osatars l permutations, all across constant remains F A A = MS ¼ MS B hs eoiao indicates denominator whose A n = emtto frwdata raw of Permutation . MS B . Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 m n IUE2Pwrcre o emtto et n h normal-theory the and tests permutation for curves Power 2 FIGURE u epoie ypruaino a data. raw of permutation by provided be residuals fteosrain nlevel in observations the of eoe h feto h etdfco,B fay neednl fA eiul ne h full the under Residuals A. of independently any, if be B, would factor, model nested the of effect the removes þ¼ ¼ ¼ ¼ ¼ emtto frsdasudrardcdmdlfrats fAmgtb oeuigthe using done be might A of test a for model reduced a under residuals of Permutation a pruaino a aaas data raw of (permutation Yab pruaino residuals), of (permutation R pruaino a data), raw of (permutation Y ()(emtto frwdt ihnlvl fA), of levels within data raw of (permutation Y(A) normal-theory r ijk ¼ y F ijk r -test. ijk ¼ y ij y : ijk þ UT-ATRA NLSSO AINE93 VARIANCE OF ANALYSIS MULTI-FACTORIAL i y ffco and A factor of i :: y ij ab : n n ih loiaieta napoiaets could test approximate an that imagine also might one and y units), ... where , y y ... ij : steetmtdoealma.Ti essentially This mean. overall estimated the is stema o the for mean the is F ts o w-a etddesign: nested two-way a for -test ij hcell, th y i :: stemean the is Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 4M .ADRO N .J .TRBRAAK TER F. J. C. AND ANDERSON J. M. 94

TABLE I Summary of results regarding power of permutation tests, based on pair-wise Wilcoxon’s signed-ranks tests for n ¼ 10 sets of 1000 simulations used to produce power curves for each of four different error distributions. The permutation methods are ordered according to the greatest number of rejections (greatest overall power) in simulations: ‘‘ ¼ ’’ indicates not significantly different (P > 0.05), while inequalities indicate significant differences (P < 0.05). Symbols in bold indicate the exact permutation test procedure in each case, where possible.

e Nð0; 1Þ e Uniformð1; 10Þ e Lognormal expfNð0; 1Þg e expð1Þ3

Nested Test of B: R ¼ F ¼ Y(A) ¼ YF> R ¼Y(A) > Y Y(A) ¼ R > F > Y Y(A) > R >Y ¼ F Test of A: F ¼ Yab ¼ YF> Y ¼ Yab F ¼ Yab ¼ YY*> Yab > F Crossed, Fixed Test of AB: F > R ¼YF> R ¼YR> F > YR>Y > F Test of A: F ¼ R ¼Y > Y(B)F> R ¼Y > Y(B)R> F > Y > Y(B)R>Y ¼ F ¼ Y(B) Crossed, Mixed, No Interaction Test of A: F > R ¼ Rab ¼ Y ¼ Y(B)ab > Y(B) F > R >Y ¼ Y(B)ab ¼ Rab > Y(B) R >Y(B)ab > Rab ¼ F ¼ Y > Y(B) R >Y(B)ab ¼ Y > Rab > F > Y(B) (fixed factor) Crossed, Mixed, Interaction Present Test of A: F > R ¼Y ¼ Y(B)ab ¼ Rab > Y(B) F ¼ Y ¼ R ¼Y(B)ab > Rab > Y(B) Y ¼ F ¼ R >Y(B)ab ¼ Rab > Y(B) Y* > R* > Y(B)ab > Rab > F > Y(B) (fixed factor)

*Type I error was inflated for these tests – see Tables II and III. Y ¼ permutation of raw data; R ¼ permutation of residuals; Y(A) ¼ permutation of raw data restricted within levels of A; Y(B) ¼ permutation of raw data restricted within levels of B; Ya b ¼ permutation of raw data, as ab units (induced by nested factor B); Rab ¼ permutation of residuals, as ab units; Y(B)ab ¼ permutation of raw data, as ab units, restricted within levels of B; F ¼ normal-theory F-test (tabled values). Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 with h eue rudrtefl oe)adtenormal-theory the and model) full under residuals the of under permutation data, or raw reduced of permutation the (unrestricted tests approximate the of agsof ranges iuain,w hl use shall we simulations, suws o h eto ihrrne atri etddesign. tests nested a approximate normal-theory in these the namely, factor of error: ranked any I type higher of had a use residuals, of the of test Thus, form the II). any for (Tab. of unwise error permutation is levels approximate I the and when type of data all inflated Furthermore, variance, raw large significantly II). of with (Tab. permutation distribution a 0.05 including from methods, drawn of were simulated level factor nested were normal-theory significance the data the of the when errors than However, cubed lower II). exponential (Tab. significantly skewed level highly significance the nominal with the to close error simulations. in used tions a rlgomlerr Tb ) n o nfr ros the errors, nor- uniform for for tests two and these I), of (Tab. power the errors in lognormal difference or significant no mal was there had normal-theory while test errors, plots the cubed permutation power than exact the power the in that greater evidenced indicated significantly methods tests two pair-wise Wilcoxon’s these However, 2c,d). between power (Fig. in difference visual striking ftense atr ic oeo h prxmt ehd fpruainwl o hsis This do. will permutation is of levels units) methods of approximate exchangeable number the of the of increasing none number by since remedied small factor, be a nested only the only can of and to drawback serious (due unique a a permutations yielding potentially permutations at possible possible be 10 of effect only must gives number the A for there of test levels levels, permutation 2 for exact of value 2 each an has in obtain B to A factor order nested permutation where in the B example, whose of factor A For nested power of the test. the of sacrifice levels normal-theory severely 4 for the may least error replication for such I as of type test, lack approximate maintain the the to as of factors, trusted None be issue. can that an alternative is situation. an this non-normality provides where tests design permutation hierarchical nested a atrAi nlgu to analogous is A factor eeso eecoe admyfo omldsrbto ihma eoadstandard and zero mean large) with (and distribution constant separate normal A, kept a factor was from of level randomly B each chosen For to deviation were increased. due gradually B was variability of A time factor levels of this effect fixed but the 4.1, while Section in described na netgto fpwr hsi h aueo NV:i soverparameterised is it ANOVA: of nature the is This power. of ( investigation an in nee nqeyb h esr fefc size: effect of measure the by uniquely indexed e.g. iuain eedn ocmaeterltv yeIerradpwro h xc et each test, exact the of power and error I type relative the compare to done were Simulations ncmaioso oe,w nyicue hs ehd htddntsfe rminflated from suffer not did that methods those included only we power, of comparisons In I type had tests all errors, lognormal or uniform normal, with simulated were data When eerhr ol ewl-die oices elcto ftelvl fnse random nested of levels the of replication increase to well-advised be would Researchers in factor ranked higher a of test any for used be should test permutation exact the Clearly, hr r nifiienme fpsil austefie effects fixed the values possible of number infinite an are There Scheffe , A ¼ F P ¼ y hl eeso ol ie3 osbepruain.Telc fasufficient a of lack The permutations. possible 35 give would B of levels 4 while , A ´ i a P 20 ¼ 99.Hwvr nivsiaino oe fragvnsml ie a be can size) sample given a (for power of investigation an However, 1959). , (or vlecnece infiac ee f00.I te od,3lvl fthe of levels 3 words, other In 0.05. of level significance a exceed can -value 1 : A 0. i y = AB a ( e.g. or s Winer , s UT-ATRA NLSSO AINE95 VARIANCE OF ANALYSIS MULTI-FACTORIAL B y B r eurdt banpwrcre o ifrn ro distribu- error different for curves power obtain to required are ) A o admfco .Ti a h osqec htdifferent that consequence the has This B. factor random a for ¼ s tal. et e f A f ¼ A 91.Frsmlct,as simplicity, For 1991). , samaueo fetsz.Nt that Note size. effect of measure a as q F ts n h xc emtto et hr a o a not was There test. permutation exact the and -test P ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi i a ¼ 1 ð A s i e F A ts o h iuto ihexponential with situation the for -test Þ 2 = a F F ts.Dt eesmltdas simulated were Data -test. ts a otpowerful. most was -test s e ð A sacntn o n e of set any for constant a is i ; F i ts a yeIerror I type had -test ¼ 1 ; ... y A ; a o fixed a for Þ a take may Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 6M .ADRO N .J .TRBRAAK TER F. J. C. AND ANDERSON J. M. 96 hr n r xdfactors, fixed are B and A where model: linear following the with design ANOVA two-way effects fixed a Consider EFFECTS FIXED DESIGN, CROSSED 5 denominator the with associated freedom of degrees power the the greater ANOVA: of increasing square parametric by mean with achieved encountered be situation can the to analogous somewhat eoe o h emtto etw sueta the that assume we test permutation the for before, cinefc fthe of effect action 02. .4 .3 .3 .4 0.039 0.050 0.053 0.044 0.045 0.056 0.057 0.060 0.031* 0.047 0.062 0.036 0.050 0.053 0.047 0.034* 0.052 0.045 0.047 0.062 0.060 0.053 0.052 0.036 0.045 0.048 0.034* 0.055 0.045 0.052 0.061 0.040 0.045 0.051 0.047 0.053 0.055 0.052 0.054 0.032* 0.045 20.0 0.052 0.040 20.0 0.062 0.059 20.0 0.036 0.050 10 0.050 1.0 0.055 5 0.044 1.0 2 0.042 1.0 0.051 5 0.058 10 0.046 0.055 5 5 0.048 0.056 5 2 20.0 4 0.058 5 20.0 0.049 4 5 20.0 0.050 4 10 5 1.0 4 5 1.0 4 2 1.0 5 4 10 5 5 5 2 4 5 4 5 4 5 4 4 4 bn ab 02. .5 .5 .5 .5 0.051 0.042 0.051 0.052 0.053 0.045 0.038 0.058 0.048 0.051 0.047 0.050 0.059 0.043 0.043 0.050 0.045 0.057 0.054 0.040 0.045 0.050 0.048 0.042 0.045 0.048 0.057 20.0 0.047 20.0 0.045 20.0 0.060 10 1.0 5 1.0 2 1.0 5 10 5 5 5 2 4 5 4 5 4 5 4 4 4 n F R(full) usd h 5 ofiec nevlfrtp ro wihhsabnma ( binomial a has (which error I type for interval confidence 95% the outside hsnrnol rmanra itiuinwt enzr n tnaddeviation standard and zero mean with distribution normal a from randomly chosen 02. .5 .6*009 .7*0.058 0.057 0.057 0.038 0.071* 0.024* 0.085* 0.026* 0.104* 0.044 0.034* 0.069* 0.038 0.070* 0.061 0.045 0.032* 0.067* 0.033* 0.068* 0.072* 0.057 0.052 0.059 0.045 0.060 0.050 20.0 0.055 20.0 0.043 20.0 0.049 10 1.0 5 1.0 2 1.0 5 10 5 5 5 2 4 5 4 5 4 5 4 4 4 simulations 1000 of out error) I (type hypothesis null true with the of design, rejections level of significance Proportion with II TABLE ab Ya R(reduced) Y ¼ ¼ ¼ normal-theory ¼ netitdpruaino a data; raw of permutation unrestricted 00smltosand simulations 1000 emtto frwdt as data raw of permutation ¼ emtto frsdasudrtefl model; full the under residuals of permutation ¼ emtto frsdasudrterdcdmodel; reduced the under residuals of permutation F A ¼ F ij ts tbe values). (tabled -test F MS hcl and cell th -ratio. A ¼ = MS .5frtets fahge akdfco A natowynse ANOVA nested two-way a in (A) factor ranked higher a of test the for 0.05 p s B ¼ B atrBi admadnse ihnlvl fA eeso were B of Levels A. of levels within nested and random is B Factor . ab 0.05, y ijk nt,wihpoie neatts nti case; this in test exact an provides which units, e ijk m i.e. ¼ steukonpplto mean, population unknown the is a (eue)RFl)F R(Full) R(Reduced) Y Yab steukonerrascae ihobservation with associated error unknown the is .3–.6)aeidctdwt nasterisk. an with indicated are 0.036–0.064) , m e þ e A e e Uniform i Lognormal þ exp N ð B 0 ð j ; 1 ð 1 þ 1 Þ 3 Þ ; 10 AB e Þ saei.i.d. are ’s ij þ e ijk n AB , p ij itiuinwith distribution ) s B niae h inter- the indicates austa lie that Values . y ijk .As Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 ein ybl r si iue2 except 2, Figure in as are Symbols design. o anefcs n a aclt n emt residuals control permute to and attempts that calculate test approximate can an one For effects. effects, main would the main that of left for each permutations possible of no levels are within there because an so using an is interaction give This an 2001). for Pesarin, test in permutation tives exact no is There Interaction of Test 5.1 IUE3Pwrcre o emtto et n h normal-theory the and tests permutation for curves Power 3 FIGURE eoe ysbrcigmeans. subtracting by removed ffco .Temto fpruainaypoial prahsteeatts because, test exact the approaches asymptotically permutation of SS method although The B. factor of where y i :: F rtodfeett h bevdvlewe emttosaersrce ooccur to restricted are permutations when value observed the to different -ratio and A n SS and y ... r speiul ecie and described previously as are B r o etcntn,vraiiydet n r siae and estimated are B and A to due variability constant, kept not are UT-ATRA NLSSO AINE97 VARIANCE OF ANALYSIS MULTI-FACTORIAL ¼ ()(emtto frwdt ihnlvl fB). of levels within data raw of (permutation Y(B) y : j : stema fosrain nlevel in observations of mean the is F ts o w-a rse xdeffects fixed crossed two-way a for -test F rto(u e oealterna- some see (but -ratio r ijk ¼ y ijk y i :: y : j : þ y ... j , Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 oe nScin51aoe h etfrterno fetwudb rvddby provided be would effect random the effects for fixed the test for The discussed as above. be would 5.1 interaction Section F of test in the model, model mixed a Interaction such No factor. Factor, For random Fixed a of is Test B factor fixed, is 6.1 A factor while design effects that fixed crossed consider the now for but as way above), same 5 the in (Section denoted be can model mixed crossed The MODEL MIXED DESIGN, CROSSED 6 I). (Tab. normal-theory distributions the (hypothesis errors while power error When null 3d), greatest uniform 3c). Fig. the false or Fig. I, had (Tab. I, normal the residuals tests (Tab. of the distribution detect of permutation error any cubed, to the of exponential of power or regardless lognormal less tests, either other were significantly the had zero. of any at test than set exact were effects the interaction situation, All increased. gradually was A factor of effect 8M .ADRO N .J .TRBRAAK TER F. J. C. AND ANDERSON J. M. 98 ol o ehl osati hspruainstrategy. permutation this in constant held be not would ie eesof levels Fixed o tep ocnrlfrtemi fet fAo ,bt si l ae ihcomplex with cases all in as but, B, the or of A properties SS of pivotal among effects permutation the main under on the for relies control designs, to attempt not h motn itnto rmtecmltl xdefcsmdlcmsaoti h etof test the the in Here, about model. mixed comes the model in effects A, fixed factor, completely fixed the the from distinction important The ( above 5.2 Section eoe by denoted et n h normal-theory the and tests rvd neatts ntestainweeasgicn Bitrcini rsn,a SS as present, is not does interaction method AB this significant that a Note exact. where is situation test the the in so A, test of exact effect an the provide of test the in permutations o atrAcnit npruigrwosrain u etitn hmt cu ihnlevels within occur to them restricting calculating but and observations raw B permuting of in consists A factor for ftets o neato sntsgicn,tsso aho h anefcsaeraoal.In reasonable. are effects main the of each of tests significant, E not case, is that interaction for test the Effect If Main of Test 5.2 I). Tab. 3b, traditional (Fig. the errors than powerful cubed or more exponential significantly permutation was however, data residuals, Wilcoxon of raw the Permutation although I). 3a), (Tab. (Fig. errors normal-theory form errors the normal that with indicated situation test the for power comparable had netitdpruaino a aa iuain eedn gi ocmaerelative compare to again done these, were For Simulations methods. data. these raw of power of permutation unrestricted eolvl fAwr e at set were A of levels zero B neaan yeIerr ftemtosddntdfe infiatyfo .5 o this For 0.05. from significantly differ not did methods the of errors I type again, Once prxmt et r rvddb ihrpruaino h residuals the of permutation either by provided are tests Approximate netitdpruaino a aai nte prxmt etta a eue.I does It used. be can that test approximate another is data raw of permutation Unrestricted iuain eedn oeaietp ro n eaiepwro hs w approximate two these of power relative and error I type examine to done were Simulations yeIerrddntdfe infiatyfo .5frayo hs ehd.Alo h tests the of All methods. these of any for 0.05 from significantly differ not did error I Type ¼ MS B = ð y MS MS AB where , R AB AB n ol rce sfrtets fmi fet ecie in described effects main of test the for as proceed would and , Þ¼ i.e. ij F eecoe opoieagaulices nteitrcineffect, interaction the in increase gradual a provide to chosen were A E h eto anefc nteasneo infiatinteraction). significant a of absence the in effect main a of test the , y ð ¼ MS AB MS F ¼ R A ts,a npeiu iutos Here, situations. previous in as -test, A Þ¼ A i q ; ¼f = SS P ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi MS s B 50 e 2 i a ¼ ; R n h oe is model the and F 1 a SS ; hsmasSS means This . rtowsms oefli hsstainadfruni- for and situation this in powerful most was -ratio P ¼ AB 50 j b ¼ 4 F 1 ; g n SS and ts ntestainwt ihrlgomlor lognormal either with situation the in -test ð b AB n fBwr e at set were B of and ¼ ij F 4 ; rtoi osrce as constructed is -ratio F R n rto hsapoc ie variability mixes approach This -ratio. AB . ¼ B y ieSS like , ijk Þ 5, 2 = ¼ ab B F j m ts a otpwru o the for powerful most was -test and ¼f þ T ean osatars all across constant remains , A 50 AB i a B þ ; j ¼ ¼ ¼f B 50 2 j P ; þ F 50 ; b r 20 A i a e ¼ ijk ¼ ; ijk 1 ¼ ; ¼ P neattest exact An . 4 50 MS ; 20 j b n y ¼ ; ijk 1 ¼ 20 g A AB = n the and ; ,non- 5, MS y ij : = j 20 : AB ab AB or g . . . Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 ffco ysbrcigmas u oueteecagal nt dnie o h exact the for identified units exchangeable the use to SS but keeps means, which subtracting test, by B factor of oocrwti eeso atrB codn oteguideline. the to according B, factor of levels within occur to ( ean osattruhu h emttos(i)SS (iii) permutations the of the permutation throughout (ii) constant permutations, remains the the throughout are constant units remains exchangeable the A, factor of test exact the for that indicates This eeso B of levels ihma eoadsadr deviation standard and zero mean ANOVAwith traditional the to a addition in tions, fmasi re oioaetets ffco .Oefrhrpsil prxmt eti to is test approximate possible further One A. factor of as test only but the residuals, isolate these to permute order in means of ecie xhne aiblt nyars SS across only variability exchanges described F i)SS (iv) emt a aavle ihu etito.Atraiey n a emt h residuals the permute can one Alternatively, restriction. without values r this data included We raw case. permute this guideline. the in to restricting according test test consequence, exact exact the a with an As comparison provide due factor. for (below) random not variability simulations a our of does in is method B component B a of that levels includes fact the within A to permutations of ( due square term, B mean interaction expected of the to the levels that within fact observations the ignores permuting simply of consist oprdfrti iuto.Frt oivsiaetp ro,dt eegnrtdwith generated were data error, I type investigate to factor. random First, the situation. of levels a this across aver- detected for on be know, compared still to can wishing effect from nested the main experimenter a is, fixed an as consistent That deter (much a model. not variation if may interaction, the of age, this the to source but component by significant does), error caused a factor contribute random random variability may a any effect above contributes interaction simply and the random case over in pre- interested this effect, the be in fixed in may which effects one significant however, main model, a test mixed for to the For illogical test interaction. considered significant generally a is of it sence model, Interaction effects of fixed the Presence Factor, For Fixed of Test the still was 6.2 I). B Tab. of 4c, levels (Fig. within tests data raw the of all permutation of power- while powerful most test, was least exact I). residuals Tab. the of 4a, This by permutation (Fig. B. cubed, followed margin of exponential wide ful, levels a or within quite lognormal data by were raw method errors of other When permutation any than the powerful for less except significantly increase tests, was to permutation the changed of normal-theory all gradually the zero. by uniform, and at or fixed set normal were was were errors effect A interaction factor the of while Levels A, A. of of effect levels the the of each ol hsnfo omldsrbto ihma eoadsadr deviation standard and zero mean with distribution normal a from chosen domly i.e. ijk A ¼ ¼ hspoie neatts o eas i emtto ihnlvl fBmasSS means B of levels within permutation (i) because A for test exact an provides This loehr hr eetu ifrn ehd fpruaincmae nteesimula- these in compared permutation of methods different 5 thus were there Altogether, eea prxmt emtto ehd ol eue o hsts.Frt n a simply can one First, test. this for used be could methods permutation approximate Several would A factor of test exact an that is situations such for wisdom common the that Note hs h v ifrn emtto ehd ulndi eto . bv eealso were above 6.1 Section in outlined methods permutation different five the Thus, When errors. I type empirical their in 0.05 from significantly differ not did methods The ¼ ¼ i.1) utemr,teeatts eursta emtto fteeuisb restricted be units these of permutation that requires test exact the Furthermore, 1c). Fig. , 4 4 ; ; MS y b b ijk T ¼ ¼ A ean osattruhu h emttos hs h emtto strategy permutation the Thus, permutations. the throughout constant remains = and 4 and 4 ð y MS B : j : j hc tep ormv h feto atrBi h aatruhsubtraction through data the in B factor of effect the remove to attempt which , ; AB j ¼ n . n ¼ 1 ¼ R ; ... .Lvl fBwr bandrnol rmanra distribution normal a from randomly obtained were B of Levels 5. osatars permutations. across constant 0adlvl fAwr e tzr,wielvl fBwr ran- were B of levels while zero, at set were A of levels and 10 UT-ATRA NLSSO AINE99 VARIANCE OF ANALYSIS MULTI-FACTORIAL ; b Þ lhuhrno,wr eeteestesm orlvl in levels four same the nevertheless were random, although , ab nt.Tertoaefrti eti ormv h effect the remove to is test this for rationale The units. s B ¼ F 20 saitc(a.I.Dt eesmltdwith simulated were Data I). (Tab. -statistic A : .Nt ht nietense ae h 4 the case, nested the unlike that, Note 0. F ts a otpwru,floe closely followed powerful, most was -test n SS and T ¼ AB SS hc seatfrats of test a for exact is which , e.g. A þ digo,19) This 1995). Edgington, , SS B ab þ SS nt en SS means units AB þ SS ab R units and R B Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 0 .J NESNADC .F E BRAAK TER F. J. C. AND ANDERSON J. M. 100 þ¼ m n s follows: as are Symbols random). normal-theory is the (B and strategies model u permutation mixed for crossed curves Power two-way a 4 FIGURE eoadsadr deviation standard and zero o ntecosdmxdmdlwudb h aea h eto ntense design nested the in A of test the as same the be would model mixed crossed the in A tor nrae.Nt hti h tnaddvainfrrno atrBi h rse mixed crossed the in B factor random for deviation standard call may the we (which if model that Note increased. e at set B ¼ ¼ ¼ ¼ ¼ ¼f pruaino residuals), of (permutation R ()b(emtto frwdt as data raw of (permutation Y(B)ab a pruaino eiul as residuals of (permutation Rab normal-theory ()(emtto frwdt etitdwti eeso B), of levels within restricted data raw of (permutation Y(B) pruaino a data), raw of (permutation Y a 0 ; ¼ 1 ; 4 5 ; ; b r10 or F ¼ -test. 4 ; g n eeso Bwr locoe rmanra itiuinwt mean with distribution normal a from chosen also were AB of Levels . ¼ 5 s ; B s s ; B crossed AB ab ¼ ¼f ab units), 1 : o lrt)i qa ozr,te h eto h xdfac- fixed the of test the then zero, to equal is clarity) for and 0 nt,rsrce ihnlvl fB), of levels within restricted units, 0 ; 1 ; 5 s ; AB r10 or ¼ 5 g : oivsiaepwr aaeeswere parameters power, investigate To . ,wieteefc fAwsgradually was A of effect the while 0, F ts o h eto xdfco A in (A) factor fixed a of test the for -test Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 whenever andtp ro nalstain Tb I) h rdtoa normal-theory traditional The III). (Tab. situations all in error as I B type factor tained of residuals permuting and line) rse ie einpoie e iuto,wihi h aeo neethere. interest of case the is which situation, new a provides design mixed crossed h xc et( test exact The enzr n tnaddeviations standard and zero mean the ofiec nevlfrtp ro wihhsabnma ( binomial a s has (which error I and type simulations for interval confidence Y(B)ab Rab Y(B) Y 1000 of out error) I (type hypothesis null true with the design, ANOVA of model rejections mixed level of significance with Proportion simulations III TABLE F 01 .4 .0*0030020000.042 0.042 0.056 0.040 0.039 0.056 0.042 0.043 0.065* 0.043 0.042 0.057 0.000* 0.000* 0.053 0.045 0.042 0.056 10 5 0 10 10 10 01 .5 .0*0030000000.058 0.046 0.046 0.060 0.050 0.045 0.060 0.049 0.047 0.063 0.046 0.046 0.000* 0.001* 0.047 0.059 0.051 0.050 10 5 0 10 10 10 01 .5 .0*0080010070.059 0.046 0.050 0.057 0.047 0.048 0.061 0.047 0.047 0.058 0.047 0.048 0.000* 0.000* 0.057 0.057 0.043 0.050 10 5 0 10 10 10 01 .4 .3*0090020020.041 0.039 0.038 0.042 0.041 0.044 0.062 0.060 0.062 0.059 0.058 0.055 0.033* 0.052 0.055 0.042 0.055 0.058 10 5 0 10 10 10 0008000 .4 .4 .4 0.045 0.052 0.050 0.041 0.049 0.054 0.043 0.048 0.048 0.048 0.050 0.054 0.042 0.055 0.049 0.046 0.053 0.052 0.043 0.051 0.052 0.000* 0.054 0.000* 0.047 0.051 0.056 0.048 0.000* 0.057 0.000* 0.059 0.055 10 0.052 5 0.044 0 0.052 5 10 5 5 0 5 0 0 0 0000000 .4 .4 .4 0.043 0.047 0.051 0.041 0.052 0.040 0.060 0.051 0.058 0.046 0.055 0.044 0.055 0.051 0.055 0.043 0.047 0.045 0.061 0.050 0.058 0.000* 0.050 0.000* 0.057 0.052 0.058 0.040 0.000* 0.044 0.000* 0.054 0.053 10 0.056 5 0.062 0 0.057 5 10 5 5 0 5 0 0 0 0002000 .3 .4 .4 0.039 0.048 0.051 0.041 0.047 0.049 0.040 0.049 0.049 0.040 0.047 0.049 0.041 0.052 0.048 0.039 0.047 0.045 0.046 0.050 0.047 0.000* 0.045 0.000* 0.047 0.054 0.047 0.042 0.000* 0.051 0.000* 0.056 0.052 10 0.049 5 0.040 0 0.047 5 10 5 5 0 5 0 0 0 0006009007 .7*0010.050 0.041 0.032* 0.051 0.055 0.046 0.038 0.038 0.032* 0.073* 0.055 0.070* 0.039 0.046 0.042 0.067* 0.074* 0.067* 0.068* 0.048 0.054 0.039 0.076* 0.059 0.065* 0.045 0.056 0.056 0.042 0.056 0.047 0.048 0.052 10 0.058 5 0.047 0 0.052 5 10 5 5 0 5 0 0 0 B ¼ ¼ s normal-theory ab ¼ emtto frwdt;R data; raw of permutation ¼ AB emtto frsdas as residuals, of permutation ¼ B; of levels within data raw of permutation eeso h neato Bwr ahcoe admyfo omldsrbto with distribution normal a from randomly chosen each were AB interaction the of levels emtto frwdt,as data, raw of permutation ¼ s AB s i.e. B ; permuting , nested p F ts tbe values). (tabled -test ¼ ()bYB a F Rab R Y Y(B) Y(B)ab 0.05, oee,when However, . UT-ATRA NLSSO AINE101 VARIANCE OF ANALYSIS MULTI-FACTORIAL i.e. ¼ .3–.6)aeidctdwt nasterisk. an with indicated are 0.036–0.064) , emtto fresiduals; of permutation ab ab ab units; ¼ nt ihnlvl ffco ,a pcfidb h guide- the by specified as B, factor of levels within units F s nt,wti eeso ,wihgvsteeatts nti case; this in test exact the gives which B, of levels within units, .5frtets fafie atr()i w-a crossed two-way a in (A) factor fixed a of test the for 0.05 A B e ¼ and e MS e e Uniform Lognormal ab s s A AB exp N = B MS ; ð nt eeteol et htaeutl main- adequately that tests only the were units crossed epciey austa i usd h 95% the outside lie that Values respectively. , 0 ð ; 1 ð AB 1 1 Þ 3 Þ ; The . 10 6¼ Þ and 0 b eeso h admfco and B factor random the of levels n , p itiuinwith distribution ) s AB 6¼ ,tets fAi the in A of test the 0, F ts maintained -test n ¼ 1000 Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 0 .J NESNADC .F E BRAAK TER F. ( J. C. term errors AND interaction lognormal ANDERSON J. or the M. uniform to normal, due of variation case included the the data when in simulated conservative conservative when too extremely were be advo- usually that to permutation ( errors tended of tests method but the such contrast, errors, for In cated normal III). (Tab. with cubed exponential situations were errors for error I type 102 ainpoeue.W atnt d htti eutwsntcue ytenme frestricted of number the by caused not was result permu- this appropriate that Manly to permu- add regard to restricted and with hasten than We wisdom Gonzalez power procedures. current tation by greater the provided challenges found residuals designs ANOVA also of for tations permutation was error that permutations fact I restricted The type (1998). nested for maintaining the power for for low best as Relatively Here, was model. test mixed exact fixed power. a a the of reasonable in distributions, test having interaction the error and non-zero was such a this error, with to of I exception designs presence type The the maintaining designs. while in crossed test, with factor powerful situations most all correct the ( virtually maintaining provided for model right-skewed 1983) to reduced (both Lane, a cubed addition and under exponential Freedman in or residuals lognormal of test, were permutation powerful errors distributions), when most However, error. the I provided type values) tabled tional situations all ( for test levels exact as the nominal was B at however, test of 4d), error powerful Fig. residuals I I, most of (Tab. type next power permutation maintained greatest The the methods III). had (Tab. these residuals of of permutation neither and data raw o h xc et n aiu prxmt emtto ehd a edn n we normal-theory and the done distributions), demonstrate symmetric be simulations. (both that can using distributed designs methods uniformly designs two-way permutation these ANOVA of for approximate two-way empirically levels various power within for their and may permutations compared tests This examples restricted have exact models. as ANOVA provided well the in as have how terms units We individual of con- for the groups factors. squares, test of mean expected permutation permutation of exact include basis an the on of indicates, that struction guideline a provided have We DISCUSSION This 7 4d). (Fig. approaches simulated other were errors all lognormal in or to method uniform power this compared either some errors, when power powerful I). have normal (Tab. least using did in the obtained B also lacking was results factor method severely extreme of the levels still to within was contrast data in raw situation, of this permutation the Although eeso h te atr( within units factor observation of other other permutation the the the among than differences powerful of significant more no were levels were them There of 4b). all Fig. except I, (Tab. tests, tests other the than ful l,ee hntepwro h te et a er10 Tb I,Fg 4b). Fig. at III, A (Tab. factor 100% of near effects was tests significant other any the find of to power failed the test when this even all, errors), normal and interaction n a aasoe nae yeIerri eti icmtne ihepnnilcubed exponential with circumstances certain in permut- error or I B non-zero type factor and of inflated residuals errors showed permuting data of raw methods approximate ing the contrast, In III). Tab. etitdpruainmtosvrulyawy eutdi et ihvr o power. low very with tests in resulted always virtually methods permutation Restricted eut rmsmltoshv hw ht ngnrl hnerr eeete omlor normal either were errors when general, in that, shown have simulations from Results ntrso oe,wt omlerr,tetraditional the errors, normal with power, of terms In hnerr eerdclynnnra ( non-normal radically were errors When s i.e. AB emtn a aawti eeso atrB aeeprcltp I type empirical gave B) factor of levels within data raw permuting , Tb III). (Tab. i.e. () a.I.I at ihteeprmtr (non-zero parameters these with fact, In I). Tab. Y(B), , ab nt,floe ythe by followed units, i.e. xoeta ue) hntepruainof permutation the then cubed), exponential , F ts a infiatymr power- more significantly was -test i.e. e.g. F ()b,floe by followed Y(B)ab), , ts Tb ,Fg 4d). Fig. I, (Tab. -test tl n ht,1981; White, and Still , i.e. o non-zero for , F ts uigtradi- (using -test s AB , Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 Fg ) hsmasteosre au osntapa xrm eaiet hsdistribution this large, individual to relative should, too extreme it average, appear as on not frequently does is, as value permutation permutation observed under the under the means A This of of Thus, 5). way. square (Fig. any distribution in mean to controlled the average, the not on causing of is A, effect of value interaction square expected interac- the mean an when the the to permutation causes due under B variability inflated of method significant null be levels this is within should the there occurred permutations Why When of rejections restricting situation. power? tion, this low rejections no in such virtually no have distributions Similarly, permutation in error 4c). of uniform resulted Fig. or (B) III, lognormal (Tab. either factor restricting for errors that random found normal was the for It I of here. hypothesis, attention levels type more within some good deserves permutations and model mixed power a an of in give greater option interaction used. will in the are latter units has results The exchangeable generally generally factors. correct one residuals those the contrast, provided of permuting in error, levels order, while within lower test permuting or exact of same or the control To residuals of units. permuting are observation individual that the be terms not for may which units, appropriate permuting by n h eutn itiuinof distribution resulting the and uainlapoc st eue nteestain,a prxmt emtto etmust per- test a permutation If approximate constructed. an be situations, ever these could and in are test used chosen. Freedman there exact be be addition, by no to empirical In is which described with 1999). approach for residuals Legendre, mutational models consistent ANOVA and of in are (Anderson permutation terms regression results some of multiple Our for method test. (1983) the appropriate Lane the the for provided for obtained error, permuted I results type this situa- were a inflated approaches simulate in units asymptotically to resulted able residuals denominator residuals not were of using we permutation test errors where non-normal permutation tion radically level a using significance by chosen of even the it level, error at nominal I pursue maintained be type not 5) will the error do I While Fig. we type that although with ensures obviously problem, test (along this explanation into insight above of some distribution The the least here. changing at permutation. further in provide restrictions under to of appears statistics role the test clarify to relevant necessary certainly is work equal not does ubro osbersrce emttosfrtets fAi h xdmdlwas model fixed the in A of test the for the permutations simulations, our restricted in example, 2 possible For power. of reasonable obtain number to few too being permutations oe htaeo ihrodr( order higher of are that model situations. main- permutation not all do approximate in they the as error of units, I none exchangeable these type situations, permute these accurate not in tain do which that used note be to should important methods is It test. the xhnebeuisfrtets.I loapisi h aeo etfrasgicn xdmain fixed significant a the for Here, test model. a mixed of a the case higher as in the a used in effects in of be interaction applies the tests test must any also the factor indicates over such It of nested effect this case the test. for by the the then induced used in for units example, residual, units be The for exchangeable hierarchy. the applies, nested must This not a error. units in is I factor exchangeable type ranked interest in These bias of avoid test. term to the order the denominator for of the Whenever units test square. exchangeable the mean residual for the square is mean square mean denominator the where : 52 h pcfi iuto fats o infiatmi fet()i h rsneo an of presence the in (A) effect main significant a for test a of situation specific The eti rd-fsatn h s fa xc etvru emtto frsdas h exact The residuals. of permutation versus test exact an of use the attend trade-offs Certain h otmln rmteesmltosi htoems oto o te em nthe in terms other for control must one that is simulations these from line bottom The h bv bevtoso etitdpruain pl oaltsso NV terms ANOVA of tests all to apply permutations restricted on observations above The 10 8 n o h ie oe a 5 was model mixed the for and a i stosal n oe scmrmsd oeie eeey Further severely. sometimes compromised, is power and small) too is (it UT-ATRA NLSSO AINE103 VARIANCE OF ANALYSIS MULTI-FACTORIAL e.g. F P saitcudrpruaint esitdt h right the to shifted be to permutation under -statistic vle sn ogruiom(i.5.S,tp error I type So, 5). (Fig. uniform longer no is -values flwrrn naheacy,weeti snecessary, is this where hierarchy), a in rank lower of : 71 10 34 . P vle edt eover-estimated be to tend -values ab nt utb emtdfor permuted be must units ð a Þ . Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 grs h ulhptei a readtedt eesmltdacrigt w-a ie oe with model mixed two-way a to according simulated were data the and true was hypothesis a null The figures. h itiuinof distribution the 0 .J NESNADC .F E BRAAK TER F. J. C. AND ANDERSON J. M. 104 IUE5Cmaio ftedsrbto fthe of distribution the of Comparison 5 FIGURE et a ecntutdfridvda em nAOAfrtowyadtrewydesigns. three-way and two-way for ANOVA in terms individual for constructed be can tests square). mean a residual in the indivi- not factor ( where is ranked units used exchangeable test somewhat higher be correct the if cannot a the reasonable, it not of a 1998), are test Manly, units provide the and dual can (Gonzalez in it test error Although approximate 4.2). I conservative, (Section type design inflated hierarchical from nested suffered also and duals ¼ o eeec,w rvd alsi nApni Tb.A–I)ta hwhwexact how show that AI–AIV) (Tabs. Appendix an in tables provide we reference, For resi- of permutation than power less significantly had data raw of permutation Unrestricted 4 ; b ¼ 4 ; n ¼ P 10, vle ne iuain(00smltos o w emtto ehd,dntda nprevious in as denoted methods, permutation two for simulations) (1000 simulation under -values e N ð 0 ; 1 Þ ; s B ¼ and 0 s AB F ¼ saitcudrpruain(hr ee99pruain)and permutations) 999 were (there permutation under -statistic 1. i.e. hntednmntrma qaefor square mean denominator the when , Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 situations. eesr,icuigb eeec oteuiiyadrltv oe fsc approaches such of is area power this relative in ANOVA and work ( complex further utility GLMMs that or in the clear GLMs is example, to tests for It to, reference for used. compared statistic by distributions of including kind permutation necessary, the of appropriate regardless develop designs, to order itiuin,sc prahsmywl emr prpit hnuigpermutation using than and appropriate permutations restrict more be skewed well highly traditional with the may data with univariate approaches methods For least-squares. such than rather distributions, 1996) Richards, and Cade of use permu- the appropriate the making using ( done dis- thus be here on can described based distributions, and as tests situation of this procedures permutation in tation contrast, kinds robust In quite different are unfeasible. matrices approaches have tance related often other may and GLMs set data same ttsis uha o et fmdaso sn es bouedvain ( deviations absolute least ( here using addressed or medians of tests for as such statistics, ( designs ANOVA complex in data multivariate S of and analysis the for of strategies tests valid for permutations. care using with experi- the observed obtained tests, the be be with parametric must of for to analysis consistent logic As and hypotheses is terms. the design individual experimental tests by of of driven exact squares logic is mean proper of expected and ANOVA construction the and parametric the mean design in for mental expected applied provide rules the we traditional of guideline the The terms pro- done. parametric in attending be also, logic to strategies The pivotal design. permutation the the to of squares, applies structure to ANOVA the as in thought and without factors cedures can or the indiscriminately test of done the nature be for the not units must permutations exchangeable ANOVA, in of designs choice the in or squares statistic mean consequences. test expected undesirable ignores a have which of approach construction an the taking in that simulations using here shown nested or terms interaction possible random as the including units for hypothesis, individual null allows with factors. the associated This under errors hypothesis. variability the null of just the than sources more under being chosen. errors there were of of levels exchangeability possibility how of on of virtue so conditional by regardless being is validity experiment, tests This the the necessity. in with ran- included by treatments, of actually fixed virtue of levels by combinations considered factor validity to are the their units obtain factors of to any assignment tests of dom randomization considers levels permuta- approach the any this in because as anathema well an as as of some vations notion by whole considered the [ that be context noting might ‘‘random’’ worth tion structures is as nested It factor random. identifying a or to fixed treating given are is factors care individual due whether provided indicate and approach, also test cell constructed. permutation They be a of residuals. using can subtraction of test (through exact permutation ‘‘removed’’ no by be which provided for to test situations need approximate the terms the which for indicate means) also tables The ehv o icse h oeta s fpruainmtoswt oerobust more with methods permutation of use potential the discussed not have We permutation appropriate choosing for utility great has here provided have we guideline The complex for tests permutation of use the in choices some has experimenter the Although have we take, to chooses experimenter an approach philosophical which of Regardless their gain tests permutation that is introduction, the in stated as approach, our contrast, In analyzed be can experiment designed ANOVA any that here philosophy the taken have We ˇ iae,19;Adro,20) o utvraedt,dfeetvralswti the within variables different data, multivariate For 2001). Anderson, 1998; milauer, i.e. e al 19) eto .,p.1213.I htfaeok h obser- the framework, that In 142–143]. pp. 7.6, Section (1997), Manly see , e.g. h hieo xhnebeuisadtecoc fwehro o to not or whether of choice the and units exchangeable of choice the , F saitcue o h etadtewyi hc h emttosare permutations the which in way the and test the for used -statistic = UT-ATRA NLSSO AINE105 VARIANCE OF ANALYSIS MULTI-FACTORIAL rpruersdas ilsilb eyipratt osdrin consider to important very be still will residuals) permute or F rto.W rps,hwvr httesm susw have we issues same the that however, propose, We -ratios. e.g. crl n nesn 2001). Anderson, and McArdle , e.g. rso n lyo,19)i different in 1993) Clayton, and Breslow , e.g. e Braak ter , e.g. see , Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 References content. Underwood its J. A. regarding and comments McArdle H. and B. Post-doctoral thank discussions We U2000 Sydney. for a of University and Award the at Postgraduate MJA Australian to Fellowship an by supported was research in This permutation by test Acknowledgements any for use its recommend we ANOVA. however, meaning, interpretable 01.Frteeattss hr r ipe ttsis uha h u fsurso mean or squares of the sum the to as related such ( statistics, permutation monotonically simpler under are are there that tests, square, ( exact designs the ANOVA For complex 2001). in tests permutation approximate ac ee hsnfrtets.Pruaino n ido eiul ilawy ieappro- guideline. give the always using will residuals constructed of those kind are any tests of Exact Permutation tests. test. ximate the for chosen level cance aypoial xc)we hi yeIerraypoial prahsthe approaches asymptotically error I type their when exact) (asymptotically seatyeult the to equal exactly is uaino a aacnb oeuigtegieieacrigt h NV oe ( model ANOVA the to according guideline the using done be can data raw of mutation posed (f) through (a) questions answering Introduction: briefly by the results in our conclude and summarize We CONCLUSION 8 nesn .J n eede .(99.A miia oprsno emtto ehd o et fpartial of tests for methods permutation of comparison empirical An (1999). P. Legendre, and variance. J. of analysis M. multivariate Anderson, non-parametric for method new A (2001). J. M. Anderson, the of square mean reliable denominator and powerful most the give may residuals have of test. will there- where exact permutation We situations approximate error. impossible, residuals in an I is Indeed, of type test applications. accomplish maintaining exact general permutation while for (to an test, that residuals exact of permutations the indicate permutation to) recommend restrict equal results fore (or generally Our than further greater residuals. to is one that permute however, power whether level to of units, maintain or choice to exchangeable test) mandatory the appropriate factors). but has other the optional, then of not identified is levels Having units within accuracy. exchangeable restrictions correct BRAAK and the TER F. units Permuting J. exchangeable C. AND appropriate ANDERSON J. permuting M. 106 nesn .J n oisn .(01.Pruaintssfrlna models. linear for tests Permutation (2001). J. Robinson, and J. M. Anderson, prxmt emtto etfritrcinterms. interaction for test permutation approximate the using prxmt et eeal ihgetrpwr st emt eiul ftoefactors. those of residuals permute to an (yielding is per- restrictions power) such restricting greater to by alternative account with The into generally test. taken test, exact be an approximate may for these levels order, their lower within or mutations similar of factors are there (f) (a) (b) (e) (d) (c) ersincefiinsi iermodel. linear a in coefficients regression 46. 75–88. hc et r xc n hc r approximate? are which and exact are tests Which hnsol emttosb etitdt cu ihnlvl fohrfactors? other of levels within occur to restricted be permutations should When hudrwdt eprue rsm omo residuals? of form some or permuted be data raw Should htts ttsi hudb used? be should statistic test What hc nt hudb permuted? be should units Which o a neato em etse sn permutations? using tested be terms interaction can How F rto emtn h eiul fmi fet rvdsapwru n valid and powerful a provides effects main of residuals the Permuting -ratio. e.g. priori a digo,19) sthe As 1995). Edgington, , infiac ee hsnfrtets.Tssaeapproximate are Tests test. the for chosen level significance F rtofrtets sdtrie yepce ensquares. mean expected by determined as test the for -ratio .Sait opt Simul. Comput. Statist. J. xhnebeuisfrtets r dnie ythe by identified are test the for units Exchangeable ioa ttsi,sc as such statistic, pivotal A F rto hsyedn equivalent yielding thus -ratio, F rtohsa meitl aiirand familiar immediately an has -ratio eti xc hnistp error I type its when exact is test A , 62 271–303. , uta.&NwZaadJ Statist. J. Zealand New & Austral. e.g. ooti neatts,per- test, exact an obtain To ognrleatts exists test exact general No nesnadRobinson, and Anderson , F sncsayfrthe for necessary is , uta.Ecol. Austral. priori a P , -values signifi- 26 When 32– , i.e. , 43 , , Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 ile .W,Bry .J n ono,E .(96.Mlirsos emtto rcdrsfrapriori a for procedures permutation Multi-response (1976). S. E. Johnson, and J. K. Berry, W., P. Mielke, crl,B .adAdro,M .(01.Ftigmliait oest omnt aa omn ndistance- (1989). on comment A. a J. data: Nelder, community and to P. models McCullagh, multivariate Fitting (2001). J. M. Anderson, and H. B. McArdle, ada .V 17) h feto o-omlt nsm utvraetssadrbsns onnnraiyi the in non-normality to robustness and tests multivariate some on non-normality of effect The (1971). V. K. Mardia, end,P .adCd,B .(96.Rnoiaintssfrmlil regression. multiple for tests Randomization (1996). S. B. inference. Cade, experimental and of E. P. aspects Kennedy, Some (1966). O. Kempthorne, regression. models. in methods mixed Distribution-free linear (1982). generalized S. J. in Maritz, inference when and Approximate F-test M. (1993). B. theory Brown, G. normal D. the Clayton, to alternative and non-robust E. a N. test: Breslow, permutation Fisher-Pitman The (1987). J. R. Boik, edr .A 16a.Teaayi frnoie xeiet ihotooa lc tutr.I lc tutr and structure Block I. structure. block orthogonal with experiments randomized of analysis The (1964a). A. J. Nelder, models. linear generalized using analysis data Longitudinal (1986). L. S. Zeger, and K.-Y. Liang, distances metric from inferred variance molecular of Analysis (1992). M. J. Quattro, and E. P. Smouse, satisfied. L., Excoffier, not (1995). evolution. are S. and E. ecology variance in Edgington, of analysis analysis data computation-intensive the for methods for Resampling assumptions (1992). the H. P. Crowley, when consequences Some (1974). structure. regression, G. community W. deviation in Cochran, absolute changes of least analysis for multivariate Non-parametric tests (1993). Permutation R. (1996). K. Clarke, D. J. Richards, and S. B. Cade, atl .adVln,R .(90.Atcnqeo oprmti utvraeanalysis. multivariate nonparametric of technique A (1970). S. R. Valand, and N. Mantel, (1997). J. F. B. Manly, ephre .(95.Ternoiainter feprmna inference. experimental of theory randomization The (1955). O. Kempthorne, (1952). O. Kempthorne, ono,C .adFed .A 19) sn xdefcsmdlmliait nlsso ainei aiebiology marine in variance of analysis multivariate model fixed-effects Using (1993). A. C. Field, and R. C. Johnson, levels. significance reported of interpretation nonstochastic A (1983). D. Lane, and D. Freedman, edr .A 16b.Teaayi frnoie xeiet ihotooa lc tutr.I.Treatment II. structure. block orthogonal with experiments randomized of analysis The (1964b). A. J. Nelder, strategy. analysis data general a as assignment Quadratic (1976). J. Schultz, and L. Hubert, and methods measures, abundances: animal of variability temporal The (1994). H. B. McArdle, and J. K. Gaston, (1935). A. R. Fisher, lo,C .(94.Cmaaierbsns fsxtssi utvraeaayi fvariance. of analysis multivariate in tests six of robustness models. Comparative linear (1974). Generalized L. (1972). C. M. W. Olson, R. Wedderburn, and A. J. Nelder, oe .C 16) ipie ot al infiac etprocedure. test significance Carlo Monte simplified A to (1968). extensions and C. data A. multivariate structured Hope, for distance of Analysis (1999). J. W. Krzanowski, and C. J. Gower, ilr .D .adOrlo and P. D. V. Pillar, (2001). F. Pesarin, ae,A .(96.Pruaints sntdsrbto free. distribution not is test Permutation (1996). F. A. Hayes, sets. data small (2000). with I. randomization P. by Good, variance of Analysis (1998). J. F. B. Manly, and L. Gonzalez, imn .J .(97.Sgicnetsswihmyb ple osmlsfo n ouain.II h nlssof analysis The III. populations. any from samples to Scheffe applied be may which tests Significance (1937). G. J. E. Pitman, ere .R,Csla .adMCloh .E (1992). E. C. McCulloch, and G. Casella, R., S. Searle, classifications. ae eudnyanalysis. redundancy based iermodel. linear tts.Assoc. Statist. heterogeneous. are variances h ulaayi fvariance. of analysis null the Comput. data. restriction DNA mitochondrial human to application haplotypes: DNA among Syst. Ecol. Rev. Biometrics 117–143. 886–902. London. 13–22. n ecology. and tutr n h eea nlsso variance. of analysis general the and structure 29 582–598. patterns. Statist. utvraeaayi fvariance. of analysis multivariate Berlin. Springer-Verlag, releve Chichester. 69 9 ainetest. variance 53–65. , ´ .(1959). H. , 190–241. , 894–908. , ´ groups. , 1 , hls rn.R o.Lno e.B Ser. London Soc. R. Trans. Philos. 292–298. , 25 , 3 923–936. , 22–38. , caor a.Bo.An Rev. Ann. Biol. Mar. Oceanogr. Biometrika , Biometrika emtto et:APatclGiet eapigMtosfrTsigHypotheses Testing for Methods Resampling to Guide Practical A Tests: Permutation .Vg Sci. Veg. J. h nlsso Variance. of Analysis The omn tt hoyadMtosA Methods and Theory – Stat. Commun. 88 , utvraePruainTss ihApiain nBiostatistics. in Applications With Tests: Permutation Multivariate eino Experiments. of Design 23 9–25. , admzto,BosrpadMneCroMtosi Biology in Methods Carlo Monte and Bootstrap Randomization, ´ i .(96.O admzto etn nvgtto cec:mliatrcmaiosof comparisons multifactor science: vegetation in testing randomization On (1996). L. ci, 405–447. , h einadAayi fExperiments. of Analysis and Design The admzto Tests Randomization , , , 29 58 UT-ATRA NLSSO AINE107 VARIANCE OF ANALYSIS MULTI-FACTORIAL 7 Ecology 585–592. , 322–335. , 105–121. , r .Mt.Sa.Psychol. Stat. Math. J. Br. rc o.Sait o.Sr A Ser. Soc. Statist. Roy. Proc. .Ry tts.Sc e.CAp.Statist. Appl. C Ser. Soc. Statist. Roy. J. , eeaie ierModels Linear Generalized 82 290–297. , lvradBy,Edinburgh. Boyd, and Oliver r d aclDke,NwYork. New Dekker, Marcel ed. 3rd , onWly&Sn,NwYork. New Sons, & Wiley John , , 31 rc o.Sait o.Sr A Ser. Soc. Statist. Roy. Proc. 345 177–221. , 335–358. , aineComponents. Variance , , sco.Methods Psychol. 40 5 1) 1409–1424. (14), , 26–42. , 283 .Ae.Sait Assoc. Statist. Amer. J. onWly&Sn,NwYork. New Sons, & Wiley John n d hpa n al London. Hall, and Chapman ed. 2nd , 147–162. , .Ry tts.Sc e.A Ser. Soc. Statist. Roy. J. .Ae.Sait Assoc. Statist. Amer. J. , , 48 onWly&Sn,NwYork. New Sons, & Wiley John uta.J Statist. J. Austral. 1 .Ry tts.Sc e.B Ser. Soc. Statist. Roy. J. , 184–198. , 505–519. , 283 omn tts.–Simulation – Statist. Commun. n d hpa n Hall, and Chapman ed. 2nd , 163–178. , , r .Mt.Sa.Psychol. Stat. Math. J. Br. Genetics Biometrics 61 onWlyadSons, and Wiley John .Ae.Sait Assoc. Statist. Amer. J. 11–34. , uta.J Ecol. J. Austral. , , Environmetrics Biometrika , Biometrics 131 24 , , 135 .Bs Econ. Bus. J. 26 50 318–331. , 479–491. , 547–558. , 946–967. , 370–384. , n ed. 2nd , .Amer. J. Annu. , , , , 18 52 73 30 , , , , , , , Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 0 .J NESNADC .F E BRAAK TER F. J. C. AND ANDERSON J. M. 108 ee,S .adLag .Y 19) noeve ftemtosfrteaayi flniuia data. longitudinal of analysis the for methods the of overview An (1992). K.-Y. Liang, and L. S. Zeger, multispecies of analysis the and Similarity Community (1967). (1990). G. J. W. Cochran, Cairns, and W. and G. W. Snedecor, K. Pontasch, P., E. Smith, ec,W .(90.Cntuto fpruaintests. permutation of mean. Construction the (1990). J. and W. Aggregation Welch, (1961). R. L. Taylor, the to alternative an as test randomization approximate The (1981). P. A. White, and W. A. Still, ie,B . rw,D .adMces .M (1991). M. K. Michels, and R. D. Brown, J., B. Winer, S and F. J. C. Braak, ter e ra,C .F 19) emtto essbosrpsgicnets nmlil ersinadANOVA. and regression multiple in test significance bootstrap versus Permutation (1992). F. J. C. Braak, ter Medicine crwHl,NwYork. New McGraw-Hill, approach. statistical unified a data: environmental variance. otaefrCnnclCmuiyOrdination Community Canonical for Software 79–86. pp. Berlin, n Jo In: ce,K-. oh,G n ede,W (Eds.), W. Sendler, and G. Rothe, K.-H., ¨ckel, , r .Mt.Sa.Psychol. Stat. Math. J. Br. 11 1825–1839. , ˇ iae,P (1998). P. milauer, , 34 AOORfrneMna n srsGiet aoofrWindows: for Canoco to Guide User’s and Manual Reference CANOCO 243–252. , ttsia Methods Statistical Nature vrin4.McoCmue oe,Ihc,NwYork. New Ithaca, Power, Computer Micro 4). (version ae Res. Water .Ae.Sait Assoc. Statist. Amer. J. , 189 ottapn n eae Techniques. Related and Bootstrapping ttsia rnilsi xeietlDesign Experimental in Principles Statistical 732–735. , t d nvriyo oaPes ms Iowa. Ames, Press, Iowa of University ed. 6th , , 24 507–514. , , 85 693–698. , F eti nlssof analysis in test Springer-Verlag, ttsisin Statistics r ed. 3rd , Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008

APPENDIX

TABLE AI Permutation strategies for tests of particular terms in two-way analysis of variance. The linear model is: yijk ¼ m þ Ai þ BðAÞjðiÞ þ eijk , where B is nested within A (i) and yijk ¼ m þ Ai þ Bj þ ABij þ eijk , where factors A and B are crossed. In the latter case, possible designs include: (ii) A is fixed and B is fixed, (iii) A is fixed and B is 109 random, (iv) A is random VARIANCE OF ANALYSIS MULTI-FACTORIAL and B is random. Note that permutation of residuals refers to residuals under a reduced model (i.e., Freedman and Lane, 1983).

Exact test: Permutation of residuals: Terms contributing components of Term whose MS is used Additional restrictions Terms in model to be variation to expected MS of term as the denominator for Exchangeable applied to removed by subtraction Term in model being tested being tested (R ¼ residual) the F-test (R ¼ residual) units for the test exchangeable units of cell means

(i) B nested within A ARþ B(A) þ AB(A)ab’s No further restrictions None B(A) R þ B(A) R Replicates Within levels of A A (ii) A fixed, B fixed ARþ A R Replicates Within levels of B B BRþ B R Replicates Within levels of A A A BRþ A B R Replicates No test of interaction A, B (iii) A fixed, B random ARþ A B þ AA B ab’s Within levels of B B BRþ B R Replicates Within levels of A A A BRþ A B R Replicates No test of interaction A, B (iv) A random, B random ARþ A B þ AA B ab’s Within levels of B B BRþ A B þ BA B ab’s Within levels of A A A BRþ A B R Replicates No test of interaction A, B Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 1 .J NESNADC .F E BRAAK TER F. J. C. AND ANDERSON J. M. 110

TABLE AII Permutation strategies for tests of particular terms in three-way analysis of variance where the experimental design involves nesting of one factor in the interaction between the other two factors. The linear model is: yijkl ¼ m þ Ai þ Bj þ ABij þ CðABÞkðijÞ þ eijkl, where factors A and B are crossed and C is nested within A B. The possible designs include: (i) A is fixed and B is fixed, (ii) A is fixed and B is random, (iii) A is random and B is random. Note that permutation of residuals refers to residuals under a reduced model (i.e., Freedman and Lane, 1983).

Exact test: Permutation of residuals: Terms contributing components Term whose MS is used Additional restrictions Terms in model to be of variation to expected MS of as the denominator for Exchangeable units applied to removed by subtraction Term in model being tested term being tested (R ¼ residual) the F-test (R ¼ residual) for the test exchangeable units of cell means

(i) A fixed, B fixed ARþ C(A B) þ A C(A B) abc’s Within levels of B B, A B BRþ C(A B) þ B C(A B) abc’s Within levels of A A, A B A BRþ C(A B) þ A B C(A B) abc’s No test of interaction A, B C(A B) R þ C(A B) R Replicates Within levels of A and B A, B, A B (ii) A fixed, B random ARþ C(A B) þ A B þ AA B ab’s No further restrictions B BRþ C(A B) þ B C(A B) abc’s Within levels of A A, A B A BRþ C(A B) þ A B C(A B) abc’s No test of interaction A, B C(A B) R þ C(A B) R Replicates Within levels of A and B A, B, A B (iii) A random, B random ARþ C(A B) þ A B þ AA B ab’s No further restrictions B BRþ C(A B) þ A B þ BA B ab’s No further restrictions A A BRþ C(A B) þ A B C(A B) abc’s No test of interaction A, B C(A B) R þ C(A B) R Replicates Within levels of A and B A, B, A B Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008

TABLE AIII Permutation strategies for tests of particular terms in three-way ANOVAwhere the experimental design involves nesting in only one of two crossed factors. The linear model is: yijkl ¼ m þ Ai þ Bj þ CðBÞkð jÞ þ ABij þ ACðBÞikð jÞ þ eijkl, where factors A and B are crossed and C is nested within B. The possible designs include: (i) A is fixed and B is fixed, (ii) A is fixed and B is random, (iii) A is random and B is fixed, and (iv) A is random and B is random. Note that permutation of residuals refers to residuals under a reduced model (i.e., Freedman and Lane, 1983).

Terms contributing components Term whose MS is Exact test: Permutation of residuals: of variation to expected used as the denominator Additional restrictions Terms in model to be MS of term being tested for the F-test Exchangeable units applied to removed by subtraction

Term in model being tested (R ¼ residual) (R ¼ residual) for the test exchangeable units 111 of cell means VARIANCE OF ANALYSIS MULTI-FACTORIAL

(i) A fixed, B fixed ARþ A C(B) þ AA C(B) abc’s Within levels of B B, A B BRþ C(B) þ B C(B) bc’s Within levels of A A, A B C(B) R þ C(B) R Replicates Within levels of A and B A, B, A B, A C(B) A BRþ A C(B) þ A BA C(B) abc’s No test of interaction A, B A C(B) R þ A C(B) R Replicates No test of interaction A, B, C(B), A B (ii) A fixed, B random ARþ A C(B) þ A B þ AA B abc’s No further restrictions B BRþ C(B) þ B C(B) bc’s Within levels of A A, A B C(B) R þ C(B) R Replicates Within levels of A and B A, B, A B, A C(B) A BRþ A C(B) þ A BA C(B) abc’s No test of interaction A, B A C(B) R þ A C(B) R Replicates No test of interaction A, B, C(B), A B (iii) A random, B fixed ARþ A C(B) þ AA C(B) abc’s Within levels of B B, A B BRþ A C(B) þ A B þ C(B) þ B None – pooling required – – – C(B) R þ A C(B) þ C(B) A C(B) abc’s Within levels of B A, B, A B A BRþ A C(B) þ A BA C(B) abc’s No test of interaction A, B A C(B) R þ A C(B) R Replicates No test of interaction A, B, C(B), A B (iv) A random, B random ARþ A C(B) þ A B þ AA B ab’s No further restrictions B BRþ A C(B) þ A B þ C(B) þ B None – pooling required – – – C(B) R þ A C(B) þ C(B) A C(B) abc’s Within levels of B A, B, A B A BRþ A C(B) þ A BA C(B) abc’s No test of interaction A, B A C(B) R þ A C(B) R Replicates No test of interaction A, B, C(B), A B Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008 1 .J NESNADC .F E BRAAK TER F. J. C. AND ANDERSON J. M. 112

TABLE AIV Permutation strategies for tests of particular terms in three-way analysis of variance where the experimental design involves three crossed factors. The linear model is: yijkl ¼ m þ Ai þ Bj þ Ck þ ABij þ ACik þ BCjk þ ABCijk þ eijkl. The possible designs include: (i) A, B and C are fixed, (ii) A and B are fixed and C is random, (iii) A is fixed and B and C are random, and (iv) A, B and C are random. Note that permutation of residuals refers to residuals under a reduced model (i.e., Freedman and Lane, 1983).

Terms contributing Term whose MS is used Exact test: Permutation of residuals: components of variation as the denominator Additional restrictions Terms in model to be Term in model to expected MS of term for the F-test Exchangeable units applied to removed by subtraction being tested being tested (R ¼ residual) (R ¼ residual) for the test exchangeable units of cell means

(i) A fixed, B fixed, C fixed ARþ A R Replicates Within levels of B and C B, C, A B, A C, B C, A B C BRþ B R Replicates Within levels of A and C A, C, A B, A C, B C, A B C CRþ C R Replicates Within levels of A and B A, B, A B, A C, B C, A B C A BRþ A B R Replicates No test of interaction A, B, C, A C, B C, A B C A CRþ A C R Replicates No test of interaction A, B, C, A B, B C, A B C B CRþ B C R Replicates No test of interaction A, B, C, A B, A C, A B C A B CRþ A B C R Replicates No test of interaction A, B, C, A B, A C, B C (ii) A fixed, B fixed, C random ARþ A C þ AA C ac’s Within levels of B B, C, A B, B C BRþ B C þ BB C bc’s Within levels of A A, C, A B, A C CRþ C R Replicates Within levels of A and B A, B, A B, A C, B C, A B C A BRþ A B C þ A BA B C abc’s No test of interaction A, B, C, A C, B C A CRþ A C R Replicates No test of interaction A, B, C, A B, B C, A B C B CRþ B C R Replicates No test of interaction A, B, C, A B, A C, A B C A B CRþ A B C R Replicates No test of interaction A, B, C, A B, A C, B C Downloaded By: [Universidad de Valencia] At: 15:23 4 March 2008

(iii) A fixed, B random, C random ARþ A B C þ A C þ A B þ A None – pooling required – – – BRþ B C þ BB C bc’s Within levels of A and C A, C, A B, A C CRþ B C þ CB C bc’s Within levels of A and B A, B, A B, A C A BRþ A B C þ A BA B C abc’s No test of interaction A, B, C, A C, B C A CRþ A B C þ A CA B C abc’s No test of interaction A, B, C, A B, B C B CRþ B C R Replicates No test of interaction A, B, C, A B, A C, A B C A B CRþ A B C R Replicates No test of interaction A, B, C, A B, A C, B C (iv) A random, B random, C random UT-ATRA NLSSO AINE113 VARIANCE OF ANALYSIS MULTI-FACTORIAL ARþ A B C þ A C þ A B þ A None – pooling required – – – BRþ A B C þ B C þ A B þ B None – pooling required – – – CRþ A B C þ B C þ A C þ C None – pooling required – – – A BRþ A B C þ A BA B C abc’s No test of interaction A, B, C, A C, B C A CRþ A B C þ A CA B C abc’s No test of interaction A, B, C, A B, B C B CRþ A B C þ B CA B C abc’s No test of interaction A, B, C, A B, A C A B CRþ A B C R Replicates No test of interaction A, B, C, A B, A C, B C