A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code Author(S): M

American Society for Quality

A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code Author(s): M. D. Mckay, R. J. Beckman, W. J. Conover Source: Technometrics, Vol. 42, No. 1, Special 40th Anniversary Issue (Feb., 2000), pp. 55-61 Published by: American Statistical Association and American Society for Quality Stable URL: http://www.jstor.org/stable/1271432 Accessed: 29/06/2010 10:02

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you may use content in the JSTOR archive only for your personal, non-commercial use.

Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at http://www.jstor.org/action/showPublisher?publisherCode=astata.

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed page of such transmission.

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected].

American Statistical Association and American Society for Quality are collaborating with JSTOR to digitize, preserve and extend access to Technometrics.

http://www.jstor.org A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code

M. D. MCKAYAND R. J. BECKMAN W. J. CONOVER Los Alamos ScientificLaboratory Departmentof Mathematics P.O.Box 1663 Texas Tech University Los Alamos, NM87545 Lubbock,TX 79409

Two types of sampling plans are examined as alternatives to simple random sampling in Monte Carlo studies. These plans are shown to be improvements over simple random sampling with respect to variance for a class of estimators which includes the sample mean and the empirical distribution function.

KEY WORDS: Latin hypercube sampling; Sampling techniques; Simulation techniques; Variance reduction.

1. INTRODUCTION to comparing the three methods with respect to their per- in code. Numerical methods have been used for years to provide formance an actual computer The code used in this was approximate solutions to fluid flow problems that defy ana- computer paper developed in the of the Theoretical Division lytical solutions because of their complexity. A mathemati- Hydrodynamics Group at the Los Alamos Scientific to reac- cal model is constructed to resemble the fluid flow problem, Laboratory, study tor and Romero The code is and a computer program (called a "code"), incorporating safety (Hirt 1975). computer named SOLA-PLOOP and is a one-dimensional version of methods of obtaining a numerical solution, is written. Then another code SOLA (Hirt, Nichols, and Romero 1975). The for any selection of input variables X = (X,..., XK) an code was used us to model the blowdown output variable Y = h(X) is produced by the computer by depressurization of a filled with water at fixed initial tem- code. If the code is accurate the output Y resembles what straight pipe perature and pressure. Input variables include: X1, the actual output would be if an experiment were performed phase change rate; X2, drag coefficient for drift velocity; X3, num- under the conditions X. It is often impractical or impossi- ber of bubbles per unit volume; and X4, pipe roughness. The ble to perform such an experiment. Moreover, the computer input variables are assumed to be uniformly distributedover codes are sometimes sufficiently complex so that a single given ranges. The output variable is pressure as a function set of input variables may require several hours of time on of time, where the initial time to is the time the pipe rup- the fastest computers presently in existence in order to pro- tures and depressurization initiates, and the final time tl is duce one output. We should mention that a single output 20 milliseconds later. The pressure is recorded at 0.1 milli- Y is usually a graph Y(t) of output as a function of time, second time intervals. The code was used repeatedly so that calculated at discrete time points t, to < t < tl. the accuracy and precision of the three sampling methods When real world with a modeling phenomena computer could be compared. code one is often faced with the problem of what values to use for the inputs. This difficulty can arise from within the physical process itself when system parameters are not 2. A DESCRIPTION OF THE THREE METHODS constant, but vary in some manner about nominal values. USED FOR SELECTING THE VALUES OF INPUT We model our uncertainty about the values of the inputs VARIABLES by treating them as random variables. The information de- From the many different methods of selecting the values sired from the code can be obtained from a study of the of input variables, we have chosen three that have consid- probability distribution of the output Y(t). Consequently, erable intuitive appeal. These are called random sampling, we model the "numerical" experiment by Y(t) as an un- stratified sampling, and Latin hypercube sampling. known transformation h(X) of the inputs X, which have a Random Sampling. Let the input values XI,..., XN be known probability distribution F(x) for x c S. Obviously a random sample from F(x). This method of sampling is several values of X, say XI,..., XN, must be selected as perhaps the most obvious, and an entire body of statistical successive inputs sets in order to obtain the desired infor- literature may be used in making inferences regarding the distribution of mation concerning Y(t). When N must be small because Y(t). of the running time of the code, the input variables should be selected with great care. ? 1979 American Statistical Association The next section describes three methods of selecting and the American Society for Quality (sampling) input variables. Sections 3, 4 and 5 are devoted TECHNOMETRICS,FEBRUARY 2000, VOL. 42, NO. 1

55 56 M. D. MCKAY,R. J. BECKMAN,AND W. J. CONOVER

StratifiedSampling. Using stratifiedsampling, all areas the following theorem,proved in the Appendix,relates the of the sample space of X are representedby input values. variances of TL and TR. Let the sample space S of X be partitionedinto I disjoint strata Si. Let pi = P(X E Si) representthe size of Si. Theorem. If Y = h(X1,... XK) is monotonic in each of its Obtain a random sample Xij, j = 1,..., ni from Si. Then arguments,and g(Y) is a monotonicfunction of Y, of course the ni sum to N. If I = 1, we have random then Var(TL) < Var(TR). samplingover the entire samplespace. 2.2 The SOLA-PLOOP Example Latin Hypercube Sampling. The same reasoning that led The to stratifiedsampling, ensuring that all portionsof S were three sampling plans were compared using the sampled,could lead further.If we wish to ensurealso that SOLA-PLOOPcomputer code with N = 16. Firsta random each of the input variablesXk has all portionsof its dis- sample consisting of 16 values of X = (X1, X2,X3, X4) was tributionrepresented by input values, we can divide the selected,entered as inputs,and 16 graphsof Y(t) were observedas range of each Xk into N strataof equal marginalproba- outputs.These outputvalues were used in the bility 1/N, and sample once from each stratum.Let this estimators. For the sample be Xkj, j = 1,..., N. These form the Xk compo- stratifiedsampling method the rangeof each in- variablewas divided at nent, k = 1,..., K, in Xi, i = 1,..., N. The components put the median into two parts of of the various Xk's are matchedat random.This method equalprobability. The combinationsof rangesthus formed 24 = 16 of selecting inputvalues is an extensionof quotasampling produced strataSi. One observationwas obtained (Steinberg1963), and can be viewed as a K-dimensional at randomfrom each Si as input,and the resultingoutputs extensionof Latin squaresampling (Raj 1968). were used to obtainthe estimates. To obtainthe One advantageof the Latin hypercubesample appears Latinhypercube sample the rangeof each variable when the output Y(t) is dominatedby only a few of input Xi was stratifiedinto 16 intervalsof equal the componentsof X. This method ensures that each of probability,and one observationwas drawnat randomfrom each those componentsis representedin a fully stratifiedman- interval.These 16 valuesfor the 4 inputvariables were ner, no matter which componentsmight turn out to be matchedat randomto form 16 inputs,and thus 16 outputs important. from the code. The entire We mentionhere that the N intervalson the rangeof each process of samplingand estimatingfor the componentof X combine to form NK cells which cover three selection methodswas repeated50 times in orderto some idea of the sample space of X. These cells, which are labeled by get the accuraciesand precisions involved. The coordinatescorresponding to the intervals,are used when total computertime spent in runningthe SOLA-PLOOP findingthe propertiesof the samplingplan. code in this study was 7 hours on a CDC-6600. Some of the standarddeviation plots appearto be inconsistentwith 2.1 Estimators the theoreticalresults. These occasionaldiscrepancies are In the Appendix(Section 8), stratifiedsampling and Latin believedto arisefrom the non-independenceof the estima- hypercubesampling are examinedand compared to random tors over time and the small samplesizes. samplingwith respectto the class of estimatorsof the form 3. ESTIMATINGTHE N MEAN The of an unbiased YN) = (1/N) goodness estimatorof the mean can T(Y1,..., g(Yi), be measured i=l by the size of its variance.For each sampling method,the estimatorof is where g(.) = arbitraryfunction. E(Y(t)) of the form If g(Y) = Y then T representsthe sample mean which is N used to estimate E(Y). If g(Y) = yr we obtain the rth Y(t) = (l/N) E Yi(t) (3.1) sample moment. By letting g(Y) = 1 for Y < y, 0 other- i=l wise, we obtainthe usual empiricaldistribution function at where the point y. Ourinterest is centeredaround these particular statistics. Yi(t) = h(X), i=1,...,N. Let T denote the expectedvalue of T when the Yt's constitutea random from sample the distributionof Y = h(X). In the case of the stratified the comes from We show in the sample, Xi Appendix that both stratifiedsampling stratum Si, pi = and = 1. For the Latin and Latin 1/N ni hypercube hypercubesampling yield unbiasedestimators sample,the Xi is obtainedin the manner of r. describedearlier. Each of the three estimators YR,Ys, and YL is an unbiased If is the T TR estimateof from a randomsample of size estimatorof The variancesof the and is E(Y(t)). estimatorsare N, Ts the estimatefrom a stratifiedsample of size given in (3.2): N, then Var(Ts) < Var(TR)when the stratifiedplan uses equal probabilitystrata with one sample per stratum(all Var(Y(t)) = = = (1/N)Var(Y(t)) pi 1/N and nij 1). No direct means of comparing the varianceof the correspondingestimator from Latin N hyper- Var(Ys(t)) = - - cube sampling,TL, to Var(Ts) has been found. However, Var(YR(t)) (1/N2) (pi ,)2 i=l TECHNOMETRICS,FEBRUARY 2000, VOL. 42, NO. 1 A COMPARISON OF THREE METHODS FOR SELECTING VALUES OF INPUT 57

140-0 - 150-0

ANDOM ...... RANDOM CY 120-0 - 0O STRATI ...... 0 STRATIFIED I- I- LAT -- 100-0 LATIN 2 2 100*0- V) V)(n LJ6 LA. La. 0 80.0 - 50-0 zLLJ z < w LLJ 60-0 - ~E2

I I

AI.v i"-(n -V A -. 0-0 5-0 1I0 15.01-020 20-0 TIME TIME

Figure 1. Estimating the Mean: The Sample Mean of YR(t), Ys(t), Figure 3. Estimating the Variance: The Sample Mean of S2 (t), S (t), and YL(t). and S2 (t).

= + - Var(YL(t)) Var(YR(t)) ((N 1)/N) YL(t) clearly demonstrates superiority as an estimator in this example, with a standarddeviation roughly one-fo[u]rth 1/(NK(N- (i - - (3.2) I)K)) E )(tj t) that of the random sampling estimator. R

where p = E(Y(t)), 4. ESTIMATINGTHE VARIANCE For each the form of the estimator of = E in the stratified or sampling method, pi E(Y(t)lX Si) sample, the variance is Pi - E(Y(t)lX e cell i) in the Latin hypercube N sample, S2(t) - (1/N)Y (Y(t)- Y(t))2, (4.1) i=l and R means the restricted of all ,ui, space pairs /j having and its expectation is no cell coordinates in common. For the SOLA-PLOOP computer code the means and E(S2(t)) Var(Y(t)) - Var(Y(t)), (4.2) standard deviations, based on 50 observations, were com- where is one of or puted for the estimators just described. Comparative plots Y(t) YR(t),Ys(t), YL(t). In the case of the random it is well known that of the means are given in Figure 1. All of the plots of the sample, NS2/(N- 1) is an unbiased estimator of the variance of means are comparable, demonstrating the unbiasedness of the estimators. Y(t). The bias in the case of the stratified sample is unknown. because < Comparative plots of the standard deviations of the es- However, Var(Ys(t)) Var(YR(t)), timators are in 2. The standard deviation of given Figure (1- 1/N)Var(Y(t)) < E(S2(t)) < Var(Y(t)). (4.3) Ys(t) is smaller than that of YR(t) as expected. However,

- 2-5 - 50'0

RANDOM . i'' RANDOM - 40-0 - 20 ii STRATIFIED or STRATFIED ------I: 0O I 1' I- LATIN LATIN 1 :, I- 30-0 - 2 1-5 - 1 I- V) 1 ' (C) 1 ' Lj t1 '

: '' LA- 20-0 - La. 1.0 - :, S i 0 0 .@: ,; O ?f\? .,i/ :t . c5 0 u' 10-0 - 0-5 - I I ,., ' -1. n.n "r,a -I a '~.....~- --t---"-- ~....._.._ ' - ....~` 0'0 ..- /- 10-0 15.015-0 20 15- 0.0 5.0 20.0 0.0 5-0 10-0 15.0 20-0 TIME TIME

Figure 4. Estimating the Variance: The Standard Deviation of S2(t), Figure 2. Estimating the Mean: The Standard Deviation of YR(t), and Ys (t), and YL(t). Ss(t), S2(t).

TECHNOMETRICS,FEBRUARY 2000, VOL. 42, NO. 1 :

58 M. D. MCKAY,R. J. BECKMAN,AND W. J. CONOVER

1.0 - o- 2.1, the expectedvalue of G(y, t) underthe three sampling plansis the same,and under random sampling, the expected RANDOM value of G(y, t) is D(y, t). cr 0.8 - 0 STRATIFID ...... - I The variancesof the three estimatorsare given in (5.2).

I-4 LATIN -- Di againrefers to eitherstratum i or cell i, as appropriate, and R the samerestricted as it did in (3.2). 2 0*6 - represents space V) L&J / Var(GR(y, t)) = (1/N)D(y, t)(l - D(y, t)) La. / 0 0-4 - / Il z Var(Gs(y, t)) = Var(GR(y, t)) LLJ N :2 0-2 - // - ...... /"" (1/N2) (D (y, t)- D(y, t))2 "_' t=l 0'0 lm . . .. - k 20-0 30-0 40-0 50-0 60-0 70-0 80.0I80 990'0 PRESSURE Var(GL(y, t)) = Var(GR(y, t))

Figure 5. the CDF: The Mean of Estimating Sample GR(Y, t), Gs(y, + ((N - 1)/N. 1/NK(N - 1)K) E (Di(y,t) t), and GL(y, t) at t = 1.4. R The bias in the Latin is also unknown,but hypercubeplan - D(y, t)). (Dj(y, t) - D(y, t)). (5.2) for the SOLA-PLOOPexample it was small. Variancesfor these estimatorswere not found. As with the cases of the mean and varianceestimators, Againusing the SOLA-PLOOPexample, means and stan- the distributionfunction estimators were comparedfor the darddeviations (based on 50 observations)were computed. three samplingplans. Figures 5 and 6 give the means and The mean plots are given in Figure 3. They indicatethat standarddeviations of the estimatorsat t = 1.4 ms. This all three estimatorsare in relative agreementconcerning time point was chosen to correspondto the time of max- the quantitiesthey are estimating.In terms of standardde- imum variancein the distributionof Y(t). Again the esti- viations of the estimators,Figure 4 shows that, although mates obtainedfrom a Latin hypercubesample appearto stratifiedsampling yields aboutthe same precisionas does be more precise in generalthan the other two types of es- randomsampling, Latin hypercube furnishes a clearly bet- timates. ter estimator. 6. DISCUSSION AND CONCLUSIONS 5. ESTIMATINGTHE DISTRIBUTIONFUNCTION We have presentedthree samplingplans and associated estimatorsof the mean,the variance,and the populationdis- The distribution function, D(y,t), of = Y(t) h(X) may tributionfunction of the outputof a computercode when be estimatedby the distributionfunction. The em- empirical the inputsare treatedas randomvariables. The firstmethod piricaldistribution function can be writtenas is simple randomsampling. The second method involves N stratifiedsampling and improves upon the firstmethod. The G(y, t) = (1/N) u(y- Yi(t)), (5.1) thirdmethod is called here Latinhypercube sampling. It is i=-i an extensionof quota sampling(Steinberg 1963), and it is where u(z) = 1 for z > 0 and is zero otherwise. Since a first cousin to the "randombalance" design discussedby equation(5.1) is of the form of the estimatorsin Section Satterthwaite(1959), Budne (1959), Youdenet al. (1959), Anscombe(1959), andto the fractionalizedfactorial 015 - highly designs discussed by Enrenfeldand Zacks (1951, 1967), . ~~~~~~~RANDOMWY~~~~~~ Dempster (1960, 1961), and Zacks (1963, 1964), and to lattice samplingas discussedby Jessen (1975). This third 0o .T5RA D ...... ---- methodimproves upon simple randomsampling when cer- o0-0- LAW -- / < tain monotonicityconditions hold, and it appearsto be a .....*: I-_ :- good methodto use for selectingvalues of inputvariables. vG(:,a:y,-, t\att=4 7. Eo , . .: . ACKNOWLEDGMENTS We extenda thanksto RonaldK. for 0. ^ ../:-2'' ^ *\ \ special Lohrding, his earlysuggestions related to this workand for his continuing supportand encouragement.We also thankour colleagues LarryBruckner, Ben Duran,C. Phive,and Tom Boullion for 0-00 /J * I I I I theirdiscussions concerning various aspects of the problem, 20'0 30-0 4O?0 5-0s 0'0 700 0-o0 90-0 and Dave Whitemanfor assistancewith the computer. PRESSURE This paperwas preparedunder the supportof the Anal- Figure 6. Estimating the CDF: The Standard Deviation of GR(y, t), ysis DevelopmentBranch, Division of ReactorSafety Re- Gs(y, t), and GL(y, t) at t = 1.4. search,Nuclear Regulatory Commission. TECHNOMETRICS,FEBRUARY 2000, VOL. 42, NO. 1 A COMPARISONOF THREE METHODS FOR SELECTING VALUES OF INPUT 59

8. APPENDIX I ni In the sections that follow we presentsome generalre- Ts = (pi/ni) E g(Yij), sults about stratifiedsampling and Latin hypercubesam- i=l j=l in order to make with pling comparisons simple random that Ts is an unbiased estimator of r with variance given by sampling.We move from the generalcase of stratifiedsampling to stratified with allocation, sampling proportional = and then to proportionalallocations with one observation Var(Ts) (p2/ni)o2. (8.2) i=l per stratum.We examineLatin hypercube sampling for the equal marginalprobability strata case only. The following resultscan be found in Tocher(1963). 8.1 Type I Estimators Stratified Sampling with Proportional Allocation. If the of the strata and the Let X denotea K variaterandom variable with probabilitysizes, pi, sample sizes, probabil- ni, are chosen so that ni = allocation function for x S. Let Y denote a piN, proportional ity density (pdf) f(x) E is achieved.In this case (8.2) becomes univariatetransformation of X given by Y = h(X). In the context of this paperwe assume I Var(Ts) = Var(TR) - (I/N) EPi(iii - r)2. (8.3) X f(x),xeS KNOWNpdf i=l Y = h(X) UNKNOWNbut observable Thus, we see that stratifiedsampling with proportionalal- transformationof X. locationoffers an improvementover randomsampling, and that the variancereduction is a functionof the differences The class of estimatorsto be consideredare those of the betweenthe stratameans and the overallmean r. form ,i Proportional Allocation with One Sample per Stratum. N Any stratifiedplan which employs subsampling,ni > 1, T(Ul,..., iUN)= -(l/N) Eg (ui), (8.1) can be improvedby furtherstratification. When all ni = 1, t=l (8.3) becomes where g(.) is an arbitrary,known function. In particular N we use = ur to estimate = g(u) moments,and g(u) 1 for Var(Ts) = Var(TR) - (1/N2) E (i - r)2. (8.4) u > 0, = 0 elsewhere,to estimatethe distributionfunction. i=l The sampling schemes describedin the following sections will be comparedto randomsampling with respect 8.3 Latin Hypercube Sampling to T. The symbol TR denotes T(Y1,..., YN) when the ar- In stratifiedsampling the range space S of X can be gumentsY, ..., YN constitutea randomsample of Y. The arbitrarilypartitioned to form strata.In Latin hypercube mean and varianceof TR are denotedby r and 02/N. The samplingthe partitionsare constructedin a specificmanner statistic T given by (8.1) will be evaluatedat arguments using partitionsof the rangesof each componentof X. We arising from stratifiedsampling to form Ts, and at argu- will only considerthe case wherethe componentsof X are ments arisingfrom Latin hypercubesampling to form TL. independent. The associatedmeans and varianceswill be comparedto Let the ranges of each of the K componentsof X be those for randomsampling. partitionedinto N intervalsof probabilitysize 1/N. The Cartesianproduct of these intervals S into NK 8.2 Stratified partitions Sampling cells each of probabilitysize N-K. Eachcell can be labeled Let the of X be rangespace, S, partitionedinto I disjoint by a set of K cell coordinates mi = (mil, i2,..., iK) subsets of size = c with Si pi P(X Si), where mij is the intervalnumber of componentXj representedin cell A I i. Latinhypercube sample of size N is obtained from a random selection N of the cells ml,..., mN, 5Pi 1. with the conditionthat for each the i=l j set {mij }N is a per- mutationof the integers1,..., N. One randomobservation Let Xij,j = 1,., ni, be a randomsample from stratum is made in each cell. The densityfunction of X given X c = Si. That is, let Xij iid f(x)/pi,j 1,..., ni, for x E Si, cell i is NKf(x) if x E cell i, zero otherwise. The marginal but with zero density elsewhere.The correspondingvalues (unconditional)distribution of Yi(t) is easily seen to be the of Y are denotedby Yij = h(Xij), andthe stratameans and same as that for a randomlydrawn X as follows: variancesof g(Y) are denotedby = P(Y < y) P(Yi < ylX E cell q)P(X c cell q) i = E(g(Yij))- j g(y)(l/pi)f(x) dx all cells q Si

= E ll N K(x)dx(1/NK) a-2 - Var(g(Yj)) = (g(y) - h(x)

TECHNOMETRICS,FEBRUARY 2000, VOL. 42, NO. 1 60 M. D. MCKAY,R. J. BECKMAN,AND W. J. CONOVER

From this we have TL as an unbiased estimator of T. Furthermore To arrive at a form for the variance of TL we introduce indicator variables wt, with NK NK Cov(wig(Yi),wj g(Yj)) f 1 if cell i is in the E Z sample i=1 j=1 Wi- l if not. 0 i#j The estimator can now be written as - - EZ E ijE{wiwj} N-2K+2 E ipj (8.11) NK Z i#j i?j TL = (1/N) z wig(Yi), (8.5) i=l1 which combines with (8.10) to give where Yi = h(Xi) and Xi c cell i. The variance of TL is given by Var(TL) = (1/N)Var(Y) - N-K-1 (t _ )2 i NK = 2 Var(TL) (1/N2) Var(wig(Yi)) + (N-K-1 N-2NK)- i=l

NK NK + (N - 1)-K+1NK-1 + (1/N2) 5 Cov(wig(Yi), Wjg(Yj)). (8.6) R i=l j=l -2K -N -^ E ij (8.12) jii EE^.~~., The following results about the wi are immediate:

1. P (wi = 1) = (1/NK-1) = E(wi) = E(w2) where R means the restricted space of NK(N - 1)K pairs Var(wi) (1/NK-1)(1- 1INK-1). ([i, ,j) corresponding to cells having no cell coordinates in 2. If wi and wj correspond to cells having no cell coor- common. After some algebra, and with ui = NKT,K the dinates in common, then final form for Var(TL) becomes E(wiwj) = E(wiw lwwj= O)P(wj = 0) Var(TL) = Var(TR) + (N - 1)/N[N-K(N - 1)-K + E(wiwjlwj = 1)P(wj = 1) - - (8.13) = 1/(N(N- *E (pi T)(pu T)]. 1))K-1 R 3. If wi and wj correspond to cells having at least one common cell coordinate, then Note that Var(TL) < Var(TR) if and only if

E(iwjw) =0. N -K(N - 1)-K (/i - )(1jt - T) < 0, (8.14) Now R = Var(wig(Yi)) E(w2)Var g(Yi) + E2(g(Yi))Var(wt) (8.7) which is equivalent to saying that the covariance between so that cells having no cell coordinates in common is negative. A sufficient condition for (8.14) to hold is given by the fol- NK NK lowing theorem. E Var(wig(Yi)) N-K+l E E(g(Yi) i)2 i=l i=l Theorem. If Y = h(X1,..., XK) is monotonic in each NK of its arguments, and if g(Y) is a monotonic function of Y, then < + (N-K+I1 N-2K+2) E 2 (8.8) Var(TL) Var(TR). ='-1 Proof The proof employs a theorem by Lehmann where ui = e cell i}. Since E{g(Y))X (1966). Two functions r(x1,..., XK) and s(y1,..., YK) are said to be concordant in each if r and s either E(g(Y)- i)2 argument increase or decrease together as a function of xi - yi, with all xj,j 7 i and yj,j - i held fixed, for each i. Also, -- NK (g(y) - + - 7)2f(x) dx (i )2 (8.9) a of random wcelli pair variables (X, Y) are said to be negatively quadrant dependent if P(X < x, Y < < < we have y) P(X x)P(Y < y) for all x,y. Lehmann's theorem states that if 5 Var(wig(Yi)) N Var(Y)- N-K+1 E (i _ T)2 (i) (X1, Y1), (X2, Y2),... (XK, YK) are independent, (ii) i i (Xi, Y,) is negatively quadrantdependent for all i, and (iii) X = r(X1,...,XK) and Y = s(Y1,...,YK) are concor- in each + (N-K+1 _ N-2K+2) E 2. (8.10) dant argument, then (X, Y) is negatively quadrant dependent.

TECHNOMETRICS,FEBRUARY 2000, VOL. 42, NO. 1 A COMPARISONOF THREE METHODS FOR SELECTING VALUES OF INPUT 61

We earlier describeda stage-wise process for selecting [Received January 1977. Revised May 1978.] cells for a Latinhypercube sample, where a cell was labeled by cell coordinates mi,..., miK. Two cells (I1,..., IK) and (ml,..., mK) with no coordinates in common may be se- REFERENCES lected as follows. Randomlyselect two integers(R11, R21) withoutreplacement from the firstN integers1,..., N. Let 11 = R11 and m1 = R21. Repeat the procedure to obtain Anscombe,F. J. (1959), "QuickAnalysis Methodsfor RandomBalance (R12,R22), (R13,R23), . .,(R1K, R2K) and let lk = RIk ScreeningExperiments," Technometrics, 1, 195-209. T. and mk = R2k. Thus two cells are selected and Budne, A. (1959),"The Application of RandomBalance Designs, Tech- randomly nometrics, 1, 139-155. lk 7 mk for k = K. 1,..., Dempster,A. P. (1960),"Random Allocation I: OnGeneral Classes Note = Designs that the pairs (Rlk, R2k), k 1,..., K, are mutually of Estimation Methods," The Annals of Mathematical Statistics, 31, 885- independent. Also note that because P(Rlk < x, R2k < 905. y) = [xy - min(x,y)]/(n(n - 1)) < P(Rlk < x)P(R2k < (1961), "RandomAllocation Designs II: ApproximateTheory for where the Simple Random Allocation," The Annals of Mathematical Statistics, 32, y), [.] represents "greatestinteger" function, each 387-405. pair (Rlk, R2k) is negatively quadrant dependent. Ehrenfeld,S., andZacks, S. (1951),"Randomization and Factorial Exper- Let /ul be the expected value of g(Y) within the cell iments," The Annals of Mathematical Statistics, 32, 270-297. designated by (I1,..., 1K), and let /2 be similarly defined (1967), "TestingHypotheses in RandomizedFactorial Experi- ments," The Annals Mathematical for (ml,... ,mK). Then /1 = ii(R11,R12,... ,R1K) and of Statistics, 38, 1494-1507. Hirt,C. W.,Nichols, B. D., and N. C. "SOLA-A A12 are concordant in each Romero, (1975), Numeri- -= I(R21, R22, ..., R2K) argu- cal Solution for TransientFluid ment under the of the theorem. Algorithm Flows,"Scientific Laboratory assumptions Lehmann's ReportLA-5852, Los AlamosNational Laboratory, NM. theorem then yields that 1i and /2 are negatively quadrant Hirt,C. W.,and Romero, N. C. (1975),"Application of a Drift-FluxModel dependent.Therefore, to Flashingin StraightPipes," Scientific Laboratory Report LA-6005- MS, Los AlamosNational Laboratory, NM. P(PI1 < X, l2 < y) < P(li1 < x)P(i2 < y). Jessen,Raymond J. (1975),"Square and Cubic Lattice Sampling," Biomet- rics, 31, 449-471. Using Hoeffding'sequation Lehmann,E. L. (1966), "SomeConcepts of Dependence,"The Annals of Mathematical Statistics, 35, 1137-1153. 1+oo r+00 Raj,Des. (1968),Sampling Theory, New York:McGraw-Hill. = < Y < Cov(X, Y) [P(X x, y) Satterthwaite,F. E. (1959), "RandomBalance Experimentation," Techno- metrics, 1, 111-137. H. A. (1963), "Generalized - P(X < x)P(Y < y)] dx dy, Steinberg, QuotaSampling," Nuclear Science and Engineering, 15, 142-145. Tocher,K. D. (1963),The Art NJ:Van (see Lehmann(1966) for a proof),we haveCov(/Al, /2) < 0. of Simulation,Princeton, Nostrand. Since = + - Youden,W. J., Kempthorne,O., Tukey,J. W., Box, G. E. P., and Hunter, Var(TL) Var(TR) (N 1)/N Cov(i,Lu2), the J. S. theoremis (1959), Discussion of the papers of Messrs. Satterthwaiteand proved. Budne, Technometrics, 1, 157-193. Since g(t) as used in both Sections 3 and 5 is an increas- Zacks, S. (1963), "Ona CompleteClass of LinearUnbiased Estimators ing functionof t, we can say that if Y = h(X) is a mono- for RandomizedFactorial Experiments," The Annals of Mathematical 769-779. tonic function of each of its arguments,Latin hypercube Statistics, 34, (1964), "GeneralizedLeast Estimatorsfor samplingis betterthan random sampling for the Squares Randomized estimating Fractional Replication Designs," The Annals of Mathematical Statistics, mean and the populationdistribution function. 35, 696-704.

TECHNOMETRICS,FEBRUARY 2000, VOL. 42, NO. 1