Reliability Engineering and System Safety 165 (2017) 376–394

Contents lists available at ScienceDirect

Reliability Engineering and System Safety

journal homepage: www.elsevier.com/locate/ress

ffi E cient Monte Carlo methods for estimating failure probabilities MARK ⁎ Andres Albana, Hardik A. Darjia,b, Atsuki Imamurac, Marvin K. Nakayamac, a Mathematical Sciences Dept., New Jersey Institute of Technology, Newark, NJ 07102, USA b Mechanical Engineering Dept., New Jersey Institute of Technology, Newark, NJ 07102, USA c Computer Science Dept., New Jersey Institute of Technology, Newark, NJ 07102, USA

ARTICLE INFO ABSTRACT

Keywords: We develop efficient Monte Carlo methods for estimating the failure probability of a system. An example of the Probabilistic safety assessment problem comes from an approach for probabilistic safety assessment of nuclear power plants known as risk- Risk analysis informed safety-margin characterization, but it also arises in other contexts, e.g., structural reliability, Structural reliability catastrophe modeling, and finance. We estimate the failure probability using different combinations of Uncertainty simulation methodologies, including stratified sampling (SS), (replicated) Latin hypercube sampling (LHS), Monte Carlo and conditional Monte Carlo (CMC). We prove theorems establishing that the combination SS+LHS (resp., SS Variance reduction Confidence intervals +CMC+LHS) has smaller asymptotic variance than SS (resp., SS+LHS). We also devise asymptotically valid (as Nuclear regulation the overall sample size grows large) upper confidence bounds for the failure probability for the methods Risk-informed safety-margin characterization considered. The confidence bounds may be employed to perform an asymptotically valid probabilistic safety assessment. We present numerical results demonstrating that the combination SS+CMC+LHS can result in substantial variance reductions compared to stratified sampling alone.

1. Introduction degradations in safety margins previously deemed acceptable, and RISMC aims to better understand their impacts. Consider a stochastic model of the behavior of a system (e.g., a Current NPP PSAs compare a random load to a fixed capacity. For structure, such as a building or ship). The system's uncertainties may example, the U.S. Nuclear Regulatory Commission (NRC) [4, para- include random loads and capacities, environmental conditions, mate- graph 50.46(b)(1)], specifies that the 0.95-quantile of the peak clad- rial properties, etc. The system fails under specified conditions—e.g., a ding temperature (PCT) during a hypothesized loss-of-coolant accident critical subset of components fails—and the goal is to determine if the (LOCA) must not exceed 2200° F. The NRC permits a plant licensee to failure probability θ is acceptably small. As the system's complexity demonstrate its facility's compliance with federal regulations using a renders computing θ as intractable, we instead apply Monte Carlo 95/95 analysis with Monte Carlo simulation; see Section 24.9 of [5] simulation to estimate θ. To account for the sampling error of the and [6]. This entails running a computer code [7] modeling the simulation estimates, a confidence interval for θ is also needed. postulated event multiple times with randomly generated inputs to Running the simulation model may be expensive, so we want to reduce explore a range of uncertainties [8], using the code's outputs to the sampling error. In this paper, we combine different variance- construct a 95% upper confidence bound (UCB) for the 0.95-quantile reduction techniques (VRTs) to estimate θ. of the PCT, and verifying that the UCB lies below the fixed threshold A motivating example comes from a framework for probabilistic 2200° F. The UCB accounts for the statistical error of the Monte Carlo safety assessments (PSAs) of nuclear power plants (NPPs) known as estimate due to sampling variability, and the difference between the risk-informed safety-margin characterization (RISMC), which was fixed threshold and the UCB provides a type of safety margin. proposed by an international effort of the Nuclear Energy Agency Several papers have developed Monte Carlo methods for perform- Committee on the Safety of Nuclear Installations [1]. The purpose of ing a 95/95 analysis. For example, [9] and [10] apply an approach of RISMC is to address recent changes in NPPs. For example, many NPPs [11] based on order statistics, which is valid when employing simple in the current U.S. fleet are aging past their original 40-year operating random sampling (SRS). However, SRS can produce unusably noisy licenses, with owners applying for lengthy extensions [2]. Also, for estimates of the 0.95-quantile, so [12] incorporates VRTs, including economic reasons, plants are sometimes run at higher output levels, antithetic variates (AV) and Latin hypercube sampling (LHS), to obtain known as power uprates [3]. These and other factors can lead to more statistically efficient quantile estimators; see Chapter 4 of [13]

⁎ Corresponding author. E-mail addresses: [email protected] (A. Alban), [email protected] (H.A. Darji), [email protected] (A. Imamura), [email protected] (M.K. Nakayama). http://dx.doi.org/10.1016/j.ress.2017.04.001 Received 25 July 2016; Received in revised form 16 March 2017; Accepted 11 April 2017 Available online 18 April 2017 0951-8320/ © 2017 Elsevier Ltd. All rights reserved. A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394 and Chapter V of [14] for overviews of these and other VRTs for by an additive function of the input random variables. As we are estimating a mean. In addition, [12] provides a UCB for the quantile estimating a probability, the response without CMC is an indicator, using a finite-difference approach of [15] and [16]. Utilizing VRTs in which has a poor additive approximation. Thus, LHS by itself may not nuclear PSAs is especially important because each simulation run may reduce variance much. In contrast, the conditioning of CMC produces a be computationally expensive, e.g., it may require numerically solving “smoother” response (in general, no longer binary-valued) with a better systems of differential equations. additive fit. Hence, LHS can yield much smaller variance when In addition to the condition on the PCT, the NRC further imposes combined with CMC; [23] observes similar synergies between CMC requirements on other criteria—the core-wide oxidation (CWO<1%) and LHS. and maximum local oxidation (MLO<17%)—for a PSA of a hypothe- In addition to [3] and [19], other related work includes [24], which sized NPP LOCA; see paragraph 50.46(b) of [4]. (The NRC is currently provides UCBs for the single-criterion (i.e., q=1) failure probability considering replacing MLO with another criterion, the integral time at when applying combinations of SS, CMC, and rLHS, but the paper does temperature [17].) The papers [9] and [10] describe approaches to not give proofs of the UCBs' asymptotic validity nor for the reduction in extend the 95/95 analysis based on an SRS quantile estimator for a asymptotic variance. The paper [25] estimates the single-criterion single criterion to handle multiple criteria with fixed capacities. failure probability using SS and CMC, but leaves out LHS. Also, [23] RISMC differs from a 95/95 analysis in several important ways. provides a theoretical analysis of integrated VRTs (CMC, control First, RISMC assumes each criterion's capacity is random rather than variates, and correlation-induction schemes, including LHS and AV) fixed. Moreover, instead of examining a quantile, as in a 95/95 study, in a general setting, but it does not include SS, which plays a key role in RISMC measures a safety margin through the failure probability θ that the RISMC framework, nor does the paper provide UCBs, as we do any random load exceeds its random capacity, where θ should be here. The paper [26] combines CMC with AV for estimating the failure smaller than a given threshold θ0, which may be specified by a probability in structural reliability, but it does not incorporate SS. As regulator. RISMC also decomposes a postulated event into scenarios explained in [23], AV and LHS can be viewed as special cases of via an event tree [18], as in a probabilistic risk analysis (PRA), with correlation-induction methods, but AV is often outclassed by LHS, uncertainties in each scenario, and Monte Carlo is employed to especially in combination with CMC. estimate each scenario's failure probability. To account for the statis- The methodologies developed in our current paper not only apply tical sampling error of the Monte Carlo estimate of the failure for nuclear PSAs, but also can be employed in a wide spectrum of other probability θ, one should further provide a UCB (or a two-sided fields. Civil engineers need to compute the failure probability θ of a confidence interval) for θ, and check if the UCB lies below θ0. The structure (e.g., a building, bridge, or dam) [20]. Insurance companies pilot RISMC studies in [3] and [19] consider only a single criterion, work with catastrophe models to determine the likelihood of infra- PCT, and assume its capacity follows a triangular distribution. These structure failures when subjected to hurricanes, floods, and earth- papers apply a combination of stratified sampling (SS) and LHS, but quakes [27]. The Basel Accords [28] specify capital requirements (i.e., they do not describe how to build a UCB for θ. (Other issues capacities) for financial institutions to ensure that they can absorb investigated in [3] and [19] include exploring the impact on θ from reasonable losses (i.e., loads). Other examples arise in the assessment altering the distributions of input random variables and from opera- of safety and reliability of nuclear weapons [29–31] and the disposal of tional changes, e.g., power uprates. A RISMC evaluation may further radioactive waste [32–34]. want to determine the core-damage frequency κ, which can be Compared to SRS, LHS tries to sample more evenly from the estimated by multiplying an estimator of θ with the (known) frequency sample space, which can produce less-variable estimators of measures of the postulated event.) of central tendency, e.g., the mean. Combining LHS with SS and CMC Our paper devises Monte Carlo methods to analyze a broad class of can lead to further substantial improvements, as our numerical results systems (not only for RISMC), with uncertainties encapsulated in a show, but perhaps not enough when estimating rare-event probabil- basic random object. The basic random object can be a random vector, ities, smaller than say 10−4. In such cases, other Monte Carlo methods as in structural reliability, where its entries are called basic variables may be more effective. In [35] and [36], the authors develop parametric [20, Section 1.5], which may be dependent and can represent, e.g., approximations to the failure probability in terms of less-rare events, random loads, environmental factors, and material properties. But the which are easier to estimate accurately via Monte Carlo. (In contrast, basic random object may be more general, e.g., a stochastic process, as rather than devising an approximation, the methods in our paper work in time-dependent reliability [20, Chapter 6]. For example, the load with the exact model.) The paper [37] employs the method of [35] to and capacity on a system may vary randomly over time, which is estimate the failure probability of a ship hull girder, modeled as a series modeled as a stochastic process. We also specify system failure in a system. In [38] the authors consider a method they call subset general way, as a given binary-valued function of the basic random simulation, also known as splitting [14, Section VI.9]; the idea is to object. An example is a series system, where the basic random object is contain the rare failure event in a sequence of successively larger, less- a random vector of the loads and capacities for a fixed number q of rare events, and the failure probability is estimated as a product of (dependent) criteria, and the system fails when any criterion's load estimates of conditional probabilities based on successive events. In exceeds its capacity, but our framework allows for many other [39] the author combines a type of CMC called directional simulation possibilities. with importance sampling. We consider applying combinations of SS, LHS, and conditional To summarize, the main contributions of the current paper are as Monte Carlo (CMC) to estimate θ. We formally prove that SS+LHS follows: (resp., SS+CMC+LHS) has smaller asymptotic variance than SS (resp., SS+LHS). We also use replicated LHS (rLHS) [21] to construct UCBs • We develop combinations of SS, CMC, and LHS to efficiently for θ, and we prove the UCBs' asymptotic validity (as the total sample estimate a failure probability θ. Our framework allows for a failure size grows large, with the number of replicates fixed). (Although we to be defined quite generally as a function of a basic random object, focus on providing UCBs, our methods can be easily modified to which may be a random vector or something more general, e.g., a produce a lower confidence bound or two-sided confidence interval.) stochastic process or random field. The combination of the three The combination of SS, CMC, and LHS can be especially effective in VRTs can produce substantial variance reduction, as our numerical reducing variance, as we show through numerical experiments. results in Section 7 indicate. We also establish theory (Theorems 1 We also give insight into why SS+CMC+LHS can work so well. As and 3) and provide insight (Sections 5.1 and 6.1) into why this is so. shown in [22, Section 10.3], LHS substantially reduces variance when • When applying SS+rLHS or SS+CMC+rLHS, we derive UCBs for θ, the response whose expectation we are estimating is well approximated which are crucial to account for the statistical error arising from

377 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394

sampling variability of the Monte Carlo estimators. We formally the failure indicator 0 ≡()=d X 1.LetH be the joint cumulative prove the UCBs' asymptotic validity; see Theorems 2 and 4. distribution function (CDF) of X,denotedbyX ∼ H,so 2 H()=x PL ( ≤ x12 , C ≤ x )for a constant x =(xx12 , )∈R .(HereL and C The remaining sections unfold as follows. Section 2 develops the may be dependent, but later in Section 6 we will assume they are problem's mathematical framework in a general setting, with later independent in this example.) The failure probability is then sections adding extra structure as needed. Section 2.1 specializes the θE= [0 ]= PLC ( ≥ )=∫ d (xx )d H ( ),anintegralofdimension2q = 2, setup to define a system failure in terms of the loads and capacities of a where E denotes the expectationoperatorinducedbyH.Nowassume

fixed number of criteria, although later sections mostly do not require that H is a mixture (e.g., Section 4.9 of [22])ofs0 = 4 bivariate normal fi λ this additional assumption. Section 3 reviews the use of simple random CDFs. Speci cally, let s, 1≤ss ≤ 0,beknown positive constants summing θ sampling to estimate and produce a UCB for , and Section 4 does the to 1. For each s,letH(s) be the joint CDF of 52(,μs Σs), a bivariate normal fi 2×2 same for strati ed sampling. We combine SS with (replicated) Latin with mean vector μsLsCs=(μμ,, , ) and covariance matrix Σs ∈ R having hypercube sampling in Section 5, and further incorporate conditional 2 2 diagonal entries σLs, and σCs, , the marginal variances of L and C, Monte Carlo in Section 6. In particular, Section 6.3 derives the s0 2 respectively. Then H()=xx∑ λHss()( ) for each x ∈ R ;i.e.,X ∼ H(s) combination of VRTs for the special setting of Section 2.1 (failure s=1 with probability λs. Let F (resp., G)bethemarginalCDFofL (resp., C); e.g., defined in terms of loads and capacities), exploiting the added s00s 2 2 s02 G is the CDF of 5( ∑s=1 λμs Cs, ,(∑s=1 λsCs(+))−( σ, μCs, ∑s=1 λμs Cs, ) ), structure. Section 7 presents numerical results from artificial RISMC where 5(,ab2) is a normal random variable with mean a and variance b2. experiments, and we give concluding remarks in Section 8. Appendices For the general mathematical framework, consider a stochastic contain most of the proofs of the theorems. A (much shorter) model of the behavior of a system (e.g., a structure, such as a building conference paper [40] outlines some of the material from the current or nuclear power plant) over a particular time period, possibly random. paper restricted to a series system, omitting Theorems 1, 3, and 5 and The system has uncertainties, e.g., loads and capacities of the system's all of the proofs; moreover, the present paper contains additional components, environmental factors, material properties, etc., which numerical results. Throughout the paper, all vectors are column may be observed at multiple times. We encapsulate all of the system's vectors, although we often write them as row vectors to save space. (aleatory and epistemic) uncertainties over the specified time horizon in a basic random object X, which may be a random vector (Example 1 List of Acronyms and Abbreviations has X =(LC , )∈R2) or a more general random object, such as a stochastic process or random field. Specifically, let (,ΩP- , ) be the AHW average half-width AV antithetic variates underlying probability space, where Ω is a sample space, - is a σ-field CDF cumulative CLT central limit theorem of subsets of Ω, and P is a probability measure on - . We assume that distribution function the basic random object X takes values in a (complete, separable) ffi CMC conditional Monte CV coe cient of variation metric space (,S :), where S is a set of possible objects x and : is a σ- Carlo field of subsets of S [41, Appendix M],soX:→Ω S. In other words, CWO core-wide oxidation d.f. degrees of freedom for each outcome ωΩ∈ , we have that Xx()=ω for some x ∈ S. Also, i.i.d. independent and KS Kolmogorov-Smirnov if S ∈ : is a subset of S, then PSPωωS(XX ∈ )= ({: ( )∈ }) is the identically distributed probability that X lies in S. For example, if S = Rd, then LHS Latin hypercube LOCA loss-of-coolant accident X =(XX[1] , [2] ,…, X [d ] ) is a d-dimensional random vector (d=2 in sampling Example 1), and the d entries in X are called basic variables in MLE maximum likelihood MLO maximum local oxidation structural reliability [20, Section 1.5]. In this case, for each outcome estimate ωΩ∈ , we have Xx()=(ωXωXωXω[1] (), [2] (),…, [d ] ())= for some NPP nuclear power plant NRC Nuclear Regulatory d x =(xx12 , ,…, xd )∈R ; also, for the set Sxxx={(12 , ,…, d) Commission d fi d ∈R :xykk ≤k , = 1, 2, …, d } ∈ : for a xed (,yy12 ,…,)∈ yd R ,we PCD probability of correct PCT peak cladding [1] [2] []d have PSPXyXyXy(∈)=(X ≤,1 ≤,…,2 ≤)d . The probability P decision temperature may specify dependencies within the basic random object X, e.g., loads PSA probabilistic safety RISMC risk-informed safety- of different components may be dependent. Section 7.2 describes how assessment margin characterization we can use copulas [42] to incorporate dependence. rLHS replicated LHS SCL SS+CMC+LHS We allow S to be more general than Rd. The space (,S :) may be SCrL SS+CMC+rLHS SRS simple random sampling infinite-dimensional, as can occur when uncertainties are modeled fi UCB upper con dence VRF variance-reduction factor through a stochastic process. For example, over a continuous time bound interval [0, t0], a structure may be subjected to random events (e.g., VRT variance-reduction earthquakes or storms) arising according to a Poisson process [43, technique Section 23], and each event has a random intensity; as the number of

events occurring by t0 is unbounded, the vector X of the event times 2. Mathematical framework and intensities then requires S to have infinite dimension. Also, as in time-dependent reliability [20, Chapter 6], the system's state (which Before developing the general framework, we first motivate the may include, e.g., the loads, capacities, environmental conditions) problem by considering a simple model. The example actually does not evolves randomly within a state space E (e.g., E ⊆ Rd ) over continuous require simulation as it can be solved numerically, but we use it to time. In this case, the basic random object is a continuous-time, illustrate the ideas and notation. We will return to the example in later E-valued stochastic process X =[Xt (): t ≥0], where X(t) is the system's sections to introduce concepts for our various Monte Carlo methods. state in E at time t,so(,S :) is an appropriate space of E-valued functions on [0, ∞) [41, Chapters 2 and 3]. For example, consider a Example 1. Considerasystemwithq=1 component, which experiences a system with p ≥1components, where each component 1≤kp ≤ has (single) random load (stress or demand) L and has a (single) random load L[]k (t) and capacity Ct[]k ( ) at each time tt≤ . Then we may define capacity (resistance or strength) C to withstand the load. The system fails if 0 X =[Xt ():0≤ t ≤ t] with E = R2p and and only if L ≥ C.LetX =(LC , ) be a random vector taking values in 0 2 fi S = R ,withR the set of real numbers. De ne the failure function Xt()=( L[1] (), t L [[2] (),…, t L [pp ] (), t C [1] (), t C [[2] (),…, t C [ ] ()) t . (1) d()=(X IL ≥ C) for I(·) the indicator function, which equals 1 (resp., 0) when its argument is true (resp., false). Thus, the system fails if and only if Next define a binary-valued failure function d:→{0,1}S such

378 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394 that if X = x with x ∈ S and d()=x 1 (resp., 0), the system fails (resp., use the output data to construct a (statistical) point estimator θnl( ) of θ, does not fail) within the given time horizon. Applying d to the basic along with a γ-level upper confidence bound (UCB) B(n) for θ; i.e., random object X yields P(Bn ( )≥ θ )=γ. (We define the phrase “point estimator” or “point estimate” in the statistical sense, as a sampling-based estimator or 0 ≡()=(0 XXd ) (2) estimate of a parameter—such as an expectation or probability—whose as the failure indicator, which is a Bernoulli random variable that true (constant) value is unknown. This is in contrast to the meaning equals 1 (resp., 0) if the system eventually fails (resp., does not fail). sometimes adopted in nuclear safety, where a “point estimate” may Computing d and 0 can be quite complicated. For example, in a PSA of denote a single conservative value for a load or capacity, instead of a nuclear power plant, d is based on a detailed computer code [7] that assuming probability distributions for them, as when incorporating models the progression of a hypothesized accident, taking as input a uncertainties.) The exact form of B(n) depends on the simulation random vector X =(XX[1] , [2] ,…, X [dd ] )∈SR = representing aleatory method applied and how θnl( ) is constructed. But without imposing and epistemic uncertainties with a specified joint CDF. The entries of unreasonable parametric assumptions (e.g., normally distributed out- X, which may determine the timing, location and size of events during puts), it is typically not possible to construct a UCB B(n) for which the accident, may be dependent and may have different marginal CDFs. P(Bn ( )≥ θ )=γ exactly holds for a fixed sample size n. Instead, we The computer code can be computationally expensive to run for each desire B(n)tobeanasymptotic γ-level UCB, satisfying ff x ∈ S, as it may numerically solve systems of di erential equations. limPBn ( ( )≥ θ )= γ , In structural reliability [20], the failure function d in (2) is often n→∞ (5) fi de ned in terms of a limit state function gS:→ R, where g()≤0X where B(n) is often derived from a central limit theorem (CLT) for θnl( ). (resp., >0) denotes that the system fails (resp., does not fail). In this We then arrive at the following decision rule: case, d()=(()≤0)XXI g . Example 1 has g(,LC )= C − L, which is a safety margin, and d(,LC )=( IL ≥ C). Section 2.1 will provide other Reject/ 00 if and only ifBn ( ) < θ, (6) fi examples illustrating how d may be de ned for common types of which asymptotically fulfills (4) as n → ∞ when (5) holds. Starting in systems in terms of multiple loads and capacities. Section 3 we consider specific simulation methodologies to construct When the basic random object X is a continuous-time stochastic asymptotic UCBs achieving (5). (See [44] for a thorough treatment of process [Xt(): t ≥0], as in time-dependent reliability analysis, we may asymptotic statistical methods.) While the current paper focuses on fi de ne the failure function d in (2) in many ways. For example, consider X UCBs, the approaches presented can be easily modified to also provide fi fi with X(t)in(1),andlett0 be a xed time horizon. Then we may de ne lower confidence bounds or two-sided confidence intervals. Generating []kk [] d(X ) =IL ( ( t ) ≥ C ( t ) for some 1 ≤ k ≤ p and some 0 ≤ t ≤ t0),sothe the basic random object X or computing the failure function d in (2) system fails if and only if some component's load exceeds its capacity may be quite costly, making it important to apply VRTs to reduce the fi beforethetimehorizon.Foranotherexample,letτ >0be a xed constant, sample size n necessary to obtain sufficiently accurate results. ⎛ t ⎞ and set d(X ) =IILtCttτ⎜∫ 0 ([]kk ( ) ≥ [] ( )) d ≥ for some 1 ≤ kp ≤ ⎟,sothe ⎝ 0 ⎠ 2.1. Examples of defining the failure function in terms of loads and system fails if and only if for some component k, the amount of time its capacities load exceeds its capacity is at least τ. We define the system's failure probability (also known as the We now present some common ways of defining the failure function unreliability)as d in (2). Our examples specify system failure in terms of a fixed number θP= (000 =1)= E [ ]=∫∫ (XX )d P =d ( )d P , q ≥ 1 of criteria (or failure modes), where each has a random load and (3) a random capacity. Specifically, let LLX≡()=(,LL[1] [2] ,…, L [q ]) and [1] [2] [q ] where E is the expectation operator induced by P. The integrals in (3) CCX≡()=(,CC ,…, C) be random vectors computed from the []k [k] have the dimension (possibly infinite) of the space S. In the third basic random object X, with L and C as the load and capacity, expression of (3), we call the quantity inside the expectation a response respectively, for criterion k =1,2,…,q. For example, each of the q (function), and we want to compute its expectation θ, which is an criteria may correspond to a component in the system, and a unknown constant. To avoid trivialities, we assume that 0<θ <1. component fails when its load exceeds its capacity; the system then fi The system is deemed to be acceptably safe if the failure probability fails when speci ed combinations of components fail. (In Example 1, [1] [1] ffi there is only q=1 component, so L ==LLand C ==CC.) The is su ciently small, i.e., θθ< 0 for some given constant θ0. We assume the stochastic model is too complex to analytically or numerically load and capacity of a component i may be observed at multiple times tt<<⋯< t, and each pair of component i and time t could compute θ, but it can be simulated on a computer (which may, e.g., ii,1 ,2 im , i i,j require discretizing time). Thus, we will develop (Monte Carlo) correspond to a criterion. A nuclear PSA may have q=3 criteria, and [1] [2] [3] simulation methods to estimate θ. The corresponding simulation model L , L , and L represent the loads for criteria PCT, CWO, and MLO, may be a computer code that generates an observation of X according respectively, during a hypothesized loss-of-coolant accident; see para- to P, and eventually outputs 0 using (2). The resulting simulation graph 50.46(b) of [4]. The initial RISMC studies [3,19] consider only estimators have statistical error because of variability in the sampled q=1 criterion, PCT, and assume a triangular distribution for its observations of X, which we account for by further requiring the capacity, but we allow for q > 1. Throughout the paper, we permit for ff following: dependence across criteria (e.g., di erent components have related loads, or capacities at different times are correlated), and except for given constants 0 <θγ0 < 1 and 0 < < 1, Section 6.3, L and C can be dependent. determine with confidence levelγθθ if <0 . (4) When defining the failure function d in (2) in terms of L and C,we will assume the following. The constants θ0 and γ may be specified by a regulator. For example, if fi 2q θ0 =0.05 and γ =0.95, we want to determine with 95% con dence if Assumption A1. There are deterministic functions v:→SRand θ <0.05; when there is only a single criterion with constant capacity, d′:R2q → {0, 1} such that (,LC )=(v X) and d()=xxdv ′(()) for all this is equivalent to a 95/95 requirement. x ∈ S. We can satisfy the requirement (4) through a hypothesis test. In Assumption A1, the function v computes the load and capacity fi De ne the null hypothesis / 00:≥θθand alternative hypothesis vectors L and C from the basic random object X. For example, L may fi /10:<θθ. We perform a hypothesis test at signi cance level be a function of random stresses, which are included in X, and C may αγ=1− by first carrying out a total of n simulation runs. We then be computed from uncertain environmental conditions and material

379 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394 properties specified in X. Moreover, A1 further stipulates that the failure function d′ in Assumption A1 is defined as failure function d computed in terms of X in (2) can be equivalently ⎛ ⎞ expressed as the function d′ of only the loads L and capacities C d′(LC , ) =ILC⎜ ∩v ∪qm+1−1 {[]kk ≥ [] }⎟ ⎝ m=1 kq= m ⎠ because d()=′(())=′(,XXLCdv d ). Example 1 already has X =(LC , ), qq− so in this case, v is simply the identity mapping, and v mm+1 =(−1)∏∑ p−1 ∑ ILC (∩{≥}).p []kkll [] d()=′(,X d LC )=( IL ≥ C). We now show how to define the failure l=1 m=1 p=1 qkk≤12 < <⋯< kqp ≤ −1 function d′ of the q ≥ 1 loads and capacities for some common types of m m+1 systems. (11)

Example 2. (Series system; e.g., Section 5.2.3.1 of [20]). Suppose a As in Example 5, we allow that some of the q criteria may be the same. system with q criteria fails if and only if at least one criterion's load A series system (Example 2) is a special case with v=1 subset having all exceeds its capacity. Then by the inclusion-exclusion principle, we can q criteria; i.e., q1 = 1 and qq2 =+1. Another special case is a parallel define d′ in A1 as system (Example 3), which has v=q subsets, where each subset m contains only criterion m; i.e., qm= for each mq=1,2,…, +1. q []kk [] m q d′(LC , ) =ILC (∪k=1 { ≥ }) Let H be the joint CDF of (,LC )∈R2 ; i.e., (,LC )∼H, and q p−1 p []kkll [] H(,xy )=P ( L ≤ xC , ≤ y ) for constant q-vectors x and y. Under =(−1)∑∑ILC (∩{≥})l=1 . Assumption A1, we can rewrite (3) as θH=∫ 2q d ′(,)d(,xy xy ).We p=1 1≤kk12 < <⋯< kp ≤ q (7) R assume that computing θ is intractable, but there is a simulation model The paper [45] defines failure as in (7) for a nuclear PSA with q=3 that produces an observation of X so that (,LC )=()∼v X H. Let F criteria (PCT, CWO, MLO) and fixed capacities. (resp., G) denote the marginal joint CDF of the random vector L (resp., C). The setting of fixed capacities is a special case with degenerate G. Example 3. (Parallel system; e.g., Section 5.2.3.2 of [20]). Suppose a Example 1 has (,LC )∈R2q with q=1 and H a mixture of bivariate system with q criteria fails if and only if each criterion's load exceeds its normals, so F and G are mixtures of univariate normals. In general, we capacity. Then the failure function d′ in Assumption A1 is given by initially make no assumptions about the joint CDF H; e.g., we do not q []kk [] d′(LC , ) =ILC (∩k=1 { ≥ }). (8) require that H is a particular multivariate parametric distribution, such as Gaussian or lognormal. Also, H may incorporate dependence among Example 4. (K-out-of-N:F system). Suppose a system with N = q the 2q elements of (,LC). Later sections impose conditions as needed criteria fails if and only if there exist (at least) K criteria when introducing different simulation methodologies, which will k <<⋯design more efficient where ⎜ m ⎟ =!/(!(−)!mjmj) and the second equality can be shown as in ⎝ j ⎠ techniques, e.g., as in Section 6.3. Throughout the paper we will clearly [46]. A series system (Example 2) is a special case with K=1. Another identify when A1 (and other assumptions) are needed. special case is a parallel system (Example 3), which has K=N=q. Assumption A1 may not hold when the basic random object is a continuous-time stochastic process, as in time-dependent reliability Example 5. (Series-parallel system). Suppose the q criteria are [20, Chapter 6]. For example, [31] defines a “loss of assured safety” as a partitioned into v ≥ 1 subsets (or subsystems), where subset system failure, as we now explain. The system has p components, mv=1,2,…, , comprises criteria k =,qqmm +1,…,− q m+1 1, and where the loads and capacities may change randomly over continuous []k []k qq12=1< <⋯< qqqvv < +1 ≡ +1. Suppose further that the system time. Let L (t) and Ct( ) be the load and capacity, respectively, of fails if and only if for at least one subset m, the load exceeds the component k at time t, and let X =[Xt (): t ≥0] be the continuous-time capacity for each of the subset's criteria. Then in Assumption A1 the stochastic process with X(t) from (1).Define the failure time of failure function d′ is defined as component k as T []kkk=inf{tLtCt ≥0: [] ()≥ [] ()}, which is the first time ⎛ ⎞ the load of component k exceeds its capacity. Then partitioning the p q −1 d′(LC , ) =ILC⎜ ∪v ∩m+1 {[]kk ≥ [] }⎟ “ ” “ ⎝ m=1 kq= m ⎠ components into two nonempty sets, Ds ( strong links ) and Dw ( weak ” fi v links ), [31] provides several possible ways to de ne system failure in ⎛ q −1 ⎞ p−1 ⎜ p ml+1 []kk []⎟ terms of the p components’ failure times. One definition is =(−1)∑∑ILC ∩∩l=1kq = {≥}. ⎝ ml ⎠ []k []k p=1 1≤mm12 < <⋯< mp ≤ v d()=(supX IT ≤inf T), so the system fails if and only if all kD∈ s kD∈ w (10) strong links fail before any weak link fails. Under the assumption that the p components are mutually independent, [31] provides a repre- Some of the q criteria may be the same, i.e., there may exist criteria sentation for the failure probability θE=[()d X ] in terms of the margin- k ≠ k such that PL(=[]kkkk1212 L [ ] , C [] = C [ ] )=1. This allows the same 1 2 al CDFs of the component failure times, and examines different ways of criterion to be parts of different subsets. A series system (Example 2)is computing θ. The Monte Carlo methods we develop allow for depen- a special case with v=q subsets, where each subset m contains only dence of the components. In its full generality, the model just described criterion m; i.e., qm= for each mq=1,2,…, +1. A parallel system m does not satisfy Assumption A1 because the continuous-time stochastic (Example 3) is another special case with only v=1 subset having all q process X cannot be reduced to a finite number of scalar loads and criteria; i.e., q = 1 and qq=+1. 1 2 capacities (but it still fits in the setting of Section 2). (As noted in the Example 6. (Parallel-series system). Suppose the q criteria are again previous paragraph, the Monte Carlo methods we develop in the later partitioned into v subsets as in Example 5. Suppose further that the sections do not require A1.) But we can develop a discrete-time version system fails if and only if for each subset mv=1,2,…, , the load over a finite time horizon for which A1 does hold by defining a criterion exceeds the capacity for at least one of the subset's criteria. Then the for each component at each discrete time step.

380 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394

3. Simple random sampling Initiating Intermediate Events Scenario Event EE12 E 3 We first consider estimating θ with simple random sampling, which 1 is also known as naive simulation, standard simulation,orcrude 0.919 Monte Carlo. The approach to construct a UCB satisfying (5) when 0.9981 3 applying SRS, which does not require Assumption A1 from Section 2.1, SBO 0.99938 8.1E-2 is well known (e.g., Example 3.2.4 of [44]), but we review the details 4 1.9E-3 here as analogous arguments will be employed for the other Monte 2 Carlo methods in later sections. 6.2E-4

Example 1. (Continued). We now explain how to sample Fig. 1. An event tree for a hypothesized station blackout (SBO).

X =(LC , )∼H, where we recall H is a mixture of the joint CDFs H(s) λ of 52(,μs Σs), 1≤ss ≤ 0, with known mixing weights s. First generate One is then sometimes also interested in estimating the core-damage an observation of a discrete random variable A with PA(=)= s λs, frequency κ ≡ νθ, where ν is the (known) frequency of the postulated ss=1,2,…, 0.IfA=s, let X =+μs ΓsZ, where Z =(ZZ12 , ) is a (column) event. When applying SRS, we can then estimate κ by vector of i.i.d. 5(0, 1) random variables, Γ is a 2×2 matrix satisfying l κ s κlSRS()=nνθn SRS (), and an asymptotic γ-level UCB for is ΓΓ⊤ = Σ (e.g., Γ may be a Cholesky factor of Σ [13, pp. 72–74]), and ss s s s κlSRS()+nzνσnnγ l SRS ()/ . For the other Monte Carlo methods considered the superscript ⊤ denotes transpose. Thus, with probability λs, we have in the rest of the paper, we can similarly estimate and construct that X ∼ H(s), as required. We then estimate θE=[0] by averaging confidence bounds for κ; we omit the details. independent and identically distributed (i.i.d.) copies of 0 =()=(≥d X IL C). 4. Stratified sampling More generally, to implement SRS, first generate a sample of n i.i.d. copies Xi, in=1,2,…, , of the basic random object X in (2), and set A key aspect of the RISMC approach is decomposing the sample 0 =(d X ). Then the SRS point estimator of θ in (3) is ii space of a hypothesized event into s0 ≥1scenarios through an event θnl ()=(1/) n∑n 0, which is unbiased; i.e., E[()]=θnl θ. The SRS SRS i=1 i SRS tree [18]. For example, Fig. 1 depicts an event tree from [3] with s0 = 4 point estimator averages i.i.d. copies of a bounded response, so scenarios of a postulated station blackout. The intermediate events n E ,,EEdetermine how the accident progresses; e.g., the lower (resp., θnl ()− θ ⇒5 (0,1)as→∞, n 123 []SRS upper) branch of E is the event that a safety relief valve is stuck open σnlSRS() (12) 2 (resp., properly closes). The branching probabilities of the intermediate by the ordinary CLT [43, Theorem 27.1], where ⇒ denotes convergence events are assumed known, as shown in the figure, from, e.g., previous in distribution (i.e., at each continuity point of the limit's CDF, the CDF risk studies or historical data. Following a path from left to right of the left side of (12) converges to the limit's CDF [43, Chapter 5]), and through the event tree leads to a scenario ss=1,2,…, 0, and the σnl2 ()=()[1−() θnll θn] consistently estimates σθθ2 =[1−]; i.e, SRS SRS SRS SRS probability λs of scenario s occurring is computed by multiplying the 2 2 σnlSRS()⇒ σSRS as n → ∞. Because the normal limit in (12) has a branching probabilities of the intermediate events along the path; e.g., continuous distribution, the portmanteau theorem (e.g., Theorem scenario 4 occurs with probability λ4 = 0.99938 × 0.0019. But the failure 29.1 of [43]) ensures that probability θ(s) of each scenario s is unknown, and must be estimated ⎛ ⎞ via some form of simulation. ⎜ n l ⎟ P [θnSRS ( )− θ ]≥ x → P (5 (0,1)≥ x ) as n →∞ ⎝ σnlSRS() ⎠ (13) Example 1. (Continued). In Fig. 1, let A be a discrete random variable denoting which scenario s occurs, so PA(=)= s λ, 1≤ss ≤ , where γ s 0 for each real-valued constant x. Let zγ be the -level upper one-sided λ the s are known. When A=s, assume that X =(LC , ) has the 52(,μs Σs) critical point of a 5(0, 1) random variable, which satisfies CDF H(s), so we can set X equal to a random vector X()ss∼ H (). Thus, as PzPz((0,1)≤)=((0,1)≥−)=55γ, e.g., z =1.645. Hence, γγ0.95 before, the unconditional CDF H of X is a mixture of CDFs H(s), λ 1≤ss ≤ 0, with mixing weights s. Also, X()sss=(LC () , ()) has the σnlSRS() B ()≡nθnzl ()+ conditional distribution of X =(LC , ) given A=s. Let F (resp., G ) SRS SRS γ n (14) (s) (s) be the conditional marginal CDF of L(s) (resp., C(s)); e.g., G(s) is the is an asymptotic γ-level UCB satisfying (5) because PB(()≥ n θ) 2 SRS 5(,μσCs, Cs, ) CDF. Given scenario s occurs, the failure indicator l n =Pnθn ( [SRS ( )− θσn ]/l SRS ( )≥− zγγ )→ P (5 (0,1)≥− z )= γas → ∞ 0()ss=(d X () )=(IL () ss ≥ C ()) has the conditional distribution of 0 in by (13) with x =−zγ. Computing BSRS(n) uses only the observed (2), and the (conditional) failure probability is then simulation outputs and known constants, and does not require any θE= [00 ] = P ( = 1) = PLC ( ≥ ) =∫ d (xx ) d H ( ). We can ()ss () () s() ss () R2 ()s unknown parameters. A simple modification of the well-known sign then estimate θ(s) by averaging i.i.d. copies of test (e.g., Example 3.2.4 of [44]), this approach has been specialized to 0()ss=(d X () )=(IL () ss ≥ C ()). multi-criteria nuclear PSAs with fixed capacities in Section 4.2 of [47]. The event-tree framework is well suited for estimating θ via

We can easily modify the UCB BSRS(n) to obtain an asymptotic stratified sampling, with each scenario corresponding to a stratum. fi γ-level lower con dence bound for θ by changing the plus in (14) to a In general, SS partitions the sample space into s0 ≥1strata, where each minus. An asymptotic γ-level two-sided confidence interval for θ is stratum has a known probability of occurring; see Section 4.3 of [13] l fi simply (()±θnSRS z 1−(1−γ )/2 σnnl SRS ()/). For the UCBs in the rest of the for an overview of SS. This is done through a strati cation variable A, paper, we can similarly modify them to obtain lower confidence bounds which is a random variable AΩ:→R defined on the same probability and two-sided confidence intervals, so we omit the details. space (,ΩP- , ) as the basic random object X. Partition the support R of If the failure function d is defined as in Assumption A1 in terms of a A into s subsets RR,,…, R, with RR=∪s0 and RR∩=∅for 0 12 s0 s=1 s ss′ function d′ of q loads and capacities, we apply SRS with sample size n ss≠ ′. We call each Rs a stratum, and its index s a scenario. Let by letting (,LCii )=(v X i), in=1,2,…, , using v from A1, where λ = (λss : = 1, 2, …, s0), where each stratum probability λss=(∈PA R) [1] [2] [q ] [1] [2] [q ] Liii=(LL , ,…, L i), Ciii=(CC , ,…, C i), and Xi is the ith i.i.d. is assumed known and positive. (We could have defined A to be a more l copy of the basic random object X, as before. Then estimator θnSRS( ) general random object, but for simplicity we only consider A as just a fi averages the 0iii=′(,d LC), 1≤in ≤ . random variable.) In Example 1 the strati cation variable A is the In a RISMC study of a postulated NPP event (e.g., a LOCA), it is randomly chosen scenario, its support R = {1, 2, 3, 4} decomposes into typically assumed that a failure, as in (7), results in damage to the core. strata Rss ={}for s =1,2,3,4, and, e.g., λ4 = 0.99938 × 0.0019. Let

381 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394

s X be an S-valued random object having the conditional distribution η* =(λσ )/∑ 0 (λ σ ). Note that η* depends on the unknown (s) s ss(),SS s′=1 ss′′(),SS of the basic random object X in (2) given AR∈ s. Assume that for each σ(),Ss S in (21), making it impossible to directly implement the optimal allocation. But one could instead apply a two-stage procedure: the first scenario s, we can generate observations of X(s), e.g., via a computer * code with random inputs. Then for the failure function d in (2), let stage estimates the σ(),Ss S, which are used to estimate η , and the second stage then collects additional observations from the scenarios using the 0 =(d X ), ()ss () (15) estimated optimal allocation. fi which has the conditional distribution of the failure indicator 0 in (2) When Assumption A1 holds (i.e., failure is de ned in terms of q fi loads and capacities), for the function v in A1, let (,LC )=()v X, given AR∈ s. The failure probability in (3) satis es ()ss () () s which has the conditional distribution of (,LC) given AR∈ , where s s s 00 [1] [2] []q [1] [2] []q L()ssss=(LL () ,() ,…, L()) and C()ssss=(CC () ,() ,…, C() ). Let H(s) denote θPARPARλθ=(∈)(=1∈)=∑∑ss0 ss( ) s=1s =1 (16) the joint CDF of (,LC()ss ()) in scenario s, and let F(s) (resp., G(s)) be the marginal CDF of the load vector L(s) (resp., capacity vector C(s)). Then by the law of total probability, where the failure indicator 0(s) in (15) becomes

θPss=(00 =1)=[ E s] () () () (17) 0()sss=′(,d LC () ()) (24) is the (conditional) failure probability for scenario s.In(16), each λs is for failure function d′ in A1. To construct the SS estimator in (18) of θ(s), known but θ is not, and we use some form of simulation to estimate (s) we use the same i.i.d. observations X(),si1≤in ≤ s,ofX(s) from before. each θ(s), which are combined as in (16) to obtain an estimate of θ.(A Then set (,LC )=()∼v X H , where L =(LL[1] ,[2] ,…, L[]q ) RISMC study may also be interested in the value of each θ .) (),si (), si (), si ( s) (),si (), si(), si(), si (s) (resp., C =(CC[1] ,[2] ,…, C[]q )) is an observation of the q loads Specifically, we implement SS with overall sample size n by letting (),si (), si(), si(), si [1] [2] [q ] [1] [2] [q ] η (,LL ,…, L) (resp., capacities (,CC ,…, C)) given AR∈ s. ns = ηns be the sample size allocated to scenario s, where s, fi The estimator θnl ( ) in (18) of θE=[0 ] averages ss=1,2,…, 0, are user-speci ed positive constants satisfying (sη ),SS, ()ss () s 0 0(),si=′(d LC (), si , (), si), 1≤in ≤ s, which are i.i.d. ∑s=1 ηs = 1. We call η = (ηss : = 1, 2, …, s0) the SS allocation. For simplicity, we assume that ns is an integer; otherwise, let ns =⌊ηns ⌋, fl where ⌊·⌋ denotes the oor function. For each scenario ss=1,2,…, 0, 5. Combined SS and Latin hypercube sampling let X(),si, in=1,2,…, s, be a sample of ns i.i.d. copies of X(s) in (15), and let 0(),si=(d X (), si) for d in (2). Then we estimate the expectation θ(s) of We now consider combining SS with Latin hypercube sampling to the response 0(s) by estimate θ. Originally proposed by [48], LHS can be thought of as an n efficient way to simultaneously stratify multiple input coordinates, and 1 s θnl ()= ∑ 0 it reduces variance by producing negatively correlated outputs (under (sη ),SS, n (),s i s i=1 (18) certain regularity conditions). The LHS estimator of the mean of a fi where the subscript SS denotes stratified sampling with simple random bounded response function satis es a CLT [49], and [50] extends this fi sampling applied within each stratum. Then to the setting when the response has a nite absolute third moment. Due to its ease of implementation and broad applicability, LHS is s0 ll widely used in practice; e.g., citing over 300 references, [51] surveys θnSS,η()=∑ λθnss( ),SS, η () s=1 (19) LHS works, with an emphasis on nuclear engineering. Applying LHS requires additional problem structure. Let <[0, 1] is the SS estimator of θλθ= ∑s0 . For each scenario ss=1,2,…, , s=1 ss( ) 0 denote a uniform distribution on [0, 1], and we will assume that the l fi the estimator θn(sη ),SS, ( ) satis es a CLT basic random object can be generated using a fixed number of i.i.d. n l <[0, 1] random numbers. []θnθ(),SS,sη()− () s ⇒5 (0,1)asn →∞, σnηl(sη ),SS, () s (20) Example 1. (Continued). Let U1 and U2 be i.i.d. <[0, 1]. We can then

2 ll generate X()s ∼(,5 2μs Σs) through a function w(s) of the ds=2 uniforms where σnθnθnl(sη ),SS, ()≡()[1−()( sη ),SS, ( sη ),SS, ] consistently estimates that operates as follows. First build vector Z =(ZZ12 , ) with 2 −1 Φ σθθ(),SSssss≡Var[0() ]= () [1− () ]. (21) ZΦUii=(), where is the 5(0, 1) CDF. Then w()s (,UU 1 2) returns 2 ⊤ X()s =+μs ΓsZ, which lies in the set S = R , where ΓΓss= Σ s. (The extra factor ηs appears on the left side of (20) because the scaling + More generally, we impose the following condition, where = is n but the estimator θnl ( ) is based on a sample size of n = ηn.) (sη ),SS, s s denotes equality in distribution. Assuming that the s0 scenarios for SS are simulated independently, we ss then have that (20) jointly holds for =1,2,…, 0, by Problem 29.2 of Assumption A2. For each scenario ss=1,2,…, 0, there is a [43], so the SS estimator θnl ( ) of the overall failure probability θ deterministic function wS: [0, 1]ds → such that if UU,,…, U are SS,η ()s 12 ds fi l satis es the CLT nθ[()−]/()⇒(0,1SS,ηη n θ σl SS, n 5 ) as n → ∞, where ds i.i.d. <[0, 1] random variables, then 2 s 2 2 σnll()≡∑ 0 λσnη()/ consistently estimates + SS,η s=1 ss( ),SS, η s w (,UU ,…,)= U X . ()sds 1 2s () (25) s0 λσ2 2 σ 2 ≡ ∑ ss(),SS; SS,η η s=1 s (22) fi In other words, the function w(s) takes a xed number ds of e.g., see p. 215 of [13]. Finally, an asymptotic γ-level UCB for θ when i.i.d. <[0, 1] random numbers as inputs, and transforms them into using SS is an observation of the basic random object X()s ∈ S having the correct distribution for scenario s. Consider the special case when [1] [2] []d σnlSS,η() X =(XX , ,…, Xs ) B ()=nθnzl ()+ , ()sss () () ()s is a random vector with independent (but SS,ηηγ SS, n (23) [j] not necessarily identically distributed) entries, where each X()s has fi marginal CDF Q . Then we set w (,UU ,…,)=( U Q−1 which satis es (5) (without requiring Assumption A1). If there is only (),sj ()sds 1 2s (),1 s =1scenario, then SS reduces to SRS. (),UQ−1 (),…, U Q−1 ( U )) , assuming Q −1 can be computed. 0 1(),2ss2(),dssd (),sj When the SS allocation η is chosen so that η = λ, which is called the [1] [2] []ds Assumption A2 also permits the random vector (,XX()ss() ,…, X()s ) proportional allocation, it is well known (e.g., p. 217 of [13]) that SS (or a more general basic random object X(s)) to have a dependence 2 2 guarantees a variance reduction compared to SRS; i.e., σσSS,λ ≤ SRS. The structure, and Section 7.2 shows how copulas [42] can be used to η 2 * * optimal choice of to minimize σSS,η is η =(ηss : = 1, 2, …, s0) with model the dependence. While the framework of Assumption A2 is quite

382 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394

lls0 general, it precludes applying variate-generation methods such as θnSS+LHS,η()=∑s=1 λθnss( ),SS+LHS, η( ). acceptance-rejection (e.g., Section II.2b of [14]), in which the number As the response 0(),s i in (29) is bounded, the LHS CLT of [49] l fi of uniforms used to generate X(s) is random and unbounded. Also, A2 implies the estimator θn(sη ),SS+LHS, ( ) of θ(s) satis es does not allow the space S in which X lives to be infinite dimensional, (s) n as is the case for some stochastic processes; see Section 2. θnθl ()− ⇒5 (0,1)as n →∞ ση/ [](sηs ),SS+LHS, ( ) Before describing how to employ LHS under Assumption A2 (but (s ),SS+LHS s (30) fi without requiring A1), we rst explain the case of applying SRS within for each scenario s, where we will give an expression for the asymptotic each scenario s to obtain ns i.i.d. failure indicators. Arrange 2 variance constant σ(s ),SS+LHS in Section 5.1. We assume the s0 scenarios i.i.d. <[0, 1] random numbers U(),,sij, 1≤in ≤ s, 1≤jd ≤ s,inan for SS are simulated independently, so (30) jointly holds for nss× d grid, and apply the composition d((·)w()s ) of (15) and (25) to ss=1,2,…, 0, by Problem 29.2 of [43]. Hence, the overall SS+LHS each row: θ l estimator of obeys the CLT nθ[()−]/⇒(0,1)SS+LHS,ηη n θ σ SS+LHS, 5 as 0 =(dw (UU , ,…, U )), n → ∞, where the asymptotic variance is (),1sssss() (),1,1 (),1,2 (),1,ds s 2 2 0(),2ss=(dw() (UU (),2,1sss , (),2,2 ,…, U (),2,d )), 0 s 2 λσss( ),SS+LHS ⋮⋮ ⋮ ⋮ ⋱ ⋮ σSS+LHS,η =.∑ η (31) 0 =(dw (UU , ,…, U )), s=1 s (),snsssss()s (),,1 sn (),,2 sn (),,snd (26) If Assumption A1 (i.e., failure is defined in terms of q loads and where for each in=1,2,…, s, we have that 0(),s i is the failure indicator capacities) holds in addition to A2, then we can sample the load and for the ith run of scenario s. Because each row in=1,2,…, s,of(26) capacity vectors for scenario s through a deterministic function has ds i.i.d. <[0, 1] random numbers, 0 has the correct distribution ds 2q fi (),s i wR′()s : [0, 1] → de ned as by (15) and (25). Also, as the rows of uniforms in (26) are independent, wv′()sdsds(,UU 1 2 ,…,)≡( Uw() (, UU 1 2 ,…,))=(, U LC() ()ss )∼H () we have that 0(),s i, in=1,2,…, s, are i.i.d. We can then use them to ss (32) l construct the estimator θn(sη ),SS, ( ) in (18). Moreover, assuming the grids wtih UU,,…, U as i.i.d. <[0, 1] random variables, where v is from 12 ds in (26) across scenarios ss=1,2,…, 0, are independent, we then apply A1, w(s) is from A2, and we recall that H(s) is the joint CDF of (,LC()ss ()). (19) to obtain the asymptotic UCB in (23). In other words, the function w′(s) takes ds i.i.d. <[0, 1] random numbers We now explain how to implement LHS under the framework of as inputs, and transforms them into an observation of the load and Assumption A2 (without requiring A1) to obtain a dependent sample of fi capacity vectors having the correct joint CDF H(s). Speci cally, w′(s) n failure indicators. For each scenario ss=1,2,…, , and each input fi s 0 accomplishes this by rst using w(s) to generate the basic random object coordinate jd=1,2,…, s,in(25), let π(),sj= (ππ (), sj (1), (), sj (2), …, X(s), and then feeds X(s) into v to obtain (,LC()ss ()). (In Example 1, πn(),sj() s) be a random permutation of (1,2,…,ns ); i.e., each of the ns! because X()sss=(LC () , ()), the function v is the identity mapping, so permutations of (1,2,…,ns )is equally likely, and πi(),sj( ) is the number ww()ss= ′ ().) Then for the failure function d′ in A1, we can generate the to which i is mapped in permutation π(),sj. Also, let π(),sj, jd=1,2,…, s, failure indicator in (15) and (24) as be independent random permutations, independent of the i.i.d. U in (),,sij 0 =(dw (,UU ,…,))=′( Ud w′ (, UU ,…,))=′(, U d LC ). (26). Then let ()ss () 1 2 dss () s 1 2 d () ss () (33) πi(),sj()−1+ U (),, sij V(),,sij= , Also, we compute each 0 in (28) as n (27) (),s i s 0 =′(dw′ (,VV ,…,) V) to obtain the estimator in (29). (),si () s (),,1 si (),,2 si (),, sids which is easily shown to be <[0, 1]. Next arrange V(),,sij, 1≤in ≤ s, 1≤jd ≤ s, into an nss× d array, which we call an LHS grid (of 5.1. Analyzing the effect of LHS uniforms), and apply the composition d((·)w()s ) of (15) and (25) to each row to get 2 We now give an expression for σ(s ),SS+LHS in (30) and (31) under Assumption A2 (without A1). Let U =(UU , ,…, U) be a vector of d 0 =(dw (VV , ,…, V )), 12 ds s (),1ss() (),1,1sss (),1,2 (),1,ds i.i.d. <[0, 1] variables. The failure indicator in (15) satisfies 0(),2ss=(dw() (VV (),2,1sss , (),2,2 ,…, V (),2,d )), s 0 ddwXUU0 ⋮⋮ ⋮ ⋮ ⋱ ⋮ ()ss=( () )=( () s ())≡ () s () (34)

0(),sn =(dw()s (VV (),,1 sn , (),,2 sn ,…, V (),,snd )). sssss (28) by (25), where we write 0()s (U) to emphasize the dependence of 0(s) on U. Next use an analysis-of-variance (ANOVA) decomposition as in [52] It is straightforward to show that V ,,…,VVin each row (),,1si (),,2 si (),, sids to approximate the response function in (34) for scenario s by an i of (28) are d i.i.d. <[0, 1] random numbers, so the failure indicator s additive function of the uniform inputs: 0(),s i has the correct distribution for scenario s by (15) and (25). But all ds entries in each column j of the V(),,sijin (28) share the same permutation 0()ss()=UUθϕU () +∑ ()+ϵ(),js() π(),sj, so the ns responses 0(),s i, in=1,2,…, s,in(28) are dependent. (),sj fi j=1 (35) Each column jd=1,2,…, s, of the V(),,sijin (28) forms a strati ed fi sample of size ns of the unit interval partitioned into ns subintervals, where the residual ϵ()()s U is de ned so the equality in (35) holds, and each of length 1/n . Specifically, for each column j, there is exactly one ϕU()=[ E0 (, UU ,…,) UU ]− θis a function of only the jth s (),sj js() 1 2 djss () value V(),,sijin the stratum [(−1)/,/)ananssfor each an=1,2,…, s,as uniform input Uj as the other coordinates have been integrated out by 2 fi can be easily seen through (27). the conditional expectation. Then for σ(),Ss S de ned in (21), [52] and Let SS+LHS denote stratified sampling with LHS used within each [49] show that the asymptotic variance in (30) satisfies scenario. (If SS has only s =1scenario, then SS+LHS reduces to LHS.) 0 ds ⎡ ⎤ 2 2 For 0(),s i, in=1,2,…, s, from (28), the SS+LHS estimator of θ(s) in (17) σσϕU= Var[ϵ (U )] = − Var⎢ ( )⎥. (sss ),SS+LHS () (),SS ∑ ⎣ (),sj j ⎦ is j=1 (36) n 1 s Thus, σσ2 ≤ 2 for each scenario s, so LHS removes the θnl ()= 0 , (ss ),SS+LHS (),SS (sη ),SS+LHS, ∑ (),si variability of the additive part of the response 0 (U). The next result, ns i=1 (29) ()s which then directly follows from putting (36) in (31) and comparing to which averages dependent terms but is still unbiased because of the (22), establishes that after combining across all scenarios, the asymp- + linearity of expectation and each 0(),si= 0 ( s). The SS+LHS estimator of totic variance for SS+LHS is no larger than that for SS. s0 the overall failure probability θλθ= ∑s=1 ss( ) is Theorem 1. Under Assumption A2, when both SS and SS+LHS use

383 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394 the same stratification allocation η, we have that σσ2 ≤ 2 , with 〈〉r 〈〉r 〈〉r 〈〉r SS+LHS,η SS,η 0 =((dws VV , ,…, V )), ⎡ ⎤ (),1s () (s ),1,1 (s ),1,2 (),1,sds σσ2 −=2 ∑s0 (/) λη2 ∑ds Var⎢ ϕU ( )⎥ ≥ 0. 0〈〉r =((dw VV〈〉r ,〈〉r ,…, V〈〉r )), SS,ηηSS+LHS, s=1 s s j=1 ⎣ (),sj j ⎦ (),2s ()s (s ),2,1 (s ),2,2 (),2,sds In general, LHS can produce substantial variance reduction when ⋮⋮ ⋮ ⋮ ⋱ ⋮ 0〈〉r =(dw (VV〈〉r ,〈〉r ,…, V〈〉r )). the response whose expectation we are estimating is nearly an additive (),smsssss()s (),sm ,1 (),sm ,2 (),smd , (38) function of the input variables. To see why, we now adapt the explanation in [22, Section 10.3] to our setting, where the response The estimator of the failure probability θ(s) in (17) for scenario s for each scenario s is the indicator function on the left side of (35). from replication r is Putting the additive representation (35) in (29) when the uniform ms l〈〉r 1 〈〉r θn()s ()= ∑ 0 (),1s , inputs are now from the LHS grid of V(),,sijin (28) leads to ms i=1 (39) nss⎡ d ⎤ 1 ⎢ ⎥ and the estimator of the overall failure probability θ in (16) from θnl ()= θϕV+()+ϵ()V (sη ),SS+LHS, ∑∑⎢ ()s (),sj (),,sij () s (), si⎥ ns i=1 ⎣ j=1 ⎦ replication r is d n n s ss s 〈〉r 0 〈〉r 1 1 θnll()= λθn (). =+θ()s ∑∑ϕV(),sj()+(),,sij ∑ϵ(()ssiV (), ), ∑ ss() j=1ns i =1 ns i=1 (37) s=1 (40) fi θ where V =(VV , ,…, V) is from the ith row of (28). Recall The nal SS+rLHS estimator of across all b replications is (),si (),,1 si (),,2 si (),, sids that for each input coordinate jd=1,2,…, s, the jth column of the b ll1 〈〉r LHS grid of inputs in (28) forms a stratified sample from the unit θnSS+rLHS,ηb , ()= ∑ θn(). b r=1 (41) interval partitioned into ns subintervals, each of length 1/ns. Thus, for each input coordinate j in the middle term on the right side of (37), the To derive a UCB for θ when employing SS+rLHS with overall average (1/nϕV ) ∑ns ( ) corresponds to a (random) Reimann ()r s i=1 (),sj (),,sij sample size n, compute the sample variance of θnl ( ), 1≤rb ≤ ,as 1 ⎡ ⎤2 approximation to the one-dimensional integral α ≡()d∫ ϕuu. 1 b 〈〉r (),sj 0 (),sj Sn2()= ∑ ⎢ θll()− n θ () n⎥ . Let τ be the upper one- b b −1 r=1 ⎣ SS+rLHS,ηb , ⎦ bγ−1, When each conditional expectation ϕ(),sj(·) is smooth enough, a sided γ-level critical point of a Student-t random variable T − with Reimann approximation of the one-dimensional integral α(),sjbased b 1 on ns points converges much more quickly than the standard Monte b − 1 degrees of freedom (d.f.); i.e., γ =(PTbbγ−1 ≤ τ −1, )= P(≥−Tτ) by the symmetry of the T − density function. For Carlo rate of1/ ns as ns → ∞. The last term in (37) has error decreasing bbγ−1 −1, b 1 example, τ =1.833. The next result, whose proof is in Appendix A, at rate 1/ ns , which thus dominates the error of the middle term in 9,0.95 (37). Hence, if the additive approximation in (35) is good in the sense includes rLHS as the special case when s0 =1, which can provide a UCB that the residual has small variance, then LHS can do much better than for θ(s) for a single scenario s. SRS for each scenario. Finally, combining across scenarios leads to SS Theorem 2. Under Assumption A2, an SS+rLHS asymptotic γ-level +LHS having substantially lower variance than SS when the response θ l UCB for is BSS+rLHS,ηb ,()=nθ SS+rLHS,ηb , ()+ nτSnb b−1, γ b ()/ , i.e., function for each scenario is nearly additive. fi limnη→∞PB ( SS+rLHS, ,b ( n )≥ θ )= γ as in (5) for any xed b ≥ 2 and SS But the response function 0()s (U) in (35) is an indicator, so the allocation η. additive approximation in (35) may be a poor fit. Hence, SS+LHS may If Assumption A1 from Section 2.1 also holds (i.e., failure defined in not reduce variance much compared to SS. In Section 6 we consider 〈〉r 〈〉r terms of q loads and capacities), (38) has 0 =′(dw′ (,V further applying conditional Monte Carlo, which produces a smoother (),si ()s (),,1si V 〈〉r ,…,V 〈〉r )), 1≤im ≤ , with functions d′ and w′ as in response function. This can lead to a better additive fit and correspond- (),,2si (),,sids s (s) Assumption A1 and (32), respectively. ingly smaller asymptotic variance when applying LHS.

6. Combined SS, conditional Monte Carlo, and LHS 5.2. Combined SS and replicated LHS to construct UCB Conditional Monte Carlo (also known as conditional expectation or While it is possible (e.g., see [49]) to consistently estimate the conditioning) reduces variance by analytically integrating out some of asymptotic variance σ 2 in (36) to construct a UCB for θ,we (s ),SS+LHS the variability; see Section V.4 of [14] for an overview of CMC. instead now apply replicated LHS [21]. Here we generate b ≥ 2 independent LHS samples, e.g., b=10, where each of the b LHS Example 1. (Continued). Recall that because the basic random object samples are as in (28), but with size mηnb=/(i.e., m rows) instead s s s in scenario s is X()sss=(LC () , ()), the function w(s) in (25) generating X(s) of ns, and n is the overall sample size across all scenarios and is the same as the function w′(s) in (32) producing the load and capacity. replications. We assume for simplicity that each ms is integer-valued; Now further suppose that ww()ss= ′ () uses two other functions otherwise, let mηnb=⌊ / ⌋. s s l()s : [0, 1] → R and c()s : [0, 1] → R to generate the load L(s) and We now give the details of estimating θ by combining SS with rLHS, fi capacity C(s), respectively, in X()s ∼(,5 2μs Σs). Speci cally, for U1 and denoted by SS+rLHS, where we use rLHS within each scenario, and U2 as ds=2 i.i.d. <[0, 1] random numbers, we assume that rLHS samples across scenarios are independent. For each scenario wlc()sss(,UU 1 2 )=((),( () U 1 () U 2 )), where ss=1,2,…, , and each replication r =1,2,…,b, let 0 −1 〈〉r 〈〉r 〈〉r 〈〉r l()s ()=Uμ 1 + σΦULFLs, ()=1()() s ∼ s and π(),sj= (ππ(),sj (1),(),sj (2), …, πm(),sj (s )) be a random permutation of Ls, (1,2,…,m ) for input coordinate jd=1,2,…, . Also, let π 〈〉r , −1 s s (),sj c()s ()=UμσΦUCG 2 Cs, +Cs, ()=2()() s ∼ s (42) 1≤jd ≤ s, 1≤rb ≤ , 1≤ss ≤ 0, be independent random permutations. fi Then independently of the permutations, de ne mds × s i.i.d. <[0, 1] are the load and capacity, respectively. Note that l(s) takes as input 〈〉r random numbers U(),,sij, 1≤im ≤ s, 1≤jd ≤ s, as in the array in (26) but ds′ = 1 uniform U1, and c(s) has the other dss− d′ = 1 uniform U2 as input. ff with only ms rows instead of ns, where the b grids of i.i.d. uniforms for As l(s) and c(s) use di erent inputs, which are independent, the load 2 2 r =1,2,…,b, are also independent. For each scenario ss=1,2,…, 0, L()s ∼(5 μσLs, ,Ls, ) is independent of the capacity Cμσ()s ∼(5 Cs, ,Cs, ).By 〈〉r 〈〉r 〈〉r and each replication r =1,2,…,b, let V(),,sij=[πi(),sj ()−1+ U(),,sij ]/ ms, taking iterated expectations (e.g., p. 448 of [43]) of the failure indicator fi 1≤im ≤ s, 1≤jd ≤ s, which we arrange in an mds × s LHS grid, and 0()sss=(IL () ≥ C ()), the failure probability in (17) for scenario s satis es apply the composition d((·)w()s ) of (15) and (25) to each row to obtain θE()ss= [00 () ]= EEL [ [ () ssss | () ]]= E [ 1 () ( L () )], (43) ms identically distributed but dependent copies of 0(s):

384 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394

10LELPCLLGL where()ss ( () ) = [ () s | () s ] = ( () s ≤ () s | () s ) = () ss ( ()) (44) 1≤in ≤ s, as a sample of ns i.i.d. copies of M(s). The SS+CMC estimator of the failure probability θ(s) for scenario s is then by the independence of L(s) and C(s). Therefore, (43) implies that l ns lls0 θnn(sη ),SS+CMC, ( )=(1/ s )∑i=1 1()ssi(M (),), and θnSS+CMC,η()=∑s=1 λθnss( ),SS+CMC, η( ) sampling identically distributed copies of 1()ss(L ()) will produce an is the SS+CMC estimator of the overall failure probability θ in (16). unbiased estimator of θ(s). This is the key idea of CMC (when combined To build a UCB for θ, let σnnl2 ( ) = (1/( − 1)) ∑ns with SS). The advantage of this approach becomes apparent from a (ss ),SS+CMC i=1 [1 ()−M θnl ()]2 be a consistent estimator of variance decomposition (e.g., p. 456 of [43]): ()ssis (), (),SS+CMC, η 2 2 s0 2 2 σ(ss ),SS+CMC =Var[1() (M ()s )],soσnllSS+CMC,η()=∑s=1 λσnηss( ),SS+CMC()/s 2 s 2 2 Var[00()ss ] = Var[EL [() | ()s ]] + E [Var[ 0() s | L ()s ] 0 consistently estimates σλSS+CMC,η = ∑s=1 ssση( ),SS+CMC/ s. Recalling zγ satisfies Pzγ(5 (0, 1) ≤ ) = , we obtain an SS+CMC asymptotic γ-level ≥Var[[EL01()ss | () ]]=Var[()s ( L () s )]. (45) γ l UCB for θ as BSS+CMC,ηηγη()=nθ SS+CMC, ()+ nzσl SS+CMC, ()/ nn; i.e., η Thus, sampling 1()ss(L ()) rather than 0(s) yields a variance reduction. But limnη→∞PB ( SS+CMC, ( n )≥ θ )=γ as in (5) for any SS allocation . The applying the combination SS+CMC here critically relies on being able SS+CMC UCB requires neither Assumption A1 nor A2. to generate and compute 1()ss(L ()), which is a function of only L(s) as C(s) We now impose Assumption A2 so we can add LHS to SS+CMC. By has been integrated out by the conditional expectation in (44). Because the first equality in (33), (46) becomes G is the 5(,μσ2 ) CDF, we can use (42) to compute (44) as (s) Cs, Cs, θE= [0 ] = E [dw ( ( UUU , , …, ))] = EEUUU [ [dw ( ( , , …, )) |M ]]. −1 ()sssdsds () () 1 2ss () 1 2 () 1()ss(LΦLμσΦσΦUμμσ () ) = ([ () s − ]/Cs,, ) = ([ Ls (1 ) + − ]/Cs, ), which Cs, Ls,, Cs (49) is a function of L(s) (or U1) but not C(s) (nor U2). In contrast to the fi failure indicator 0()sss=(IL () ≥ C ()), the response 1()ss(L ()) is not binary- We assume the conditioning random object M(s) satis es the following: valued but rather lies in the interval (0, 1) because it is a conditional probability. Assumption A3. For M(s) an S′-valued conditioning random object, * ds ds′ Returning to the general problem, we now explain how to combine there exist functions wSS()s :[0,1] → × ′ and m()s : [0, 1] → S′ with θE=[0 ] 1≤dd′ ≤ such that for wS: [0, 1]ds → in (25) and UU,,…, U CMC with SS to estimate ()ss () in (17) for each scenario ss ()s 12 ds ss=1,2,…, 0. (Later we will add LHS.) Let M(s) be an S′-valued i.i.d. <[0, 1], fi conditioning random object for CMC de ned on the same probability + (XM,)=wwm* (,UU ,…,)=((, U UU ,…,), U (, UU ,…,)). U′ space as X(s), where (′,′S : ) is a metric space, so each simulation run ()s ()ssdsd () 1 2 sss() 1 2 ()sd 1 2 generates M(s) along with X(s); Example 1 has M()ss= L () and SR′= . (50) Substituting M for L in (43) then results in (s) (s) * In other words, the function w(s) uses the same ds i.i.d. uniforms to θE= [00 ]= EE [ [ |MM ]]= E [ 1 ( )], ()ss () () ssss () () () (46) generate both the S-valued basic random object X(s) and the S′-valued conditioning random object M , so they are dependent. Moreover, where the SS+CMC response function generalizing (44) is the condi- (s) while w(s) employs all ds uniforms to output X(s), function m(s) builds tional expectation (probability) fi M(s) from just the rst ds′ ≤ ds of them, which may require relabeling the 1()ss(MM () )=[EP00 () s | () s ]=(=1|)=(()=1| () s MXM () s Pd () s () s) (47) inputs. Example 1 considers q=1 criterion with 2 X()sss=(LC () , () )∈SR = ,soX(s) is a random vector. The for d in (2).By(46), SS+CMC produces an unbiased estimator of θ by (s) conditioning random object in Example 1 is M()ss= L (), which is a averaging identically distributed copies of 1 (M ), rather than copies ()ss () random variable lying in SR′= and is generated by function m()ss= l () of 0 as when applying only SS. The response 0 in (17) when not (s) (s) in (42) from just ds′ = 1 uniform U1 out of the ds=2 uniforms that w(s) applying conditioning is binary, but in general the SS+CMC response uses to produce X(s); the other uniform U2 is used by function c(s) in (42) 1 (M ) in (47) is not binary but rather lies in [0, 1] because it is a ff ()ss () to sample C(s).AsinExample 1, the spaces S′ and S in A3 can di er in 2 ff conditional probability. As σ(),SSss=Var[0()] by (21), replacing L(s) with general, so M(s) and X(s) may be di erent types of random objects, e.g., M(s) in (45) leads to random vectors of unequal dimensions. θ 2 2 We next develop the estimator of when applying SS+CMC+LHS Var[10()ss (MM () )] =σE (),SS s − [Var[()s () s ]] ≤ σ (),SS s. (48) (abbreviated SCL). (If SS has only s0 =1scenario, then SCL reduces to CMC+LHS.) By (46), the average of identically distributed copies of Thus, estimating θ(s) by averaging identically distributed copies of the SS+CMC response 1 (M ) from (47) rather than the SS response 0 1()ss(M ()), which is also the SS+CMC+LHS response, will be an unbiased ()ss () (s) fi shows that SS+CMC yields a variance reduction compared to SS for estimator of θ(s) in (17). Thus, we rst generate an LHS grid of uniforms each scenario s. V(),,sij(with dependent rows) as in (28) for each scenario ss=1,2,…, 0, Applying CMC crucially relies on being able to generate and but with ds′ columns instead of ds because computing (47) requires only the conditioning random object M in (50), which uses d′ uniforms compute the conditional expectation 1()ss(M ()) in (47), which depends (s) s rather than ds that 0 =(d X ) does in (25). Then apply the function on the particular forms of the basic random object X(s), the failure ()ss () m in (50) to each of the ns rows of the LHS grid to obtain ns function d in (2), and the conditioning random object M(s). Section 6.3 (s) dependent but identically distributed copies M =(m V , will show how to compute 1()ss(M ()) for a generalization of Example 1 (),si () s (),,1 si V ,…,V ′ ), 1≤in ≤ , of the conditioning random object M . with q ≥ 1 loads and capacities when ML()ss= (), the vector of loads. (),,2si (),, sids s (s) Other CMC work also assumes the basic random object is a vector of Next compute an (unbiased) estimator of θ(s) as basic variables. [26] conditions on various subsets of the basic ns l 1 variables, assumed to be mutually independent, and combines CMC θn(),SCL,sη()= ∑ 1()ssi()M (), , ns (51) with antithetic variates. When the basic variables are i.i.d. standard i=1 normals, [39] expresses the normal vector in polar coordinates and which is the average of dependent copies of non-binary values, in conditions on the angle, so the conditional expectation integrates out general. The SS+CMC+LHS estimator of the overall failure probability the radius, which is independent of the angle. CMC also applies for s0 lls0 θλθ= ∑s=1 ss( ) is θnSCL,η()=∑s=1 λθnss(),SCL, η( ). stochastic processes; e.g., to analyze a highly reliable system modeled As the response function 1()ss(M ()) in (47) inherits the boundedness as a continuous-time Markov chain, [53] conditions on its embedded l of the binary-valued 0(s), the estimator θn(),SCL,sη( ) of θ(s) for each discrete-time chain, and the conditional expectation replaces the scenario s satisfies the LHS CLT of [49] when Assumptions A2 and exponential holding time in each state visited with its conditional A3 hold: mean. n l To implement SS+CMC (which reduces to CMC when SS has only [θnθ(),SCL,sη ( )− () s ]⇒5 (0,1) as n →∞, ση(),SCLs / s (52) s0 =1scenario), for each scenario ss=1,2,…, 0, we generate M(),s i,

385 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394

m where we will give an expression for the asymptotic variance constant ∼〈〉r 1 s θn()= 1 (M〈〉r ) σ 2 in Section 6.1. We assume the s scenarios for SS are simulated ()s ∑ ()s (),si (),SCLs 0 ms i=1 (57) independently, so the overall SS+CMC+LHS estimator of θ satisfies the l CLT nθ[()−]/⇒(0,1SCL,ηη n θ σ SCL, 5 ) as n → ∞, where the asymptotic for the response function 1(s) in (47). Compared to the SS+rLHS variance is ∼〈〉r estimator without CMC in (39), the SS+CMC+rLHS estimator θn()s ( ) s0 λσ2 2 replaces the indicator in (39) with its conditional expectation 1(s). The 2 ss(),SCL θ σSCL,η =.∑ estimator of the overall failure probability in (16) from replication r is η 〈〉r 〈〉r s=1 s (53) ∼∼s0 fi θ θn()=∑s=1 λθnss()( ). The nal estimator of across all b replications l b ∼〈〉r is θnbθnSCrL,ηb , ()=(1/)∑r=1 ( ). Also, compute the sample variance of 6.1. Analyzing the effect of CMC on LHS 2 ∼()r ∼2 ⎡∼〈〉r ⎤ θn( ), 1≤rb ≤ ,asSn()= 1 ∑b ⎢ θ()− n θl () n⎥ . The next b b −1 r=1 ⎣ SCrL,ηb , ⎦ Recall that when not applying CMC, (36) shows that for each result, proven in Appendix C, provides an asymptotically valid SS+CMC scenario s, LHS removes the variability of the additive part of the SS +rLHS UCB, which reduces to a UCB for CMC+rLHS when SS has only indicator response in (35). When we further incorporate CMC, s0 =1scenario. generating the conditioning random object M(s) in (50) requires a vector U′=(UU , ,…, U) of d′ i.i.d. <[0, 1] random numbers. To 12 ds′ s Theorem 4. Under Assumptions A2 and A3, an SS+CMC+rLHS γ θ l analyze the SS+CMC+LHS response 1()ss(M ()) in (47) for estimating θ(s), asymptotic -level UCB for is BSCrL,ηb ,()=nθ SCrL,ηb , ()+ nτ b−1,γ ∼ we emphasize the dependence of M(s) on the vector U′ and approximate fi Snb()/ b , i.e., limnη→∞PB ( SCrL, ,b ( n )≥ θ )=γ as in (5) for any xed the response as an additive function of the uniform inputs U′ based on b ≥ 2 and SS allocation η. an ANOVA decomposition:

ds′ 6.3. SS+CMC+LHS for failures defined in terms of loads and 1()ss(MU () ( ′)) =θE () s +∑ () [1()ss (MU () ( ′)) Uθ j ] − () s + ϵ () s (U ′), capacities, with loads independent of capacities j=1 (54) fi where the residual ϵ(′()s U ) is de ned so that the above equality holds. As noted in Section 6, the key to applying CMC is being able to Then as shown in [52] and [49], we have that the asymptotic variance sample and compute the response 1()ss(M ()) in (47), which depends on fi 2 fi in (52) satis es σ(),SCLss= Var[ϵ() (U ′)]. The following result, proven in how we de ne the basic random object X(s), failure function d, and 2 conditioning random object M . We now show how to do this for a Appendix B, establishes that the asymptotic variance σSCL,η in (53) for (s) 2 generalization of Example 1 when Assumption A1 holds (i.e., failure is SS+CMC+LHS is no greater than σ in (31) for SS+LHS. SS+LHS,η fi de ned as a function d′ of q ≥ 1 loads and capacities) and M(s) is the Theorem 3. Under Assumptions A2 and A3, when both SCL and SS load vector. We first augment Assumptions A2 and A3. +LHS use the same stratification allocation η, we have that σσ2 ≤ 2 , where, for ϵ(,UU ,…, U) defined in (35), Assumption A4. For each scenario 1≤ss ≤ 0, the conditioning SCL,η SS+LHS,η ()sd 1 2 s random object in Assumption A3 is ML()ss= (). The function 2 2 σσ− ds 2q SS+LHS,ηηSCL, wR′()s : [0, 1] → in (32) used to generate the q loads L(s) and s 0 λ 2 capacities C decomposes into functions l : [0, 1]dsL, → Rq =Var[ϵ(,,…,)|(,,…,)]≥0.∑ s EUUUUUU[]M (s) ()s ()sdsd 1 2ss () 1 2 ′ c dqsC, R s=1 ηs (55) generating loads and ()s : [0, 1] → generating capacities such that dsL,,+=dd sC s and ⎛ ⎞ The key point here is that the conditioning in (47) leads to ⎜ ⎟ wlc′()sdsds(uu 1 , 2 ,…, u )=() ( uu 1 , 2 ,…, u ),() ( uddd +1 , u +2 ,…, u +d ) “ ” ss⎝ ,,,,,LsLsLsLsC⎠ 1()ss((′)MU () ) in (54) being a smoother response function than the indicator 0 (U) in (34). As a consequence, the additive approximation for (uu , , …, u ) ∈ [0, 1]ds . ()s 12 ds (54) can be a better fit than that in (35). This can result in SS+CMC q +LHS having much lower asymptotic variance than SS+LHS, as we will Thus, Assumption A3 has m()ss= l (),sodssL′ = d , , S′=R , and see in the numerical results of Section 7. The idea of smoothing a wSR* :[0,1]ds → × q in (50) satisfies w* (,uu ,…,)= u ()s ()sd 1 2 s response function through conditioning has also been fruitfully applied ((,,…,),(,,…,)wluu u uu u ) for (,uu ,…,)∈[0,1] u ds ()sd 1 2 ss()sd 1 2 ,L12 ds in other contexts, such as gradient estimation [54]. and w(s) in (25). fi Assumption A4 speci es the conditioning random object M(s) in A3 6.2. Combined SS+CMC+rLHS to construct UCB to be the vector L(s) of the q ≥ 1 criteria's loads. In the setting of A1, (33) specializes A2 to compute the failure indicator 0(s) in terms of L(s) We now provide details of the combined SS+CMC+rLHS approach and C(s) output by the function w′(s) in (32). Now A4 further stipulates θ to construct a UCB for under Assumptions A2 and A3. For each that w′(s) decomposes into functions l(s) and c(s) operating on disjoint sets scenario ss=1,2,…, 0, and each replication r =1,2,…,b, with b ≥ 2, of inputs, where l(s) outputs the loads and c(s) the capacities. (Example 1 〈〉r has q=1 and in (42), function l has input U and c takes input U .) let V(),,sij,1≤im ≤ s, 1≤jd ≤ s′,beanmdss× ′ LHS grid of uniforms (with (s) 1 (s) 2 Thus, if UUU,…, , ,…, U are d i.i.d. <[0, 1] random dependent rows) as in (38), but with ds′ columns instead of ds. LHS 1+1+ddsL,, sL ddsL ,, sC s variables, then grids across the s0 scenarios and b replications are independent. Then apply function m from (50) to each row of the LHS grid to get m (s) s L =(,,…,)∼l UU U F and dependent but identically distributed copies of the conditioning ran- ()ss () 1 2 dsL, () s dom object: C =(c UU , ,…, U )∼, G ()ssdd ()sL,, +1 sL +2 dd sL ,,+ sC () s (58) M〈〉r =(,m VV〈〉r 〈〉r ,…,), V〈〉r [1] [2] []q [1] [2] []q (),1s ()s (s ),1,1 (s ),1,2 (),1,sds′ where L()ssss=(LL () ,() ,…, L()) and C()ssss=(CC () ,() ,…, C() ), and we fi M〈〉r =(,m VV〈〉r 〈〉r ,…,), V〈〉r recall that F(s) and G(s) were de ned at the end of Section 4 as the (),2s ()s (s ),2,1 (s ),2,2 (),2,sds′ marginal CDFs of the load and capacity vectors, respectively, for ⋮⋮ ⋮ ⋮ ⋱ ⋮ scenario s. Consequently, as the two functions l(s) and c(s) use disjoint M〈〉r =(m VV〈〉r ,〈〉r ,…,). V〈〉r (),smsss()s (),sm ,1 (),sm ,2 (),smdss , ′ (56) sets of i.i.d. arguments, the following holds: fi For the rth replicated LHS for scenario s,de ne the estimator of the Proposition 1. Assumption A4 implies L(s) is independent of C(s). failure probability θ(s) in (46) as Proposition 1 ensures that L(s) is independent of C(s), but each vector

386 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394 may still have a dependence structure for its own q entries. Without A4, computed analytically or numerically for each [,kk12 ,…,] kp Assumption A2 and w′(s) in (32) allow L(s) and C(s) to be dependent. 1≤kk12 < <⋯< kqp ≤ such that a ≠0 in (59), each In structural reliability, loads are sometimes (but not always) 1≤p ≤q, and each scenario ss=1,2,…, 0. assumed to be independent of capacities [20, Chapter 1]. In many Finally putting (62) into (60) yields the following. NPP PSAs, loads are determined by the way in which a hypothesized Theorem 5. If Assumptions A1–A6 hold, then the SS+CMC+LHS accident evolves, whereas the capacities depend on the material response function in (47) is computable as properties and manufacturing variability of the components. Hence, q it may be reasonable to model loads as independent of capacities, as is [kk, ,…, k ] []k 1 ()=+L aaGLLL[0] ∑∑ [kk1, 2,…, kp ] 1 2 p ([k1] ,[k2] ,…,)p . assumed in the initial RISMC studies with a single criterion in [3] and ()ss () ()s ()s ()s ()s p=1 1≤kk1< 2<⋯< kp ≤q [19]. Moreover, [3] and [19] take the capacity distribution G to be the (s) (63) same for all scenarios s, although our development here does not require this. To compute the estimator in (51), we simply replace 1()ssi(M (),) with fi fi 1 (L ) using (63), where L =(l VV , ,… V ) with V We next further re ne how a failure is de ned. ()ssi (), (),si () s (),,1 si (),,2 si (),, sidsL, (),,sij from the LHS grid of uniforms in (28) but with dsL, columns instead of Assumption A5. The failure function d′:R2q → {0, 1} in Assumption ds, and function l(s) is from Assumption A4 and (58). Similarly, we A1 has the form 〈〉r 〈〉r compute the estimator in (57) by replacing 1()s (M(),si) with 1()s (L(),si) q using (63) with L 〈〉r =(l VV〈〉r ,〈〉r ,… V〈〉r ), where each V 〈〉r is [0] [,kk12 ,…,] kpllp []kk [] (),si ()s (),,1si (),,2si (),,sidsL, (),,sij d′(LC , ) =aaILC +∑∑ (∩l=1 { ≥ }), from the LHS grid of uniforms in (56) with d′ = d columns. p=1 1≤kk12 < <⋯< kp ≤ q (59) ssL, The number of terms in (63) with nonzero coefficient a[,kk12 ,…, kp] can [0] [,kk ,…, k] where a and each a 12 p are constant coefficients. grow exponentially in the number q of criteria. For example, a series Assumption A5 requires that the failure function d′ is a linear function of system (Example 2 in Section 2.1) uses d′ in (7),so(63) has 2q − 1 the indicators that the load exceeds capacity for each criterion in each nonzero summands; this can practically limit the number of criteria nonempty subset of criteria. In Section 2.1, Examples 2–6, which define that can be considered. In contrast, a parallel system (Example 3) has failure in terms of q loads and capacities, all satisfy Assumption A5.In d′ in (8),so(63) has only one nonzero summand; thus, as long as Example 2 (series system), (7) equals d′ in (59) with a[0] = 0 and [1,2,…,q ] ffi G()s can be computed e ciently, we are not restricted to small q.If [,kk ,…,] k p −1 a 12 p =(−1) for each 1≤p ≤q.InExample 3 (parallel system), the number of summands in (63) is overwhelming, we may instead [0] [1,2,…,q ] fi (8) corresponds to (59) with a = 0, a = 1,andallother obtain an estimator of an approximation to θ(s) by truncating the rst a[,kk12 ,…,] kp =0.InExample 4 (K-out-of-N:F system), (9) fits into (59) sum in (63) at some upper index qq′< ; see [20, Section 5.6.2] for with aa[0]==0 [kk12 , ,…, kp ] for each 1≤p

387 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394 these criteria's load data (sample sizes ≈180) output from a computer <[0, 1], where there may be dependence among the entries in W. code of a hypothesized LOCA. Based on the shape of the histogram of Then a copula K: [0, 1]q → [0, 1] is the joint CDF of W; i.e., each criterion's load data, we chose a lognormal for the marginal CDF KwwwPWwWwWw(, ,…,)=([1] ≤ ,[2] ≤ ,…,[]q ≤ ), of the PCT load, and marginal Weibull distributions for both CWO and 12 q 1 2 q q MLO loads. for (ww1 ,wq , …, w ) ∈ [0, 1] . (64) [1] For Models 1 and 2, the lognormal PCT load CDF F()s for scenario s =1,2,3,4, corresponds to exponentiating a normal random variable Now suppose we want to generate a load or capacity vector, generically [1] [1] [2] [q ] [i] denoted by Y =(YY , ,…, Y),withjointCDFF0, where each Y has a with mean μ()s = 7.35 + 0.01 s and standard deviation [1] marginal CDF Fi as specified in Section 7.1.BySklar'stheorem[55, σ()s = 0.14 + 0.01 s, where the parameter values for scenario s=1 are the maximum likelihood estimates (MLEs; Section 7.1 of [44]) from Theorem 5.3],thereexistsacopulaK such that Fy(, y ,…,)= y KFy ((),(),…,() Fy Fy) and K(,ww ,…, w) the PCT load data of [10]. To assess the appropriateness of our choice 0 12 q 1 1 2 2 q q 12 q [1] −1 −1 −1 F()s for scenario s=1, we applied the Kolmogorov-Smirnov (KS) and =(FF01 (), w12 F ( w2 ),…, Fqq ()) w ; K is unique when each marginal Fi is chi-square goodness-of-fit tests (e.g., Section 5.7 of [44]) with the data continuous. Thus, any joint CDF has a copula, and we may combine a from [10], which yielded p-values of 0.45 and 0.39, respectively; thus, copula and marginal CDFs to construct a joint CDF. Hence, to generate Y fi [1] at signi cance level 0.05, we do not reject the CDF F()s for scenario s=1. using W ∼ K in (64),whereK is the copula of F0, note that because each For Models 3 and 4, we changed the parameters of the PCT load's W []i ∼[0,1< ],wesimplyset [1] lognormal CDF for scenario s to μ()s =7.4+0.1s and []i −1 []i [1] Y =FWi ( ) ∼ Fi , for each i. (65) σ()s = 0.01 + 0.01 s, which are used in [24]. For Models 1–4, the CWO load for scenario s has marginal Weibull If the marginals Fi have been specified (e.g., as in Section 7.1), the [2] [2] [2] β()s next issue is how to construct a copula K (appropriate for the problem) CDF Fx()ss()=1−exp{−(/xα() ) } for x ≥ 0, where q [2] [2] and to sample W ∼ K. One approach begins with a CDF H0 on R α()s = 0.010 + 0.005 s and β()s = 0.90 + 0.02 s are the scale and shape parameters, respectively. For scenario s=1, the parameter values are having continuous marginals Hi. For a random vector [1] [2] [q ] the MLEs from the CWO data in [10], and the KS and chi-square tests J =(JJ , ,…, J )∼ H0, let do not reject this choice of CDF (p-values 0.22 and 0.61, respectively). W = (WW[1] , [2] , …, W [qi ] ) with each W [] = HJ ( []i ) ∼< [0, 1]. [3] i (66) For the MLO load, its marginal Weibull CDF F()s for scenario s in [1] [2] [q] Models 1–4 has parameters α [3] = 0.35 + 0.3 s and β[3] = 0.82 + 0.03 s. In general, W ,,…,WWare dependent, and their joint CDF is the ()s ()s ff The parameter values for scenario s=1 are the MLEs from the MLO copula K in (64) induced by H0. Our experiments use two di erent – load data in [10], and our CDF choice for s=1 is not rejected by the KS copulas. Models 1 and 3 employ a Gaussian copula [55, pp. 190 191], and chi-square tests (p-values 0.82 and 0.08, respectively). in which H0 is a multivariate normal CDF. Models 2 and 4 use a Although the load CDFs depend on the scenario s, we assume that Student-t copula [56], where H0 is a multivariate Student-t CDF. fi the capacity CDFs remain the same for all s, as is the case in [3] and To de ne a Gaussian copula, let H0 be the 5q(,0 Σ) normal CDF []kk [] with mean vector 0 and covariance matrix Σ =(Σij[,]ij : , =1,2,…, q), [19],soG()s = G for each criterion k =1,2,3. [,]ii As in [3,19], we take the PCT capacity C[1] to have a triangular CDF where each Σ = 1. Thus, each marginal is HΦi = , the standard (i.e., G[1] with support [ab[1], [1] ] = [1800, 2600] and mode c[1] = 2200. The mean 0 and variance 1) univariate normal CDF, and Σ is also the mode c[1] is the fixed limit prescribed by the NRC [4] for the PCT (Pearson linear) correlation matrix. Now let [1] [2] [q ] criterion, and the support parameters are symmetric and separated Z0=(ZZ , ,…, Z )∼5q ( ,Σ). from the mode by approximately 20%. We can generate the (column) vector Z by multiplying a qq× ⊤ For the CWO and MLO capacities C[2] and C[3], respectively, we also matrix Γ satisfying Σ = ΓΓ (e.g., Γ is a Cholesky factor of Σ) with a q- assume marginal triangular distributions with parameters following a vector D of i.i.d. standard normals: similar approach as the one used for PCT capacity. Because the NRC ZD=.Γ (67) specifies a 1% fixed limit for CWO (see Section 1), we set the CWO Then let W be as in (66) with JZ= and each HΦ= . The Gaussian capacity CDF G[2] as triangular with support [ab[2], [2] ] = [0.8, 1.2] and i copula K with input (Pearson linear) correlation matrix Σ is finally mode c[2] = 1. Finally, the NRC fixed limit for MLO is 17%, so the MLO given by (64), and W ∼ K. capacity CDF G[3] has support [ab[3], [3] ] = [13.6, 20.4] and mode For a Student-t copula, let χ 2 be a univariate chi-squared random c[3] =17. ν variable with ν ≥ 1 degrees of freedom, independent of the Z0∼(,5q Σ) in (67). We then obtain a multivariate Student-t random vector with ν d.f. as 7.2. Specifying dependence structures through copulas 2 J =/Z νχν . (68) So far we have discussed the marginal CDFs of loads and capacities, but we further need to specify their joint distributions F(s) and G.Inan Let W be as in (66) with each Hiν= Υ , the univariate Student-t CDF actual NPP PSA with a computer code, each code run produces a vector with ν d.f., and then (64) defines the Student-t copula K with ν d.f. and of the loads for PCT, CWO, and MLO, so the q=3 criteria's loads have a input dispersion matrix Σ, which is the (Pearson linear) correlation particular stochastic dependence. For example, if the PCT load is matrix of the normal Z used to build J in (68).Ifν > 2, then the unusually high in one run, then the same is likely true for the CWO and covariance matrix of J is Cov[J ] = (νν /( − 2))Σ, and the Student-t MLO. But as our numerical experiments do not use such a computer vector J in (68) also has (Pearson linear) correlation matrix Σ. But code, we require a mechanism to generate observations of the load when ν ≤ 2, each J[i] has infinite variance, so the Pearson correlation of vector with a reasonable dependence structure. Moreover, the criteria's J is undefined. capacities in a run should also be statistically dependent. But as A key difference between the Gaussian and Student-t copulas lies in Assumption A4 implies by Proposition 1, we have that loads are their tail dependence, as we now explain for dimension q=2. Suppose stochastically independent of the capacities. that Y =(YY[1] , [2]) is a bivariate output random vector with copula K

We specify the dependence among the loads and among the and marginal CDFs F1 and F2. Now define [2] −1 [1] −1 capacities via copulas [42]. While (Pearson) correlation measures only υ =limp→1PY ( > F2 ()| p Y > F1 () p) as the limiting conditional linear dependence, a copula fully characterizes the dependence struc- probability that the second entry of Y exceeds its p-quantile given that ture for given marginal CDFs. To define a copula, we start with a the first is larger than its p-quantile. If the marginals F1 and F2 are random vector W =(WW[1] , [2] ,…, W [q ]) in which each marginal is continuous, then (65) implies that the value of υ depends only on the

388 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394

[1,2] [1,3] [2,3] copula K and not on F1 and F2. A Gaussian copula has υ =0, whereas dispersions as ΣL =0.92, ΣL =0.96, and ΣL =0.85, which are υ >0for a Student-t copula. Hence, a Gaussian copula has asymptotic used across every scenario for the loads. For the input matrix Σ = ΣC of independence in the tails, but a Student-t copula tends to produce joint the capacity (Gaussian and Student-t) copulas, we let [1,2] [1,3] [2,3] extreme events. For more details, see Section 3.2 of [56]. ΣΣΣCCC===0.85 for all models and scenarios. As Section 7.1 already specified marginal CDFs for the loads and capacities, our next task is to pick a copula K so that the generated 7.3. Generating loads and capacities (load or capacity) vector Y =(YY[1] , [2] …, Y [q ]) has a desired dependence structure. For example, we may want a particular tail dependence υ for We now combine the marginals chosen in Section 7.1 with copulas Y. We may also want Y to have a target (Pearson linear) correlation from Section 7.2 to specify the functions w′(s), l(s), and c(s) generating the ij ij i j matrix ρ =(ρij[,] : , =1,2,…, q), where ρ[,] = Cov[YY[][] , ]/ load and capacity vectors for scenario s in Assumption A4 and (58).In (Var[YY[]ij ]Var[ [] ]) 1/2, which depends on both K and the marginals. If general, to evaluate a d-dimensional integral, Monte Carlo methods we have selected a copula family (e.g., Gaussian), it still remains to become attractive alternatives to numerical quadrature when d is high, fi specify its parameters (e.g., the input correlation matrix Σ of a so we arti cially increase the number dssLs=+dd,,C of input variables fi Gaussian copula) to achieve the target ρ, assuming it is indeed of function w′(s) in Assumption A4.We rst explain how to generate an attainable (Theorem 5.25 of [55]). There usually is no closed-form observation of the load vector for scenario s using a Gaussian copula fi fi formula for ρ in terms of the copula parameters, and we can instead use with the input correlation matrix ΣL speci ed in Section 7.2.Wede ne [1] [2] [3] a sampling-based search procedure, e.g., [57], to identify the copula the load function l(s) in (58) to have dsL,,=++ddd sL sL, sL, i.i.d. <[0, 1] parameters to obtain the target ρ. inputs UU,,…, U , with each d[]k =10, and let D =(DDD[1] , [2] , [3]) 12 dsL, sL, L LLL Another possible dependence measure to specify for Y is rank [1] [1] [2] [1] dsL, −1 [1] [2] ddsL, + sL, −1 [2] correlation, which has different variants [55, Section 5.2.2]. The with DL = ∑j=1 ΦU()/jsL d, , DL = ∑ [1] ΦU()/jsL d, , and jd=+1sL, []i []j (population) Spearman rank correlation of Y ∼ F and Y ∼ F is ddd[1]++[2] [3] i j D[3] = ∑ sL, sL, sL, ΦU−1()/ d[3] . Thus, the entries of D are i.i.d. fi [,]ij []i []j []i []j 1/2 L jd=++1[1] d[2] jsL, L de ned as ρS = Cov[FYi ( ), FYj ( )]/(Var[ FYi ( )]Var[ FYj ( )]) . The sL, sL, [i] [j] standard normal. Then, as in (67),define the vector Kendall (tau) rank correlation of Y and Y is [1] [2] [3] [,]ij [] i ∼∼[]i []j []j ∼∼[]ij [] ZL=(ZZZ LLL , , ) as ZDLLL= Γ , where ΓL is a Cholesky factor of ΣL. ρτ =EYYYY [sgn(( − )( − ))], where (,YY) is an independent [1] [2] [3] Letting J = ZL, we obtain WW==(,LWWW LLL , ) as in (66), with copy of (,YY[]ij []), and sgn(x ) = 1 (resp., −1 and 0) if x > 0 (resp., x < 0 each HΦi = .By(65), the load vector and x=0). When the marginal CDFs Fi are continuous (which we will [1] [2] [3] [1] −1 [1] [2] −1 [2] [3] −1 [3] assume for the rest of the paragraph), both rank correlations of Y L()s=(LLL () sss ,() ,() )=(( FWFWFW() sLsLsL )( ),(() )( ),(() )( ))∼ F( s) depend only on the copula and not on the Fi [55, Proposition 5.29],in (69) contrast to the (Pearson linear) correlation ρ[,ij], which depends on has a Gaussian copula with input correlation matrix Σ , and each L [k] both. For a Gaussian copula, the Spearman and Kendall rank correla- L ()s [,ij] [k] tions relate to the input Pearson correlation Σ of the Gaussian copula has marginal CDF F()s given in Section 7.1. fi [,]ij [,]ij [,]ij [,] ij We apply similar ideas to de ne the function c(s) in (58) to get an by ρS = (6/πΣ )arcsin( /2) and ρτ = (2/πΣ )arcsin( ); see [55, Theorem 5.36]. Thus, if we want to generate Y with a Gaussian copula observation of the capacity vector for scenario s when employing the fi to have a specified Spearman or Kendall rank correlation, we choose Gaussian copula with input correlation matrix ΣC speci ed in Section [1] [2] [3] the input correlation matrix Σ of the normal Z in (67) to have entries 7.2. We use dsC,,=++ddd sC sC, s,C i.i.d. <[0, 1] inputs [,]ij [,]ij [,]ij [,]ij UU,…, , with each d[]k = 1, to obtain normal vector Σπρ=2sin(S /6) or Σπρ=sin(τ /2), respectively; then use (66) dddsL,,+1sL + sC, sC, with JZ= and HΦ= , and build Y via (65). For a Gaussian copula, the [1] [2] [3] ⊤ i ZC=(ZZZ CCC , , ) as ZDCC= Γ C, where ΓΓCC= ΣC and difference between the Spearman rank correlation and the input −1 −1 −1 DCd=(ΦU (+1 ), ΦU ( d+2 ), ΦU ( d+3 )). Letting J = ZC, we obtain Pearson correlation is quite small [55, p. 216]: |ρΣ[,]ij− [,] ij | ≤ 0.0181. sL,,,sL sL S WW==(,WWW[1] [2] , [3] )as in (66), with each HΦ= .By(65), the This fact is implicitly exploited by [58], which essentially employs a C CCC i capacity vector Gaussian copula to generate a sample with a specified sample [1] [2] [3] [1] −1 [1] [2] −1 [2] [3] −1 [3] Spearman rank correlation. As [55, Example 5.54] notes, if we instead C()s=(CCC , , )=(( GWGWGW )(CCC ),( )( ),( )( ))∼G want to generate vector Y with a Student-t copula, we can similarly (70) calibrate the copula's input dispersion matrix Σ so that Y has a target Kendall rank correlation, as follows. For a Student-t copula, the has Gaussian copula with input correlation matrix ΣC, and each [,]ij [,] ij CG[]kk∼ [] from Section 7.1.As(,…,UU) and (,…,UU) Kendall rank correlation is again ρτ = (2/πΣ )arcsin( ). Hence, to 1 dsL, dddsL,,+1sL + sC, [,ij] are independent as in Assumption A4, L(s) and C(s) are also (Proposition achieve a target Kendall value ρτ , assign the copula's dispersion [,]ij [,]ij 1). Note that w′(s) in A4 uses dssLsC=+=3dd,,3 i.i.d. uniforms to matrix Σ to have entries Σπρ=sin(τ /2); apply (66) with Hiν= Υ , and use (65) to obtain Y. The Spearman rank correlation for a Student-t generate X()sss=(LC () , () )when employing Gaussian copulas. copula is not known in closed form. When instead assuming Student-t copulas, we make the following l c In our experiments, we chose the input (correlation or dispersion) changes to the load and capacity functions (s) and (s). We now let [1] [2] [3] [1] [2] [3] matrix Σ of the Gaussian or Student-t copula so that the generated dsL,,=+++ddd sL sL, sL, 1 and dsC,,=+++ddd sC sC, sC, 1, with each d[]k =10 and d[]k = 1 as before. We set χΨU2−1=() and output random vector Y in (65) has an appropriate (Pearson) correla- sL, sC, νL, νdsL, tion matrix ρ. When Y is the load vector, which has size q=3, we χΨU2−1=( ), where Ψ is the CDF of a univariate chi-squared νC, νddsL,,+ sC ν calibrated Σ =Σ = (Σij[,]ij : , = 1, 2, 3) so as to match an estimated LL random variable with ν = 3 d.f. Then let J =/Z νχ2 with Z as (Pearson) correlation matrix ρ =(ρij[,]ij : , =1,2,3) with ρ[1,2] =0.85, L νL, L L L L before, and apply (66) with each H = Υ to get WW= . Finally, (69) ρ[1,3] =0.87, and ρ[2,3] =0.83, which [10] computed from a sample of iν L L L yields a load vector having a Student-t copula with ν = 3 d.f. and the the (PCT, CWO, MLO) load vectors produced from multiple runs of a input dispersion matrix Σ specified in Section 7.2. For the capacities, nuclear-specific computer code. Applying the search algorithm of [57] L J Z νχ2 Z WW with the MLEs of the parameters of the specified marginal load CDFs we instead use =/C νC, (with C as before) in (66) to get = C [1,2] [1,3] [2,3] and apply (70). Thus, w′ in A4 takes d =+=3dd 5 uniforms to (see Section 7.1), we found ΣL =0.92, ΣL =0.96, and ΣL =0.86 (s) ssLsC,, produce L(s) and C(s) with Student-t cields a load vector having a for the Gaussian copula to yield the target load correlations ρL (to two decimal places of accuracy). We used these input correlations for the Studentopulas. loads across every scenario for Models 1 and 3. The Student-t copula in Our experiments use the series-system failure function d′ from (7) [0] [,kk ,…,] k p −1 Models 2 and 4 have ν = 3 d.f. in (68), and we identified the input in Assumption A5,so(59) has a = 0 and each a 12 p =(−1) .To ensure the tractability of the response function (63) when CMC is

389 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394 applied, Assumption A6 requires that we can compute the marginal Table 2 CDF G[,kk12 ,…, kp] in (61) for each subcollection CC[]kk12,,…, [ ] C [ kp] of Results for Model 1 (θ = 0.0254630). 1≤p ≤3 criteria's capacities, which we next show holds for both Method nb95% AHW Coverage PCD CV VRF SS/x copulas. When p=1, we have a univariate CDF G[k], which is trivial to [k] evaluate as each G is triangular. For p=2 with the PCT and CWO SS 400 1 2.16E−02 0.900 0.578 5.51E−01 1.00 capacities C[1] and C[2], note that (70) implies that their marginal joint 1600 1 1.14E−02 0.908 0.929 2.74E−01 1.00 − − CDF satisfies 6400 1 5.68E 03 0.928 1.000 1.38E 01 1.00 25,600 1 2.82E−03 0.936 1.000 6.90E−02 1.00 −1 −1 − − GxxPCxCxPGW[1,2]( , )= ([1] ≤ ,[2] ≤ )= (([1] ) ([1] )≤ xGW ,([2] ) ([2] )≤ x ) SS+CMC 400 1 1.40E 02 0.846 0.785 3.51E 01 2.45 12 1 2 CC1 2 1600 1 7.31E−03 0.904 0.998 1.78E−01 2.37 [1] [1] [2] [2] =(≤(),≤())PWCC G x1 W G x2 6400 1 3.64E−03 0.931 1.000 8.78E−02 2.45 − − =(()≤(),()≤())PHJGxHJGx[1] [1] [2] [2] 25,600 1 1.82E 03 0.937 1.000 4.44E 02 2.41 1 12 2 SS+rLHS 400 10 2.26E−02 0.923 0.567 5.13E−01 1.15 [1] −1 [1] [2] −1 [2] =(≤(()),≤(()))PJ HGx1 1 J HGx2 2 1600 10 1.14E−02 0.923 0.940 2.52E−01 1.18 − − (71) 6400 10 5.69E 03 0.936 1.000 1.26E 01 1.20 25,600 10 2.84E−03 0.944 1.000 6.27E−02 1.21 by (66). When the capacities are based on a Gaussian (resp., Student-t) SS+CMC 400 10 1.36E−02 0.871 0.820 3.09E−01 3.17 copula, the vector J used in (66) and (71) is multivariate normal (resp., +rLHS − − multivariate Student-t) and each H is a standard univariate normal 1600 10 6.77E 03 0.928 0.999 1.48E 01 3.42 i 6400 10 3.32E−03 0.936 1.000 7.38E−02 3.47 (resp., univariate Student-t) CDF, so (71) is a multivariate normal 25,600 10 1.64E−03 0.941 1.000 3.67E−02 3.55 (resp., multivariate Student-t) CDF evaluation, which can be handled numerically. Similarly, we can numerically compute the other marginal [1,3] [2,3] [1,2,3] joint CDFs G , G , and G in Assumption A6. These computa- Table 3 tions are somewhat involved, but in an actual RISMC analysis, the LHS results for Model 3 (θ = 0.0464784). computer code l in (58) generating loads typically takes substantially (s) Method nb95% AHW Coverage PCD CV VRF SS/x more time, making the overhead of CMC negligible. SS+rLHS 400 10 7.40E−03 0.777 0.616 1.15E−01 1.39 7.4. Discussion of Monte Carlo results 1600 10 4.43E−03 0.793 0.395 5.59E−02 1.48 6400 10 2.20E−03 0.921 0.801 2.67E−02 1.67 25,600 10 1.08E−03 0.936 1.000 1.30E−02 1.73 We ran Monte Carlo experiments with our four models, whose SS+CMC 400 10 8.59E−04 0.933 1.000 1.06E−02 161.39 differences are summarized in Table 1. Because of the analytical +rLHS tractability of our models (which would not be the case for a real 1600 10 3.45E−04 0.942 1.000 4.15E−03 268.13 NPP analysis), we are able to numerically evaluate various multi- 6400 10 1.62E−04 0.940 1.000 2.00E−03 299.20 25,600 10 7.91E−05 0.942 1.000 9.79E−04 307.71 dimensional integrals to determine the true failure probability θ in (3) and (16). The last column of Table 1 shows that for our models, the θ ff value of does not vary much for the di erent copulas, but we will see Table 4 below that the performances of the Monte Carlo methods are affected, Results for Model 4 (θ = 0.0464744). sometimes substantially, by the change in the dependence structure. (Section 7.5 will give results for another model demonstrating sig- Method nb95% AHW Coverage PCD CV VRF SS/x nificant differences in θ as we change the copula.) SS 400 1 9.25E−03 0.864 0.345 1.38E−01 1.00 Tables 2–4 present results from our Monte Carlo experiments with 1600 1 5.08E−03 0.879 0.394 6.89E−02 1.00 Models 1, 3, and 4, respectively. We omit the results for Model 2 as 6400 1 2.59E−03 0.919 0.695 3.45E−02 1.00 they are very similar to those for Model 1. Our experiments construct 25,600 1 1.31E−03 0.934 0.992 1.73E−02 1.00 − − upper confidence bounds B(n) satisfying (5) using four different SS+CMC 400 1 2.56E 03 0.946 0.725 3.39E 02 16.63 1600 1 1.30E−03 0.954 0.997 1.66E−02 17.11 combinations of simulation methods: SS (see Section 4), SS+CMC 6400 1 6.37E−04 0.947 1.000 8.44E−03 16.70 (developed in [25]), SS+rLHS (Section 5.2), and SS+CMC+rLHS 25,600 1 3.21E−04 0.951 1.000 4.25E−03 16.68 (Section 6.2). Table 3 includes Model 3's results for only SS+rLHS SS+rLHS 400 10 7.82E−03 0.798 0.581 1.16E−01 1.41 and SS+CMC+rLHS, which also appear in [40], as the other methods’ 1600 10 4.55E−03 0.793 0.404 5.79E−02 1.42 6400 10 2.34E−03 0.911 0.756 2.87E−02 1.45 results do not differ significantly from those for Model 4. We varied the 25,600 10 1.19E−03 0.934 0.996 1.45E−02 1.44 v total sample size n = 4 × 100 for v =1,2,3,4. For each n and each SS+CMC 400 10 1.25E−03 0.947 0.998 1.48E−02 87.41 Monte Carlo method, we ran 104 independent experiments. The +rLHS stratification allocation has η =0.25 for each scenario s =1,2,3,4. 1600 10 5.63E−04 0.947 1.000 6.79E−03 102.71 s − − For rLHS, we used b=10 independent replicates of LHS samples within 6400 10 2.78E 04 0.949 1.000 3.34E 03 106.44 25,600 10 1.36E−04 0.948 1.000 1.67E−03 108.48 a scenario, so for each scenario s, each replicate has size mηnbs =/s . Methods that do not require replications to create an upper confidence bound are denoted with b=1. For the safety requirement (4), we set 4 θ0 =0.05 and γ =0.95. As seen in Table 1, all four models have θθ< 0, half-width across the 10 experiments, where the half-width is the although our experiments do not use this knowledge. difference between the 95% UCB and the point estimate of θ. For a UCB In Tables 2–4, the column labeled “95% AHW” presents the average B(n) and overall sample size n, the coverage is the probability PBn(()≥ θ). As noted throughout the paper, each method's UCB Table 1 satisfies (5), so the coverage converges to the nominal level γ =0.95 Summary of model differences. as n → ∞. But for a fixed n, the coverage may differ from γ.We 4 Model PCT load CDF parameters Copula θ estimate the coverage as the fraction of the 10 experiments in which B()≥n θ, and one gauge of the convergence of a method is how far its 1 Based on data from [10] Gaussian 0.0254630 coverage is from γ. The probability of correct decision (PCD) is 2 Based on data from [10] Student-t with ν = 3 d.f. 0.0254639 estimated as the fraction of the 104 experiments that the decision rule 3 Taken from [24] Gaussian 0.0464784 “ ” 4 Taken from [24] Student-t with ν = 3 d.f. 0.0464774 in (6) correctly determined that θθ< 0. The column CV gives the coefficient of variation of the point estimate of θ, computed as the

390 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394 sample standard deviation of the point estimate of θ across the 104 7.5. Additional numerical results experiments divided by the numerically-computed value of θ in Table 1. For each particular method x, the last column presents the While Table 1 displays little difference in the exact values of θ for variance-reduction factor (VRF), which for a given overall sample size the Gaussian and Student-t copulas, this is not always the case, as we n is the ratio of the sample variances for the base case of SS and for will see later. But first, we explain why the two copulas lead to method x. The VRF gives the factor by which the overall sample size for essentially the same values for θ for Models 1 and 2 and for Models []k s0 []k []k method x can be reduced compared to SS to achieve about the same 3 and 4. For each criterion k =1,2,3, let θλPLC= ∑s=1 ss(≥() ()s ) be AHW. For example, in Table 4, SS+CMC has VRF ≈ 16, and its AHW the marginal failure probability for criterion k by itself. Models 1–4 all [2] [3] for n=400 (resp., 1600) is about the same as the SS AHW for n=6400 have that θ = 2.13E−15 and θ = 2.73E−6. For criterion k=1 (PCT), [1] [1] (resp., 25,600). Models 1 and 2 (resp., 3 and 4) have θ = 0.0255 (resp., θ = 0.0465), [2] [3] Tables 2–4 clearly show that each method's performance depends which overwhelm θ and θ . Thus, the overall failure probability θ is on the model. For example, for small n in Model 1 (and 2, not shown), mainly determined by PCT, so the particular dependence structure the coverages for the methods with CMC are lower than those for the among the criteria has little impact. corresponding approaches without CMC, but the opposite is true for To demonstrate that differing copulas can yield substantial changes Models 3 and 4. Also, each method exhibits variations across models in in θ, we altered the parameters of the load distributions. For each the sample size n at which convergence appears to have been roughly scenario s, the lognormal CDF parameters for criterion k=1 are instead [1] [1] achieved (as defined by coverage being within 5% of the nominal level μs()s = 7.33 + 0.1 and σ()s = 0.01 + 0.01 s; the Weibull parameters for [2] [2] γ =0.95). For example, SS+rLHS appears to approximately converge by criterion k=2 (CWO) are α()s = 0.22 + 0.005 s and β()s = 0.90 + 0.02 s; n=400 for Model 1; in contrast, Model 3 requires n=6400. However, by and the Weibull parameters for criterion k=3 (MLO) are [3] [3] n=6400, the methods SS and SS+rLHS seem to have nearly converged αs()s =3.1+0.3 and β()s = 0.82 + 0.03 s. For the loads, we slightly for all models, whereas SS+CMC and SS+CMC+rLHS appear to need [1,2] [1,3] changed the input correlations to ΣL =0.93, ΣL =0.96, and n=1600. [2,3] ΣL =0.84 for the Gaussian copula, and the input dispersions to But even with the performance differences across models for each [1,2] [1,3] [2,3] ΣL =0.94, ΣL =0.97, and ΣL =0.83 for the Student-t copula, in method, Tables 2–4 also illustrate some universal trends. For all [1,2] [1,3] both cases producing load target correlations ρL =0.83, ρL =0.84, models, each combined VRT has VRF > 1, so we see the effectiveness [2,3] and ρL =0.81. The capacities' marginal distributions and copulas are of combining methods. This agrees with the asymptotic theory estab- the same as in Models 1–4. With these parameters we now have that lished in Theorems 1 and 3 for SS+LHS and SS+CMC+LHS, respec- θ[1] = 0.0201, θ[2] = 0.0201, and θ[3] = 0.0208, so no single criterion's tively. In particular, the combination of SS+CMC+rLHS provides the marginal failure probability dominates the others, in contrast to what most variance reduction for each model, with VRFs of about 300 and occurs for Models 1–4. 100 for Models 3 (Table 3) and 4 (Table 4), respectively. The VRFs for Table 5 presents the values of θ, computed numerically, for the SS+CMC+rLHS are more modest (about 3.5) for Models 1 (Table 2) Gaussian and Student-t copulas. Now we see a distinct difference, with ff and Table 2 (not shown). Thus, the e ectiveness of the method depends θ decreasing by about 5% when changing the copula from Gaussian to on the marginal distributions (compare Models 1 and 3) and the copula Student-t for system failure defined as for a series system. (Models 3 and 4). We can observe an even greater difference in θ between the copulas For all four models, the VRF for SS+CMC+rLHS exceeds the by changing the definition of a failure from that for a series system in product of the VRFs for SS+CMC and SS+rLHS, illustrating the (7) to instead be for a parallel system, as in (8). As seen in the bottom ff synergistic e ect of integrating LHS with CMC. As explained in of Table 5, now θ increases by about 19% when changing the copula ff Section 5.1, LHS can be particularly e ective when the response is a from Gaussian to Student-t. To explain the larger change when failure nearly additive function of the input random variables. Without CMC, is defined by (8) instead of (7), note that the marginal failure the response function (35) is an indicator, for which an additive probabilities θ[k] are fairly small (≈0.02), so the tail dependence plays fi approximation can be a poor t, so LHS by itself may not reduce an important role in determining θ. As we discussed in the paragraph variance by much. But incorporating CMC leads to a smoother after (68)inSection 7.2, the two copulas give rise to very different response function (47) through the integration performed for the dependence behaviors in the tails. Because the Student-t copula fi conditional expectation, so its additive t(54) can be much better; as asymptotically has non-zero tail dependence, if one criterion is Section 6.1 notes, LHS can then produce substantially more variance violated, then there is a reasonable chance that the others are as well, – reduction, as Tables 2 4 show. leading to higher θ when failure is defined for a parallel system. In fi Another bene t of reduced variance becomes evident when exam- contrast, the asymptotic tail independence of the Gaussian copula does ining the PCD. Many application domains critically rely on a high not produce such joint behavior, so θ is notably smaller. probability of correctly deciding if the safety requirement θθ< 0 in (4) holds. But it can be difficult to achieve high PCD when θ is close to θ0, as in Models 3 and 4 where θ ≈ 0.0465 and θ0 =0.05. In general it is 8. Concluding remarks easier for the decision rule (6) to make a correct determination when the point estimator of θ has small statistical error. Because the error We combined variance-reduction techniques to estimate a failure asymptotically decreases at rate n−1/2 by virtue of the relevant CLT, the PCD converges to 1 as n → ∞, as seen throughout Tables 2–4 for each Table 5 method, so we can ensure high PCD by having a large sample size. But Values of θ for another set of distribution parameters for the loads with different copulas ff fi this may not always be feasible, e.g., when a simulation run is and di erent de nitions for failure. extremely time-consuming. Instead, we can also achieve higher PCD Copula Failure definition θ for a fixed n by applying VRTs that reduce the error. For example, Table 4 shows that for n=400, method SS has PCD = 0.345, whereas SS Gaussian Series system, Eq. (7) 0.0493 +CMC+rLHS has PCD = 0.998. Thus, the much smaller variance for SS Student-t Series system, Eq. (7) 0.0470 Gaussian Parallel system, Eq. (8) 6.83E−4 +CMC+rLHS results in a substantially higher chance of making a Student-t Parallel system, Eq. (8) 8.11E−4 correct decision for the same sample size.

391 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394 probability. We developed the methods in a general setting, where a that the models in Section 7 are synthetic examples, although they are failure is defined in terms of a basic random object, which may be a loosely based on data from actual nuclear computer codes [10,3,19]. random vector or a more general random object, e.g., a stochastic While SS+CMC+LHS is guaranteed to outperform SS+LHS in terms of process. We also specialized our approaches to the setting in which a asymptotic variance, it would be interesting to see how much our failure is specified as a function of loads and capacities of q criteria techniques reduce variance when using actual nuclear computer codes. (e.g., as in the RISMC problem for a nuclear PSA), and we exploited the As noted in Section 1, LHS is most appropriate for estimating measures added structure to obtain readily implementable techniques. The of central tendency, e.g., a mean, so adding LHS to SS+CMC may not combination SS+CMC+LHS can be especially effective, substantially be effective in sufficiently reducing variance when estimating rare- reducing variance compared to SS, SS+CMC, and SS+LHS, as shown in event probabilities, smaller than say 10−4. In such cases, other Monte the numerical experiments in Section 7. To account for the statistical Carlo methods, such as importance sampling or splitting [14, Chapter error of our Monte Carlo estimators, we further devised upper VI], may be needed. confidence bounds for θ, and we proved the asymptotic validity (as Developing appropriate stochastic models for multivariate capaci- the total sample size n → ∞) of the UCBs when applying SS+rLHS and ties also deserves further investigation. The marginal distributions and SS+CMC+rLHS in Theorems 2 and 4, respectively. pairwise (Pearson linear) correlations do not fully specify the entire Theorem 1 (resp., 3) further established that SS+LHS (resp., SS dependence structure, and there can be infinitely many joint distribu- +CMC+LHS) has smaller asymptotic variance than SS (resp., SS+LHS). tions fitting that limited information. On the other hand, replacing We also provided analysis (Sections 5.1 and 6.1) giving insight into why correlations with a copula does completely determine the joint combining CMC with LHS can be so powerful (e.g., Table 3 shows SS distribution for given continuous marginals. In our numerical experi- +CMC+rLHS reduces variance by about a factor of 300 compared to ments, we assumed that each criterion's (PCT, CWO, and MLO) SS). It is known (e.g., Section 10.3 of [22]) that LHS can tremendously capacity has a marginal triangular distribution, with the joint depen- reduce variance when the response function whose mean is being dence structure defined by a Gaussian or Student-t copula. We chose estimated is well approximated by an additive function of the input marginal triangular capacity distributions because [3,19] use this when random numbers. But when estimating a probability without applying considering only a single criterion, PCT. As our numerical results CMC, the response is an indicator function, and the additive approx- indicate, the value of the failure probability θ and the performance of imation to it can be a poor fit. Incorporating CMC smooths the our developed methodologies depend on the particular stochastic response function through a conditional expectation (so the response structure specified, so it is critical to build accurate models. is no longer necessarily binary-valued), which can lead to a better additive fit and consequently lower variance when combining CMC with LHS. In our development of SS+CMC+LHS in Section 6.3, where Acknowledgements failure is defined in terms of loads and capacities of q criteria, Assumption A4 implied that the load vector is independent of the This work has been supported in part by the National Science capacity vector (Proposition 1), which holds in many applications. (If it Foundation under Grants No. CMMI-1200065, DMS-1331010, and does not for a particular problem, SS+CMC+LHS can still be applied CMMI-1537322. Any opinions, findings, and conclusions or recom- but it then requires working instead with the response function (47) mendations expressed in this material are those of the authors and do which may be less tractable than (63).) Our methods can be applied to not necessarily reflect the views of the National Science Foundation. high-dimensional problems; e.g., the models in Section 7 used ds=33 Also, additional funding came from a NJIT Provost Undergraduate and 35 input uniforms for each scenario s. We should note, however, Summer Research Fellowship.

Appendix A. Proof of Theorem 2

l ffi Note that PB(SS+rLHS,ηb , ( n )≥ θ )= P ( b [ θSS+rLHS,ηb , ( n )− θ ]/ Sb ( n )≥− τb−1, γ ),so it su ces to show that ll bθ[][]SS+rLHS,ηb , ()− n θ bn θSS+rLHS,ηb , ()− n θ = ⇒ Tb−1 Snb() nSb() n (A.1) fi l〈〉r as n → ∞ for xed b ≥2, where we recall that Tb−1 is a Student-t random variable with b − 1 d.f. To accomplish this, we will start by analyzing θn()s ( ) l in (39) and build up to θnSS+rLHS,ηb , ( ) and Sb(n) through (40) and (41). l〈〉r 〈〉r fi fi The estimator θn()s ( ) in (39) of θ(s) from replication r has bounded response functions 0 (),s i de ned in (38) and (25), so the estimator satis es the LHS CLT of [49]:

⎡ ms ⎤ ⎡ 〈〉r ⎤ bm 1 ++b nθ⎢l ()− n θ⎥ = s ⎢ ∑ 055〈〉r −⇒=θN⎥ 〈〉r (0,)=(0,σσbη2 2 /) ⎣ ()s ()s ⎦ ⎣⎢ (),si ()s ⎦⎥ ()s (ss ),SS+LHS ( ),SS+LHS s ηms s i=1 ηs (A.2) 2 l〈〉r as n → ∞ because mηnbs =/→∞s , where an expression for σ(s ),SS+LHS is given in (36). The mutual independence of θn()s ( ), 1≤rb ≤ , 1≤ss ≤ 0, ensures the joint convergence ⎛ ⎞ ⎡ 〈〉r ⎤ ⎜ ⎢l ⎥ ⎟ 〈〉r nθnθrbssNrbss()s ( ) −()s : = 1, 2, …, ; = 1, 2, …, 0 ⇒( ()s : = 1, 2, …, ; = 1, 2, …, 0) ⎝ ⎣ ⎦ ⎠ (A.3) n 〈〉r 2 as → ∞ by Problem 29.2 of [43], where N()s , 1≤rb ≤ , 1≤ss ≤ 0, are independent 5(0,σbη(s ),SS+LHS / s) random variables. Thus, for each replication 〈〉r r =1,2,…,b, we have that (16) and (40) imply θnl ( ) satisfies s s ⎡ 〈〉r ⎤ 00⎡ 〈〉r ⎤ ⎢ll⎥ ⎢ ⎥ 〈〉r 〈〉r + 2 nθ⎣ ()− n θ⎦ =∑∑ λss nθ⎣ () ()− n θ()s ⎦ ⇒ λNs ()s ≡ N =5 (0, ψ) s=1 s=1 (A.4)

392 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394 as n → ∞ by (A.3) and the continuous-mapping theorem (e.g., Theorem 29.2 of [43]), where the asymptotic variance 2 s0 2 2 2 〈〉r 〈〉r ψ = ∑s=1 λσss( ),SS+LHS bη/=s bσSS+LHS,η by (31) and the independence of N()s , 1≤ss ≤ 0. Moreover, N , 1≤rb ≤ , are i.i.d. Hence, (41) and (A.4) ensure that for the numerator in the middle term of (A.1), we have

b b 1 ⎡ 〈〉r ⎤ 1 ll⎢ ⎥ 〈〉r + bn[ θSS+rLHS,ηb , ( n )− θ ]= ∑∑nθ⎣ ()− n θ⎦ ⇒ Nψ=(0,15 ) b r=1 b r=1 (A.5) n fi b 〈〉r 2 as → ∞ for xed b ≥ 2 because ∑r=1 Nbψ∼(0,5 ). Now we consider the square of the denominator of the middle term in (A.1):

2 b ⎡ ⎛ b ⎞⎤ b ⎡ ⎡ ⎤ b ⎡ ⎤⎤2 2 1 ⎢ 〈〉r 1 〈r′〉 ⎥ 1 〈〉r 1 〈r′〉 nS()= n ∑ nθ⎜ll()− n ∑ θn()⎟ = ∑ ⎢ nθ⎢l()− n θ⎥ − ∑ nθ⎢ l()− n θ⎥⎥ b ⎢ ⎥ ⎣⎢ ⎣ ⎦ ⎣ ⎦⎦⎥ b −1r=1 ⎣ ⎝ b r′=1 ⎠⎦ b −1r=1 b r′=1

b ⎡ b ⎤2 2 1 〈〉r 1 〈r′〉 + 2 χb ⇒ ∑ ⎢N − ∑ Nψ⎥ = −1 b ⎣⎢ b ⎦⎥ b −1r=1 r′=1 −1 (A.6) 2 as n → ∞ by (A.3), (A.4), and the continuous-mapping theorem, where χb−1 is a chi-squared random variable with b − 1 d.f., which is independent of b 〈r〉 ∑r=1 N in (A.5). Putting (A.5) and (A.6) into the middle term of (A.1) and using (A.3) yields l 2 2 2 + n bθ[SS+rLHS,ηb , ( nθSnψ ) − ]/b ( ) ⇒55 (0, 1)/ ψχbbb−1 /( − 1) = (0, 1)/ χb−1 /( − 1) = Tb−1 as → ∞ by the continuous-mapping theorem and the independence of the limit's numerator and denominator. Thus, (A.1) holds, so the portmanteau theorem ensures l PB(SS+rLHS,ηb , ( n )≥ θ )= P ( b [ θSS+rLHS,ηb , ( n )− θ ]/ Sb ( n )≥− τb−1, γ )→ PT ( b−1 ≥− τ b −1, γ )=γ as n → ∞ by the continuity and symmetry of the Student-t distribution, completing the proof.

Appendix B. Proof of Theorem 3

As in Assumption A3, let U′=(UU , ,…, U) be a subvector of vector U =(UU , ,…, U) of i.i.d. uniforms. It can be shown that the conditions 12 ds′ 12 ds of Theorem 5ii of [23] hold under our Assumptions A2 and A3. Thus, as established in Eq. (77) of the proof of Theorem 5ii of [23], the asymptotic 2 fi variance σ(),SCLs in (52) for scenario s satis es 2 2 σE(),SCLssss= Var[ [ϵ() (UM )| () ( U ′)]] = Var[ϵ () (U )] −E [Var[ϵ () sssss (UM )| () ( U ′)]] =σE (),SS+LHS − [Var[ϵ() (UM )| () ( U ′)]] (B.1) by a variance decomposition and (36). Hence, putting (B.1) into (53) yields

s0 2 s0 2 2 λs 2 2 λs σSCL,η = ∑∑()σE(sssη ),SS+LHS − [Var[ϵ() (UM )| () ( U ′)]] =σ SS+LHS, −E [Var[ϵ()ss (UM )| () ( U ′)]] s=1 ηs s=1 ηs by (31), establishing the equality in (55). The inequality in (55) holds as the conditional variances in (55) are nonnegative, implying that 2 2 σσSCL,η ≤ SS+LHS,η.

Appendix C. Proof of Theorem 4

We can prove Theorem 4 employing the same basic arguments applied to show Theorem 2. The only difference is that we now use the estimator

(57)ofθ(s) from each replication r instead of the SS+rLHS estimator without CMC in (39).By(47), the response function 1(s) in (57) is bounded, so the analogue of the CLT in (A.2) also holds for each r, and the rest of the proof of Theorem 2 goes through.

References S. Nuclear Regulatory Commission, Washington, DC; 1989. [9] Guba A, Makai M, Pál L. Statistical aspects of best estimate method–I. Reliab Eng Syst Saf 2003;80:217–32. [1] Nuclear Energy Agency Committee on the Safety of Nuclear Installations. Safety [10] Nutt WT, Wallis GB. Evaluation of nuclear safety from the outputs of computer margins action plan (SMAP) – final report. Tech Rep NEA/CSNI/R(2007)9. codes in the presence of uncertainties. Reliab Eng Syst Saf 2004;83:57–77. Organization for Economic Cooperation and Development; 2007. [11] Wilks SS. Determination of sample sizes for setting tolerance limits. Ann Math Stat [2] Smith C, Rabiti C, Martineau R. Risk informed safety margin characterization 1941;12:91–6. (RISMC) pathway technical program plan. Tech Rep INL/EXT-11-22977. Idaho [12] Grabaskas D, Nakayama MK, Denning R, Aldemir T. Advantages of variance National Laboratory; 2012. reduction techniques in establishing confidence intervals for quantiles. Reliab Eng [3] Dube DA, Sherry RR, Gabor JR, Hess SM. Application of risk informed safety Syst Saf 2016;149:187–203. margin characterization to extended power uprate analysis. Reliab Eng Syst Saf [13] Glasserman P. Monte Carlo methods in financial engineering. New York: Springer; 2014;129:19–28. 2004. [4] U.S. Nuclear Regulatory Commission. Acceptance criteria for emergency core [14] Asmussen S, Glynn P. Stochastic simulation: algorithms and analysis. New York: cooling systems for light-water nuclear power reactors. Title 10, Code of Federal Springer; 2007. Regulations Section 50.46 (10CFR50.46). U.S. Nuclear Regulatory Commission, [15] Chu F, Nakayama MK. Confidence intervals for quantiles when applying variance- Washington, DC; 2010. reduction techniques. ACM Trans Model Comput Simul; 36; 2012 Article 7 (25 [5] U.S. Nuclear Regulatory Commission. Applying statistics. U.S. Nuclear Regulatory pages plus 12–page online–only appendix). Commission Report NUREG-1475, Rev 1. U.S. Nuclear Regulatory Commission, [16] Nakayama MK. Asymptotically valid confidence intervals for quantiles and values- Washington, DC; 2011. at-risk when applying Latin hypercube sampling. Int J Adv Syst Meas [6] U.S. Nuclear Regulatory Commission. Safety evaluation by the Office of the Nuclear 2011;4:86–94. Reactor Regulation Topical Report WCAP-16009-P. Revision 0, realistic large break [17] Szilard RH, Youngblood R, Frepoli C, Yurko JP, Swindlehrst G, Zhang H, et al. LOCA evaluation methodology using automated statistical treatment of uncertainty Industry application emergency core cooling system cladding acceptance criteria method (ASTRUM). Tech Rep. U.S. Nuclear Regulatory Commission; 2004. problem statement. Tech Rep INL/EXT-15-35073. DOE Office of Nuclear Energy, [7] Hess S, Gaertner J, Gabor J, Shanley L, Enos-Sylla L, Prasad S, Hollingsworth S. Idaho National Laboratory; 2015. Framework for risk-informed safety margin characterization. Tech Rep 1019206. [18] Papazoglou IA. Mathematical foundations of event trees. Reliab Eng Syst Saf Electric Power Research Institute, Palo Alto, California; 2009. 1998;61:169–83. [8] U.S. Nuclear Regulatory Commission. Best-estimate calculations of emergency core [19] Sherry RR, Gabor JR, Hess SM. Pilot application of risk informed safety margin cooling performance. Nuclear Regulatory Commission Regulatory Guide 1.157. U. characterization to a total loss of feedwater event. Reliab Eng Syst Saf

393 A. Alban et al. Reliability Engineering and System Safety 165 (2017) 376–394

2013;117:65–72. Saf 2009;31:349–55. [20] Melchers RE. Structural reliability analysis and prediction, 2nd edition. Chichester, [36] Bucher C. Asymptotic sampling for high-dimensional reliability analysis. Probab UK: John Wiley & Sons; 1999. Eng Mech 2009;24:504–10. [21] Iman RL. Statistical methods for including uncertainties associated with the [37] Gaspar B, Soares CG. Hull girder reliability using a Monte Carlo based simulation geologic isolation of radioactive waste which allow for a comparison with licensing method. Probab Eng Mech 2013;31:65–75. criteria. In: Kocher DC, editor. Proceedings of the symposium on uncertainties [38] Au S-K, Beck JL. Estimation of small failure probabilities by subset simulation. associated with the regulation of the geologic disposal of high-level radioactive Probab Eng Mech 2001;16:263–77. waste. US Nuclear Regulatory Commission, Directorate of Technical Information [39] Bjerager P. Probability integration by directional simulation. J Eng Mech and Document Control, Washington, DC; 1981. p. 145–157. 1988;114(8):1285–302. [22] Owen AB, Monte Carlo theory, methods and examples, Draft; 2013 (In prepara- [40] Alban A, Darji H, Imamura A, Nakayama MK. Variance reduction for estimating a tion). failure probability with multiple criteria. In: Roeder TMK, Frazier PI, Szechtman [23] Avramidis AN, Wilson JR. Integrated variance reduction strategies for simulation. R, Zhou E, Huschka T, Chick SE, editors. Proceedings of the winter simulation Oper Res 1996;44:327–46. conference; 2016. p. 302–13. [24] Nakayama MK. Estimating a failure probability using a combination of variance- [41] Billingsley P. Convergence of probability measures, 2nd edition. New York: John reduction tecniques. In: Yilmaz L, Chan WKV, Moon I, Roeder TMK, Macal C, Wiley and Sons; 1999. Rossetti MD, editors. Proceedings of the IEEE winter simulation conference; 2015. [42] Nelsen RB. An introduction to copulas. New York: Springer; 1999. p. 621–32. [43] Billingsley P. Probability and measure, 3rd edition. New York: John Wiley and [25] Nakayama MK, Grabaskas D. Conditional Monte Carlo for efficient simulation for Sons; 1995. risk-informed safety margin characterization of nuclear power plants; 2016 (In [44] Lehmann EL. Elements of large-sample theory. New York: Springer; 1999. preparation). [45] Orechwa Y. Comments on evaluation of nuclear safety from the outputs of [26] Ayyub BM, Chia C-Y. Generalized conditional expectation for structural reliability computer codes in the presence of uncertainties by WT Nutt and GB Wallis. Reliab assessment. Struct Saf 1992;11:131–46. Eng Syst Saf; 87; 2005. p. 133–5. [27] Grossi P, Kunreuther H, editors. Catastrophe modeling: a new approach to [46] Heidtmann KD. Improved method of inclusion-exclusion applied to k-out-of-n managing risk. Springer, New York; 2005. systems. IEEE Trans Reliab 1982;R-31(1):36–40. [28] Basel Committee on Banking Supervision. Basel II: international convergence of [47] Pál L, Makai M. Remarks on the statistical aspects of the safety analysis. Tech Rep. capital measurement and capital standards: a revised framework. Tech Rep. Bank arxiv, 1336v1 [physics.data-an]; 6 Jun 2013. for International Settlements, Basel, Switzerland 〈http://www.bis.org/publ/ [48] McKay MD, Conover WJ, Beckman RJ. A comparison of three methods for bcbs107.htm〉; 2004. selecting input variables in the analysis of output from a computer code. [29] Committee on the evaluation of quantification of margins and uncertainties Technometrics 1979;21:239–45. methodology for assessing and certifying the reliability of the nuclear stockpile. [49] Owen AB. A central limit theorem for Latin hypercube sampling. J R Stat Soc B Division on Engineering and Physical Sciences. Evaluation of quantification of 1992;54:541–51. margins and uncertainties methodology for assessing and certifying the reliability [50] Loh W-L. On Latin hypercube sampling. Ann Stat 1996;24:2058–80. of the nuclear stockpile. Tech Rep. National Research Council of the National [51] Helton JC, Davis FJ. Latin hypercube sampling and the propagation of uncertainty Academies, Washington, DC; 2008. in analyses of complex systems. Reliab Eng Syst Saf 2003;81:23–69. [30] Helton JC. Quantification of margins and uncertainties: conceptual and computa- [52] Stein M. Large sample properties of simulations using Latin hypercube sampling. tional basis. Reliab Eng Syst Saf 2011;96:976–1013. Technometrics 1987;32:367, correction 32:367. [31] Helton JC, Pilch M, Sallaberry C. Probability of loss of assured safety in systems [53] Goyal A, Shahabuddin P, Heidelberger P, Nicola V, Glynn PW. A unified framework with multiple time-dependent failure modes: representations with aleatory and for simulating Markovian models of highly dependable systems. IEEE Trans epistemic uncertainty. Reliab Eng Syst Saf 2014;124:171–200. Comput 1992;C-41:36–51. [32] Agency NE, editor. Methods for safety assessment of geological disposal facilities [54] Fu MC, Hu JQ. Conditional Monte Carlo: gradient estimation and optimization for radioactive waste: outcomes of the NEA MeSA initiative. No. NEA No. 6923. applications. New York: Kluwer Academic; 1997. Organisation for Economic Co-operation and Development, Issy-les-Moulineaux, [55] McNeil AJ, Frey R, Embrechts P. Quantitative risk management: concepts, France; 2012. techniques, tools. Princeton, NJ: Princeton University Press; 2005. [33] Helton JC, Hansen CW, Sallaberry CJ. Conceptual structure and computational [56] Demarta S, McNeil AJ. The t copula and related copulas. Int Stat Rev organization of the 2008 performance assessment for the proposed high-level 2005;73(1):111–29. radioactive waste repository at Yucca Mountain, Nevada. Reliab Eng Syst Saf; 122; [57] Cario MC, Nelson BL. Modeling and generating random vectors with arbitrary 2014. p. 223–48. marginal distributions and correlation matrix. Tech Rep. Department of Industrial [34] Helton JC, Anderson DR, Basabilvazo G, Jow H-N, Marietta MG. Conceptual Engineering and Management Sciences, Northwestern University, Evanston, IL; structure of the 1996 performance assessment for the waste isolation pilot plant. 1997. Reliab Eng Syst Saf; 69; 2000. p. 151–65. [58] Iman RL, Conover WJ. A distribution-free approach to inducing rank correlation [35] Naess A, Leira BJ, Batsevych O. System reliability by enhanced Monte Carlo. Struct among input variables. Commun Stat: Simul Comput 1982;B11:311–34.

394