arXiv:1706.08591v1 [cs.DS] 26 Jun 2017 saalbeat available is http://creativecommons © iuainotu,i,wtotls fgenerality, of the loss of function without a is, such is which simulation output, objective, stochastic simulation target the a a of to settings that optimize optimization specific parameters to those Simulation input for used search the involves techniques simulations. for stochastic term great umbrella the a in be freedom there them. of optimize that to degrees natural simulations and manipulating only phenomena in is model interest It to science, simulations to systems. of communities the cheap use research enabled make of business and availability have engineering, and power modeling computational in Advances Introduction 1 eiw oeo h ies plctosta aebe ake yt by tackled been Keywords: have field. a that the applications examines in mat diverse directions field, to the the of compared thes in some as of reviews algorithms optimization each state-of-the-art simulation to for in reference algorithms difficulties competing the several emphasizes exist be sim have there cheap algorithms a imagine, or noise—various To heterogeneous expensive simulation. objectiv or stochastic decisions, homogeneous an a continuous through of evaluated or optimization be simulation—discrete the can which to of refers both (SO) Optimization Simulation Abstract ayjt Amaran Satyajith 07 hsmnsrp eso smd vial ne h CC the under available made is version manuscript This 2017. h emSmlto piiain(O san is (SO) Optimization Simulation term The 2 o tCr &,TeDwCeia opn,Fepr,T 77541 TX Freeport, Company, Chemical Dow The R&D, Core at Now 3 4 oeRD h o hmclCmay reot X751 USA 77541, TX Freeport, Company, Chemical Dow The R&D, Core 1 oeRD h o hmclCmay iln,M 86,USA 48667, MI Midland, Company, Chemical Dow The R&D, Core eateto hmclEgneig angeMlo Unive Mellon Carnegie Engineering, Chemical of Department iuainotmzto:Arve of review A optimization: Simulation iuainotmzto;Otmzto i iuain eiaiefe op Derivative-free simulation; via Optimization optimization; Simulation https://link loihsadapplications and algorithms 1,2 . org/licenses/by-nc-nd/4 ioasV Sahinidis V. Nikolaos , . springer itbrh A123 USA 15213, PA Pittsburgh, . com/article/10 Bury USA tcatcsmltosaentdtriitc and deterministic, this there in not these result, simulations are from few a Outputs simulations perform As parameters. stochastic to optimal of need for terms search a resources. in also or run, is to money, expensive time, be may simulations input optimal algorithmic for search SO their settings. in many data a simulation input-output the the fact, such for from on allows In constraints depend only solely and approaches that input. objective box may particular the black simulation of of a evaluation description available—the as available algebraic is be an simulation that assume the not does minimized. . 0/ 4 nadto,mn ag-cl n/rdetailed and/or large-scale many addition, In SO programming, mathematical to opposed As h omlpbiaino hsarticle this of publication formal The . . 1007/s10479-015-2019-x 1 irmSharda Bikram , dcnrsstedffrn prahsused, approaches different the contrasts nd npooe nteltrtr.A n can one As literature. the in proposed en eemtos n pcltso future on speculates and methods, hese lse fpolm.Ti document This problems. of classes e desseicfaue faparticular a of features specific ddress ltos igeo utpeoutputs, multiple or single ulations, ucinsbett constraints, to subject function e eaia rgamn,makes programming, hematical B-CN . license 4.0 -BY-NC-ND timization 3 . n ct J. Scott and , rsity, , Simulation optimization: A review of algorithms and applications 2

usually follow some output distribution, which may knowledge. Relatively few existing algorithms or may not vary across the parametric space. This attempt to address both discrete and continuous uncertainty or variability in output also adds to choices simultaneously, although some broad classes the challenge of optimization, as it becomes harder of approaches naturally lend themselves to be to discern the quality of the parametric input in applicable in either, and therefore both, settings. the presence of this output noise. In addition, Further, the discrete variables may either be binary, when an algebraic description of the simulation integer-ordered, or categorical and lie in some discrete is not accessible, derivative information is usually space D. unavailable, and the estimation of derivatives from As can be seen, the formulation P1 is extremely the use of finite differences may not be suitable general, and therefore a wide variety of applications due to noisy outputs and the expensive nature of fall under the scope of simulation optimization. simulations. Various applications of simulation optimization in The nature of the stochastic simulations under diverse research fields are tabulated in Section 2. study will determine the specific technique chosen Another common assumption is that f is a to optimize them. The simulations, which are real-valued function and g is a real vector-valued often discrete-event simulations, may be partially function, both of whose expected values may or accessible to us in algebraic form, or may be purely may not be smooth or continuous functions. The available as an input-output model (as a black most common objective in SO is to optimize the box); they may have single or multiple outputs; expected value of some performance metric, but other they may have deterministic or stochastic output(s); objective functions may be appropriate depending they may involve discrete or continuous parameters; on the application. For instance, an objective and they may or may not involve explicit, or even that minimizes risk could be a possible alternative, implicit/hidden constraints. in which case one would incorporate some sort of A very general simulation optimization problem variance measure as well into the objective. can be represented by (P1). This paper is an updated version of Amaran et al. [4] and is meant to be a survey of available techniques E min ω[f(x,y,ω)] as well as recent advances in simulation optimization. s.t. Eω[g(x,y,ω)] 0 The remainder of the introduction section provides a ≤ literature survey of prior reviews, and elaborates on h(x, y) 0 (P1) ≤ the relationship of simulation optimization to math- xl x xu ematical programming, derivative-free optimization, x ≤Rn≤,y Dm. ∈ ∈ and machine learning. Section 2 provides a glimpse The function f can be evaluated through simulation into the wide variety of applications of simulation for a particular instance of the continuous inputs x, optimization that have appeared in the literature. discrete inputs y, and a realization of the random Section 3 focuses on various algorithms for discrete variables in the simulation, the vector ω (which may and continuous simulation optimization, provides or may not be a function of the inputs, x and y). basic pseudocode for major categories of algorithms, Similarly, the constraints defined by the vector-valued and provides comprehensive references for each type function g are also evaluated with each simulation of algorithm. Section 4 provides a listing of available run. In this formulation, expected values for these software for simulation optimization and Section stochastic functions are used. There may be other 5 discusses means to compare their performance. constraints (represented by h) that do not involve Section 6 summarizes the progress of the field, and random variables, as well as bound constraints on outlines some current and future topics for research. the decision variables. The relaxation of any of these conditions would 1.1 Prior reviews of simulation optimization constitute a problem that would fall under the purview of SO. Most algorithms focus on problems Several review papers (cf. [135, 99, 174, 15, 59, 29, that either have solely discrete choices, or solely 7, 16, 191, 61, 60, 194, 62, 9, 88, 5, 152]), books and continuous decisions to make. Each constraint may research monographs (cf. [186, 171, 115, 35]), and be thought of as representing additional outputs theses (cf. [12, 53, 45, 30, 58, 104]) have traced the from the simulation that need to be taken into development of simulation optimization. consideration. In addition, there may be bound Meketon [135] provides a classification of algo- constraints imposed on decision variables, that may rithmic approaches for optimization over simulations either be available or obtained from domain-specific based on how much information or structure about Amaran, Sahinidis, Sharda and Bury 3

the underlying model is known. The paper surveys while expounding on more recent methods in the progress of the field between 1975 and 1987, a concise manner. Though several reviews and focuses on continuous simulation optimization. exist, we catalog the most recent developments— Andrad´ottir [7] provides a tutorial on gradient-based the emergence of derivative-free optimization and procedures for continuous problems. Carson and its relationship with simulation optimization, the Maria [29] and Azadivar [16] also give brief outlines appearance of simulation test-beds for comparing of and pointers to prevailing simulation optimization algorithms, the recent application of simulation algorithms. optimization in diverse fields, the development of Fu et al. [61] contains several position and interest in related techniques and theory by the statements of eminent researchers and practitioners machine learning community and the optimization in the field of simulation, where the integration community, as well as the sheer unprecedented nature of simulation with optimization is discussed. The of recent interest in optimizing over simulations. A issues addressed include generality vs. specificity reflection of a surge in recent interest is evidenced of an algorithm, the wider scope of problems by the fact that more than half of the works we that simulation optimization methodologies have the reference were published in the last decade. The potential to address, and the need for integrating intent is to not only trace the progress of the field, provably convergent algorithms proposed by the but to provide an update on state-of-the-art methods research community with metaheuristics often used and implementations, point the familiar as well as by commercial simulation software packages. the uninitiated reader to relevant sources in the Of the more recent surveys, Fu [59] provides an literature, and to speculate on future directions in excellent tutorial on simulation optimization, and the field. focuses on continuous optimization problems more than discrete optimization problems. The paper 1.2 A note on terminology and scope focuses specifically on discrete-event simulations. Fu [60] provides a comprehensive survey of the As simulation optimization involves the use of field and its scope—the paper outlines the different algorithms that arose from widely differing fields ways in which optimization and simulation interact, (Section 3), has relationships to many diverse gives examples of real-world applications, introduces disciplines (Section 1.3), and has been applied to simulation software and the optimization routines many different practical applications from biology that each of them use, provides a very basic tutorial to engineering to logistics (Section 2), it is not on simulation output analysis and convergence theory surprising that it is known by various names for simulation optimization, elaborates on algorithms in different fields. It has also been referred for both continuous and discrete problems, and to as simulation-based optimization, stochastic provides pointers to many useful sources. Fu et optimization, parametric optimization, black-box al. [62] provide a concise, updated version of all of optimization, and Optimization via Simulation this, and also talk about estimation of distribution (OvS), where the continuous and discrete versions algorithms. are accordingly known as Continuous Optimization Tekin and Sabuncuoglu [194] provide a table that via Simulation (COvS) and Discrete Optimization analyzes past review papers and the techniques they via Simulation (DOvS). Each algorithmic technique focus on. Apart from providing detailed updates on may also go by different names, and we attempt to advances in approaches and algorithms, the paper reconcile these in Section 3. also lists references that attempt to compare different Inputs to the simulation may be variously SO techniques. Hong and Nelson [88] classify referred to as parameter settings, input simulation optimization problems into those with (1) settings, variables, controls, solutions, designs, a finite number of solutions; (2) continuous decision experiments (or experimental designs), factors, variables; and (3) discrete variables that are integer- or configurations. Outputs from the simulation ordered. The paper describes procedures for each are called measurements, responses, performance of these classes. Perhaps the most recent survey, metrics, objective values, simulation replications, [5], classifies simulation optimization algorithms and realizations, or results. The performance of a provides a survey of methods as well as applications simulation may also be referred to as an experiment, appearing in the literature between 1995 and 2010. an objective function evaluation, or simply a The present paper provides an overview of function evaluation. We will use the term ‘iteration’ techniques, and briefly outlines well-established to refer to a fixed number of function evaluations methods with pointers to more detailed surveys, (usually one) performed by a simulation optimization Simulation optimization: A review of algorithms and applications 4

algorithm. or due to the large expense associated with obtaining function evaluations, or both. The inherent A note of caution while using SO methods is to stochasticity in output also renders automatic incorporate as much domain specific knowledge as differentiation (AD) [161, 76] tools not directly possible in the use of an SO algorithm. This may applicable. Moreover, automatic differentiation be in terms of (1) screening relevant input variables, may not be used when one has no access to (2) scaling and range reduction of decision variables, source code, does not possess an AD interface to (3) providing good initial guesses for the algorithm; proprietary simulation software, and, of course, when and (4) gleaning information from known problem one is dealing with a physical experiment. The structure, such as derivative estimates. lack of availability of derivative information has Table 1 classifies the techniques that are usually further implications—it complicates the search for most suitable in practice for different scenarios in descent directions, proofs of convergence, and the the universe of optimization problems. Certain characterization of optimal points. broad classes of algorithms, such as random search Simulation optimization, like stochastic program- methods, may be applicable to all of these types ming, also attempts to optimize under uncertainty. of problems, but they are often most suitable when However, stochastic programming differs in that dealing with pathological problems (e.g., problems it makes heavy use of the model structure itself with discontinuities, nonsmoothness) and are often [27]. Optimization under uncertainty techniques that used because they are relatively easy to implement. make heavy use of mathematical programming are The possibilities of combining simulation and reviewed in [175]. optimization procedures are vast: simulation with optimization-based iterations; optimization with Derivative-Free Optimization. Both simulation-based iterations; sequential simulation Simulation Optimization and Derivative-Free and optimization; and alternate simulation and Optimization (DFO) are referred to in the literature optimization are four such paradigms. A recent as black-box optimization methods. Output paper by Figueira and Almada-Lobo [56] delves variability is the key factor that distinguishes SO into the taxonomy of such problems, and provides from DFO, where the output from the simulation is a guide to choosing an appropriate approach for deterministic. However, there are many approaches a given problem. As detailed by Meketon [135], to DFO that have analogs in SO as well (e.g., response different techniques may be applicable or more surfaces, direct search methods, metaheuristics), cf. suitable depending on how much is known about Section 3. the underlying simulation, such as its structure or Another distinction is that most algorithms in associated probability distributions. We focus on DFO are specifically designed keeping in mind that approaches that are applicable in situations where function evaluations or simulations are expensive. all the optimization scheme has to work with are This is not necessarily the case with SO algorithms. evaluations of f(x,y,ω) and g(x,y,ω), or simply, With regard to rates of convergence, SO observations with noise. algorithms are generally inefficient and convergence rates are typically very slow. In general, one would 1.3 Relationship to other fields expect SO to have a slower convergence rate than DFO algorithms simply because of the additional Mathematical Programming. As men- complication of uncertainty in function evaluations. tioned earlier, most mathematical programming As explained in [43], some DFO algorithms, under methods rely on the presence of an algebraic model. certain assumptions, expect rates that are closer The availability of an algebraic model has many to linear than quadratic, and therefore early obvious implications to a mathematical programming termination may be suitable. As described in some expert, including the ability to evaluate a function detail by [59], the best possible convergence rates quickly, the availability of derivative information, and for SO algorithms are generally (1/√k), where the possibility of formulating a dual problem. None of k is the number of samples. ThisO is true from the these may be possible to do/obtain in an SO setting. central limit theorem that tells us the rate at which In the case with continuous decisions, derivative the best possible estimator converges to the true information is often hard to estimate accurately expected function value at a point. This implies through finite differences, either due to the stochastic that though one would ideally incorporate rigorous noise associated with objective function evaluations, termination criteria in algorithm implementations, Amaran, Sahinidis, Sharda and Bury 5

Table 1: Terminology of optimization problems

Algebraic model available Unknown/complex problem structure Deterministic Traditional math program- Derivative- ming (linear, integer, and free nonlinear programming) optimization Uncertainty present Stochastic programming, ro- Simulation bust optimization optimization most practical applications have a fixed simulation certain number of slot machines, and a certain total or function evaluation budget that is reached first. budget to play them. Here, each choice of sample corresponds to which slot machine to play. Each play on a slot machine results in random winnings. Machine Learning. Several subcommunities This setting is analogous to discrete simulation in the machine learning community address problems optimization (DOvS) over finite sets, although with closely related to simulation optimization. Tradi- a different objective [158]. Again, in DOvS over tional machine learning settings assume the avail- finite sets, we are only concerned with finding the ability of a fixed dataset. Active learning methods best alternative eventually, whereas the cumulative [40, 178] extend machine learning algorithms to the winnings is the concern in the multi-armed bandit case where the algorithms are allowed to query an problem. oracle for additional data to infer better statistical models. Active learning is closely related in that Relationship to other fields. Most, if this choice of sampling occurs at every iteration in not all, simulation optimization procedures have a simulation optimization setting as well. The focus elements that are derived from or highly related to of active learning is usually to learn better predictive several other fields. Direct search procedures and models rather than to perform optimization. response surface methodologies (RSM) have strong Reinforcement learning [190] is broadly concerned relationships with the field of experimental design. with what set of actions to take in an environment to RSM, sample path optimization procedures, and maximize some notion of cumulative reward. Rein- gradient-based methods heavily incorporate ideas forcement learning methods have strong connections from mathematical programming. RSM also involves to information theory, optimal control, and statistics. the use of nonparametric and Bayesian regression The similarity with simulation optimization is that techniques, whereas estimation of distribution the common problem of exploration of the search algorithms involves probabilistic inference, and space vs. exploitation of known structure of the therefore these techniques are related to statistics cost function arises. However, in the reinforcement and machine learning. Simulation optimization learning setting, each action usually also incurs a has been described as being part of a larger field cost, and the task is to maximize the accumulated called computational stochastic optimization. More rewards from all actions—as opposed to finding a information is available at [157]. good point in the parameter space eventually. Policy gradient methods [155] are a subfield of reinforcement learning, where the set of all possible 2 Applications sequences of actions form the policy space, and a gradient in this policy space is estimated and a SO techniques are most commonly applied to either gradient ascent-type method is then used to move (1) discrete-event simulations, or (2) systems of to a local optimum. Bandit optimization [68] stochastic nonlinear and/or differential equations. is another subfield of reinforcement learning that As mentioned in [59], discrete event simulations involves methods for the solution to the multi-armed can be used to model many real-world systems such bandit problem. The canonical example involves a as queues, operations, and networks. Here, the Simulation optimization: A review of algorithms and applications 6

simulation of a system usually involves switching or approach taken by each of these algorithms. Many jumping from one state to another at discrete points of the sections include pointers to convergence in time as events occur. The occurrence of events is proofs for individual algorithms. Optimality modeled using probability distributions to model the in simulation optimization is harder to establish randomness involved. than in mathematical programming or derivative- Stochastic differential equations may be used to free optimization due to the presence of output model phenomena ranging from financial risk [137] variability. Notions of optimality for simulation to the control of nonlinear systems [183] to the optimization are explored in [59]; for the discrete electrophoretic separation of DNA molecules [39]. case, [206], for instance, establishes conditions for With both discrete-event simulations and stochas- local convergence, where a point being ‘better’ tic differential equation systems, there may be than its 2m + 1 neighboring solutions is said to several parameters that one controls that affect be locally optimal. There has also been some some performance measure of the system under work in establishing Karush-Kuhn-Tucker (KKT) consideration, which are essentially degrees of optimality conditions for multiresponse simulation freedom that may be optimized through SO optimization [24]. Globally convergent algorithms techniques. Several applications of SO from diverse will locate the global optimal solution eventually, but areas have been addressed in the literature and we assuring this would require all feasible solutions to be list some of them in Table 2. evaluated through infinite observations; in practice, a convergence property that translates to a practical stopping criterion may make more sense [88]. 3 Algorithms Based on their scope, the broad classes of algorithms are classified in Table 3. Algorithms Algorithms for SO are diverse, and their applicability are classified based on whether they are applicable may be highly dependent on the particular applica- to problems with discrete/continuous variables, and tion. For instance, algorithms may (1) attempt to whether they focus on global or local optimization. find local or global solutions; (2) address discrete However, there may be specific algorithms that have or continuous variables; (3) incorporate random been tweaked to make them applicable to a different elements or not; (4) be tailored for cases where class as well, which may not be captured by this table. function evaluations are expensive; (5) emphasize exploration or exploitation to different extents; (6) assume that the uncertainty in simulation output 3.1 Discrete optimization via simulation is homoscedastic or that it comes from a certain Discrete optimization via simulation is concerned probability distribution; or (7) rely on underlying with finding optimal settings for variables that can continuity or differentiability of the expectation (or only take discrete values. This may be in the form some function of a chosen moment) of the simulation of integer-ordered variables or categorical variables output. The sheer diversity of these algorithms [153]. Integer-ordered variables are allowed to take also makes it somewhat difficult to assert which one on integer or discrete values within a finite interval, is better than another in general, and also makes where the order of these values translates to some it hard to compare between algorithms or their physical interpretation. For example, this could be implementations. the number of trucks available for vehicle routing, As mentioned in Section 1.3, many algorithms that or the set of standard pipe diameters that are are available for continuous simulation optimization available for the construction of a manufacturing have analogs in derivative-based optimization and plant. Categorical variables refer to more general in derivative-free optimization, where function kinds of discrete decisions, ranging from conventional evaluations are deterministic. In any case, the key on-off (0-1 or binary) variables to more abstract lies in the statistics of how noise is handled, and decisions such as the sequence of actions to take how it is integrated into the optimization scheme. given a finite set of actions. It should be noted We will provide pointers to references that are that though integer-ordered variables, for instance, applicable to simulation optimization in particular. may be logically represented using binary variables, A comprehensive review of methods for derivative- it may be beneficial to retain them as integer-ordered free optimization is available in [166]. to exploit correlations in objective function values Each major subsection below is accompanied by between adjacent integer values. pseudocode to give researchers and practitioners A rich literature in DOvS has developed over unfamiliar with the field an idea of the general the last 50 years, and the specific methods Amaran, Sahinidis, Sharda and Bury 7

Table 2: Partial list of published works that apply simulation optimization

Domain of application Application and citations Operations Buffer location [132], nurse scheduling [193], inventory management [120, 176], health care [11], queuing networks [63, 25, 138] Manufacturing PCB production [48], engine manufacturing [192], production planning [107, 114], manufacturing-cell design [98], kanban sizing [79] Medicine and biology Protein engineering [168], cardiovascular surgery [202], breast cancer epidemiology [55], bioprocess control [197, 165], ECG analysis [67], medical image analysis [136] Engineering Welded beam design [209], solid waste management [210], pollution source identification [14], chemical supply chains [103], antenna design [159], aerodynamic design [203, 205, 204, 122], distillation column optimization [162], well placement [18], servo system control [160], power systems [54], radar analysis [108] Computer science, networks, Server assignment [125], wireless sensor networks [49], electronics circuit design [130], network reliability [123] Transportation and logistics Traffic control and simulation [211, 17, 151], metro/transit travel times [83, 148], air traffic control [119, 96]

Table 3: Classification of simulation optimization algorithms

Algorithm class Discrete Continuous Local Global Ranking and selection × × Metaheuristics Response surface methodology × × × Gradient-based methods × × × × × Direct search Model-based methods × × × Lipschitzian optimization × × × × × × Simulation optimization: A review of algorithms and applications 8

developed are tailored to the specific problem setting. An alternative formulation of the ranking and Broadly, methods are tailored for finite or for very selection of the problem would be to try to do the large/potentially infinite parameter spaces. best within a specified computational budget, called the optimal computing budget allocation formulation [33]. Chen et al. [36] present more recent work, while 3.1.1 Finite parameter spaces the stochastically constrained case is considered in In the finite case, where the number of alternatives is [129]. small and fixed, the primary goal is to decide how to Recent work [95] in the area of DOvS over finite allocate the simulation runs among the alternatives. sets provides a quick overview of the field of ranking In this setting, there is no emphasis on ‘search’, and selection, and considers general probability dis- as the candidate solution pool is small and known; tributions and the presence of stochastic constraints each iteration is used to infer the best, in some simultaneously. statistical sense, simulation run(s) to be performed A basic ranking and selection procedure [112] is subsequently. outlined in Algorithm 1, where it is assumed that The optimization that is desired may differ independent data comes from normal distributions depending on the situation, and could involve: with unknown, different variances. 1. The selection of the best candidate solution from a finite set of alternatives; Algorithm 1 Basic ranking and selection procedure for SO 2. The comparison of simulation performance measures of each alternative to a known Require: Confidence level 1 α, indifference standard or control; or zone parameter δ − 3. The pairwise comparison between all solution 1: Take n0 samples from each of the 1,...,K candidates. potential designs

Item (1) is referred to as the ranking and selection 2: Compute sample means, t¯k,n0 and sample problem. Items (2) and (3) are addressed under variances, Sk, for each of the designs literature on multiple comparison procedures, with 3: Determine how many new samples, Nk := the former referred to as multiple comparisons with 2 2 ψ Sk n , 2 a control. max n 0 l δ mo, to take from each system, where the Rinott constant ψ is Ranking and Selection. In traditional obtained from [21] ranking and selection, the task is to minimize the 4: Select the system with the best new number of simulation replications while ensuring a sample mean, t¯ + . certain probability of correct selection of alternatives. k,Nk n0 Most procedures try to guarantee that the design ultimately selected is better than all competing alternatives by δ with a probability at least 1 Multiple comparison procedures. Here, α. δ is called the indifference zone, and is the− a number of simulation replications are performed on value deemed to be sufficient to distinguish between all the potential designs, and conclusions are made by expected performance among solution candidates. constructing confidence intervals on the performance Conventional procedures make use of the Bon- metric. The main ideas and techniques for multiple ferroni inequality which relates probabilities of the comparisons in the context of pairwise comparisons, occurrence of multiple events with probabilities or against a known standard are presented in [86], of each event. Other approaches involve the [59], and [90]. Recent work in multiple comparisons incorporation of covariance induced by, for example, with a control include [110] and [146], which provide the use of common random numbers to expedite the fully sequential and two-stage frequentist procedures algorithmic performance over the more conservative respectively; and [201], which addresses the problem Bonferroni approach. Kim and Nelson [111, 112] using a Bayesian approach. and Chick [38] provide a detailed review and provide algorithms and procedures for this setting. Comprehensive treatment of ranking and selection Extensions of fully sequential ranking and selection and multiple comparison procedures may be found procedures to the constrained case have been in Goldsman and Nelson [74] and Bechhofer et al. explored as well, e.g., [10]. Amaran, Sahinidis, Sharda and Bury 9

[21]. A detailed survey that traces the development rather than pre-determined; a most promising ‘index’ of techniques in simulation optimization over finite is defined that classifies each candidate solution sets is available in [194]. based on a nearest neighbor metric. More recently, the Adaptive Hyberbox Algorithm [207] claims to have superior performance on high-dimensional 3.1.2 Large/Infinite parameter spaces problems (problems with more than ten or fifteen To address DOvS problems with a large number of variables); and the R-SPLINE algorithm [198], potential alternatives, algorithms that have a search which alternates between a continuous search on component are required. Many of the algorithms a continuous piecewise-linear interpolation and a that are applicable to the continuous optimization discrete neighborhood search, compares favorably as via simulation case are, with suitable modifications, well. applicable to the case with large/infinite parameter A review of random search methods is presented in spaces. These include (1) ordinal optimization [8, 149]. Recent progress, outlines of basic algorithms, (2) random search methods and (3) direct search and pointers to specific references for some of these methods. methods are presented in [26], [88], and [145]. Ordinal optimization methods [84] are suitable Direct search methods such as pattern search and when the number of alternatives is too large to Nelder-Mead simplex methods are elaborated on in find the globally optimal design in the discrete-event Section 3.5. simulation context. Instead, the task is to find a satisfactory solution with some guarantees on quality 3.2 Response surface methodology (called alignment probability) [127]. Here, the focus is on sampling a chosen subset of the solutions and Response surface methodology (RSM) is typically evaluating them to determine the best among them. useful in the context of continuous optimization The key lies in choosing this subset such that it problems and focuses on learning input-output rela- contains a subset of satisfactory solutions. The tionships to approximate the underlying simulation quality or satisfaction level of this selected subset can by a surface (also known as a metamodel or surrogate be quantified [34]. A comparison of subset selection model) for which we define a functional form. This rules is presented in [100] and the multi-objective case functional form can then be made use of by leveraging is treated in [195]. powerful derivative-based optimization techniques. Random search methods include techniques such The literature in RSM is vast and equivalent as simulated annealing (e.g., [3]), genetic algorithms, approaches have variously been referred to as multi- stochastic ruler methods (e.g., [208]), stochastic disciplinary design optimization, metamodel-based comparison (e.g., [75]), nested partitions (e.g., optimization, and sequential parameter optimization. [182]), ant colony optimization (e.g., [52, 51]), RSM was originally developed in the context of and tabu search (e.g., [71]). Some of these— experimental design for physical processes [28], but simulated annealing, genetic algorithms, and tabu has since been applied to computer experiments. search—are described in Section 3.6). Ant Metamodel-based optimization is a currently popular colony optimization is described under model-based technique for addressing simulation optimization methods (cf. Section 3.7.2). Proofs of global problems [20, 115]. convergence, i.e., convergence to a global solution, Different response surface algorithms differ in the or local convergence are available for most of these choice between regression and interpolation; the algorithms [88] (note that these definitions differ from nature of the functional form used for approximation mathematical programming where global convergence (polynomials, splines, Kriging, radial basis functions, properties ensure convergence to a local optimum neural networks); the choice of how many and where regardless of the starting point). new samples must be taken; and how they update the Nested partition methods [181] attempt to response surface. adaptively sample from the feasible region. The RSM approaches can either (1) build surrogate feasible region is then partitioned, and sampling models that are effective in local regions, and is concentrated in regions adjudged to be the sequentially use these models to guide the search, or; most promising by the algorithm from a pre- (2) build surrogate models for the entire parameter determined collection of nested sets. Hong and space from space-filling designs, and then use them Nelson propose the COMPASS algorithm [87] which to choose samples in areas of interest, i.e., where uses a unique neighborhood structure, defined as the likelihood of finding better solutions is good the most promising region that is fully adaptive Simulation optimization: A review of algorithms and applications 10

Algorithm 2 Basic RSM procedure simulation metamodeling is explored in [22, 117, Require: Initial region of approximation , 116]. Other criteria that have been used to X choose samples are most probable improvement [139], choice of regression surface r knowledge gradient for continuous parameters [177], 1: while under simulation budget and not and maximum information gain [189]. converged do 2: Perform a design of experiments in Trust region methods. Trust region meth- relevant region, using k data points ods [42] can be used to implement sequential RSM. Trust regions provide a means of controlling the 3: ti simulate(xi), i = 1,...,k ← { } region of approximation, providing update criteria Evaluate noisy function f(xi,ω) { ∗ } 2 for surrogate models, and are useful in analyzing 4: λ arg minλ (ti r(xi,λ)) ← P − convergence properties. Once a metamodel or Fit regression surface r through points response surface, g, is built around a trust region { using squared loss function center xi, trust region algorithms involve the solution ∗ ∗} of the trust-region subproblem (mins g(xi + s): s 5: x arg minx r(x, λ ): x ∈ ← { ∈ X} (xi, ∆)), where is a ball defined by the center- Optimize surface B B { } radius pair (xi, ∆). There are well-defined criteria to 6: Update set of available data points update the trust region center and radius [42] that and region of approximation will define the subsequent region of approximation. 7: end while The use of trust regions in simulation optimization is relatively recent, and has been investigated to some extent [46, 32]. Trust-region algorithms have according to a specified metric. A generic framework been used, for example, to optimize simulations of for RSM is presented in Algorithm 2. urban traffic networks [151].

Classical sequential RSM. Originally, RSM consisted of a Phase I, where first-order 3.3 Gradient-based methods models were built using samples from a design of experiments. A steepest descent rule was used to Stochastic approximation methods or gradient-based move in a certain direction, and this would continue approaches are those that attempt to descend iteratively until the estimated gradient would be using estimated gradient information. Stochastic close to zero. Then, a Phase II procedure that approximation techniques are one of the oldest built a more detailed quadratic model would be methods for simulation optimization. Robbins and used for verifying the optimality of the experimental Monro [167] and Kiefer and Wolfowitz [109] were design. A thorough introduction to response surface the first to develop stochastic approximation schemes methodology is available in [142]. Recent work in in the early 1950s. These procedures initially were the field includes automating RSM [143, 147] and meant to be used under very restrictive conditions, the capability to handle stochastic constraints [13]. but much progress has been made since then. These methods can be thought of being analogous to steepest descent methods in derivative-based Bayesian global optimization. These optimization. One may obtain direct gradients or methods seek to build a global response surface, may estimate gradients using some finite difference commonly using techniques such as Kriging/Gaussian scheme. Direct gradients may be calculated by process regression [173, 163]. Subsequent samples a number of methods: (1) Perturbation Analysis chosen based on some sort of improvement metric (specifically, Infinitesimal Perturbation Analysis) may balance exploitation and exploration. The (PA or IPA), (2) Likelihood Ratio/Score Function seminal paper by Jones et al. [102] which introduced (LR/SF), and (3) Frequency Domain Analysis the EGO algorithm for simulations with deterministic (FDA). Detailed books on these methods are output, uses Kriging to interpolate between function available in the literature [85, 69, 172, 156, 65] and values, and chooses future samples based on an more high-level descriptions are available in papers expected improvement metric [140]. Examples of [194, 60]. Most of these direct methods, however, are analogs to this for simulation optimization are either applicable to specific kinds of problems, need provided in [93, 118]. The use of Kriging for some information about underlying distributions, or Amaran, Sahinidis, Sharda and Bury 11

are difficult to apply. Fu [60] outlines which methods of results in stochastic approximation is presented in are applicable in which situations, and Tekin and [154]. Sabuncuoglu [194] discuss a number of applications that have used these methods. 3.4 Sample path optimization Stochastic approximation schemes attempt to estimate a gradient by means of finite differences. Sample path optimization involves working with an Typically, a forward difference estimate would estimate of the underlying unknown function, as involve sampling at least n + 1 distinct points, opposed to the function itself. The estimate is but superior performance has been observed by usually a consistent estimator such as the sample simultaneous perturbation estimates that require mean of independent function evaluations at a point, samples at just two points [185], a method or replications. For instance, one may work with 1 n referred to as Simultaneous Perturbation Stochastic Fn = n i=1 f(x,y,ωi), instead of the underlying Approximation (SPSA). The advantage gained in function PE[f(x,y,ω)] itself. It should be noted that SPSA is that the samples required are now the functional form of Fn is still unknown, it is just independent of the problem size, and, interestingly, that Fn can be observed or evaluated at a point in the this has been shown to have the same asymptotic search space visited by an algorithm iteration. The convergence rate as the naive method that requires alternative name of sample average approximation n + 1 points [184]. A typical gradient-based scheme reflects this use of an estimator. is outlined in Algorithm 3. As the algorithm now has to work with an esti- mator, a deterministic realization of the underlying Algorithm 3 Basic gradient-based proce- stochastic function, sophisticated techniques from dure traditional mathematical programming can now be leveraged. Sample path methods can be viewed as the Require: Specify initial point, x0. Define use of deterministic optimization techniques within initial parameters such as step size (α), a well-defined stochastic setting. Yet another name distances between points for performing for them is stochastic counterpart. Some of the finite difference, etc. first papers using sample path methods are [82] and 1: i 0 [179]. Several papers [172, 37, 77, 180, 46] discuss ← convergence results and algorithms in this context. 2: while under simulation budget and not converged do ji 3.5 Direct search methods 3: Perform required simulations, t i ← simulate(xi), with ji replications to Direct search can be defined as the sequential estimate gradient, J, using either IPA, examination of trial solutions generated by a certain strategy [89]. As opposed to stochastic LR/SF, FDA or finiteb differences approximation, direct search methods rely on direct 4: xi+1 xi αJ comparison of function values without attempting ← − 5: i i +1 b to approximate derivatives. Direct search methods ← 6: end while typically rely on some sort of ranking of quality of points, rather than on function values. Most direct search algorithms developed for Recent extensions of the SPSA method include simulation optimization are extensions of ideas for introducing a global search component to the derivative-free optimization. A comprehensive review algorithm by injecting Monte Carlo noise during of classical and modern methods is provided in the update step [134], and using it to solve [121]. A formal theory of direct search methods for combined discrete/continuous optimization problems stochastic optimization is developed in [196]. Direct [199]. Recent work also addresses improving Jacobian search methods can be tailored for both discrete as well as Hessian estimates in the context of the and continuous optimization settings. Pattern search SPSA algorithm [187]. Much of the progress in and Nelder-Mead simplex procedures are the most stochastic approximation has been cataloged in the popular direct search methods. There is some proceedings of the Winter Simulation Conference classical as well as relatively recent work done on over the years (http://informs-sim.org/). A investigating both pattern search methods [196, 6, recent review of stochastic approximation methods is 131] and Nelder-Mead simplex algorithms [144, 19, available in [188], and an excellent tutorial and review Simulation optimization: A review of algorithms and applications 12

94, 31] and their convergence in the context of The cross over and mutation functions ensure that simulation optimization. a diversity of solutions is maintained. Genetic These methods remain attractive as they are algorithms are popular as they are easy to implement relatively easy to describe and implement, and are and are used in several commercial simulation not affected if a gradient does not exist everywhere, optimization software packages (Table 4). The as they do not rely on gradient information. Since GECCO (Genetic and Evolutionary Computation conventional procedures can be affected by noise, Conference) catalogs progress in genetic algorithms effective sampling schemes to control the noise are and implementations. required. A basic Nelder-Mead procedure is outlined in Algorithm 4. 3.6.2 Simulated annealing Algorithm 4 Basic Nelder-Mead simplex Simulated Annealing uses a probabilistic method procedure for SO that is derived from the annealing process in which the material is slowly cooled so that, while its Require: A set of n 1 points in the − structure freezes, it reaches a minimum energy state parameter space to form the initial [113, 23]. Starting with a current point i in a ′ simplex state j, a neighborhood point i of the point i is generated. The algorithm moves from point i to i′ 1: while under simulation budget and not using a probabilistic criteria that is dependent on converged do the ‘temperature’ in state j. This temperature is 2: Generate a new candidate solution, analogous to that in physical annealing, and serves ′ xi, through simplex centroid reflections, here as a control parameter. If the solution at i contractions or other means is better than the existing solution, then this new j point is accepted. If the new solution is worse than 3: t i simulate(x ), i = i n + i ← i { − existing solution, then the probability of accepting 1,...,i , ji = 1,...,Ni Evaluate the point is defined as exp( (f(i′) f(i))/T (j)), } { } { − − noisy function f(x, ω) Ni times, where Ni where f(.) is the value of objective function at a is determined by some sampling scheme given point, and T (j) is temperature at the state j P t i } j. After a certain number of neighborhood points 4: Calculate ji i , or some similar Ni are evaluated, the temperature is decreased and new metric to determine which point (i.e., state is j + 1 is created. Due to the exponential with the highest metric value) should be form, the probability of acceptance of a neighborhood eliminated point is higher at high temperature, and is lower as temperature is reduced. In this way, the algorithm 5: end while searches for a large number of neighborhood points in the beginning, but a lower number of points as temperature is reduced. Implementation of simulated annealing procedures 3.6 Random search methods require choosing parameters such as the initial and 3.6.1 Genetic algorithms final temperatures, the rate of cooling, and number of function evaluations at each temperature. A variety Genetic algorithms use concepts of mutation and of cooling ‘schedules’ have been suggested in [41] selection [164, 200]. In general, a genetic algorithm and [78]. Though simulated annealing was originally works by creating a population of strings and each meant for optimizing deterministic functions, the of these strings are called chromosomes. Each of framework has been extended to the case of stochastic these chromosome strings is basically a vector of point simulations [2]. The ease of implementing a simulated in the search space. New chromosomes are created annealing procedure is high and it remains a popular by using selection, mutation and crossover functions. technique used by several commercial simulation The selection process is guided by evaluating the optimization packages. fitness (or objective function) of each chromosome and selecting the chromosomes according to their 3.6.3 Tabu search fitness values (using methods such as mapping onto Roulette Wheel). Additional chromosomes are then Tabu search [70] uses special memory structures generated using crossover and mutation functions. (short-term and long-term) during the search Amaran, Sahinidis, Sharda and Bury 13

process that allow the method to go beyond local optimality to explore promising regions of the search space. The basic form of tabu search consists of a modified neighborhood search procedure that employs adaptive memory to keep track of relevant solution history, together with strategies for exploiting this memory [66]. More advanced forms of tabu search and its applications are described in [72]. Algorithm 5 Basic scatter search procedure 3.6.4 Scatter search for SO Require: An initial set of trial points x Scatter search and its generalized form, path ∈ relinking, were originally introduced by Glover P , chosen to be diversified according to a and Laguna [73]. Scatter search differs from pre-specified metric other evolutionary approaches (such as Genetic 1: t simulate(x ), where j =1,..., P j ← j | | Algorithms (GA)) by using strategic designs and 2: k 0 search path construction from a population of ← 3: solutions as compared to randomization (by crossover Use a comparison procedure (such as and mutation in GA). Similar to Tabu search, Scatter ranking and selection) to gather the best Search also utilize adaptive memory in storing best b solutions (based on objective value solutions [73, 133]. Algorithm 5 provides the scatter or diversity) from the current set of search algorithm. solutions P , called the reference set, Rk 4: R−1 = ∅ 3.7 Model-based methods 5: while under simulation budget and R = k 6 Model-based simulation optimization methods at- Rk−1 do tempt to build a probability distribution over the 6: k k +1 space of solutions and use it to guide the search ← 7: Choose Si R, where i = 1,...,r process. Use a subset⊂ generation procedure to { select r subsets of set R, to be used as a 3.7.1 Estimation of distribution algorithms basis for generating new solution points } Estimation of distribution algorithms (EDAs) [126] 8: for i =1 to r do are model-based methods that belong to the 9: Combine the points in Si, to form evolutionary computation field. However, generation new solution points, xj, where j = of new candidate solutions is done by sampling from ∈ J P +1,..., P + J, using weighted linear the inferred probability distribution over the space of | | | | solutions, rather than, say, a genetic operator such combinations, for example as crossover or mutation. A comprehensive review of 10: t simulate(x ), j j ← j ∈ J estimation of distribution algorithms is presented in sample the objective function at new [64]. EDAs usually consider interactions between the trial{ solutions problem variables and exploit them through different } 11: R P probability models. Update sets k, Cross-entropy methods and Model Reference 12: end for Adaptive Search (MRAS) are discussed next and can 13: end while be seen as specific instances of EDAs.

Cross-Entropy Methods. Cross-entropy methods first sample randomly from a chosen probability distribution over the space of decision variables. For each sample, which is a vector defining a point in decision space, a corresponding function evaluation is obtained. Based on the function values Simulation optimization: A review of algorithms and applications 14

observed, a pre-defined percentile of the best samples decision variables are correlated, the covariance of the are picked. A new distribution is built around distribution will reflect this. this ‘elite set’ of points via maximum likelihood The immediately apparent merits of cross-entropy estimation or some other fitting method, and the methods are that they are easy to implement, process is repeated. One possible method that require few algorithmic parameters, are based on implements cross-entropy is formally described in fundamental principles such as KL-divergence and Algorithm 6. maximum likelihood, and give consistently accurate results [124]. A potential drawback is that cross- Algorithm 6 Pseudocode for a simple cross- entropy may require a significant number of new entropy implementation samples at every iteration. It is not clear as to how this would affect performance if samples were Require: θ, an initial set of parameters for expensive to obtain. The cross-entropy method has a pre-chosen distribution p(x; θ) over the analogs in simulated annealing, genetic algorithms, set of decision variables; k, a number and ant colony optimization, but differs from each of these in important ways [44]. of simulations to be performed; e, the More detailed information on the use of cross- number of elite samples representing the entropy methods for optimization can be found top δ percentile of the k samples in [44], a tutorial on cross-entropy and in 1: while under simulation budget and not [171], a monograph. The cross-entropy webpage, converged do http://iew3.technion.ac.il/CE/ provides up-to-date information on progress in the field. 2: for i =1 k do → 3: sample x from p(x; θ) i Model reference adaptive search 4: t simulate(x ) i ← i (MRAS). The MRAS method [91, 92] is closely 5: end for related to the cross-entropy method. It also works 6: E by minimizing the Kullback-Leibler divergence to ←∅ 7: for i =1 e do update the parameters of the inferred probability → distribution. However, the parameter update 8: E arg max ∈ t j ← i/E i step involves the use of a sequence of implicit 9: end for probability distributions. In other words, while the 10: p(x; θ) fit(xE) cross-entropy method uses the optimal importance ← 11: end while sampling distribution for parameter updates, MRAS minimizes the KL-divergence with respect to the distribution in the current iteration, called the The method is guaranteed (probabilistically) to reference model. converge to a local optimum, but it also incorporates an exploration component as random samples are obtained at each step. However, the intuition Covariance Matrix Adaptation–Evolu- behind the selection of subsequent samples can be tion Strategy (CMA-ES). In the CMA- shown to be analogous to minimizing the Kullback- ES algorithm [81], new samples are obtained from a Leibler divergence (KL-divergence) between the multivariate normal distribution, and inter-variable optimal importance sampling distribution and the dependencies are encoded in the covariance matrix. distribution used in the current iterate [171]. The CMA-ES method provides a way to update the There exist variants of the cross-entropy method covariance matrix. Updating the covariance matrix to address both continuous [124] and discrete is analogous to learning an approximate inverse optimization [170] problems. A possible modification Hessian, as is used in Quasi-Newton methods in is to use mixtures of distributions from current and mathematical programming. The update of the previous iterations, with the current distribution mean and covariance is done by maximizing the weighted higher. This can be done by linearly likelihood of previously successful candidate solutions interpolating the mean covariance in the case of and search steps, respectively. This is in contrast to Gaussian distributions. This also helps in avoiding other EDAs and the cross-entropy method, where the singular covariance matrices. Cross-entropy can covariance is updated by maximizing the likelihood also deal with noisy function evaluations, with of the successful points. Other sophistications such irrelevant decision variables, and constraints [124]. If as step-size control, and weighting of candidate Amaran, Sahinidis, Sharda and Bury 15

solutions are part of modern implementations [80]. have some sort of optimization routine; fewer still have black-box optimizers that interact with the simulation. 3.7.2 Ant colony optimization

Ant colony optimization methods [52, 51] are heuris- 4.2 Academic implementations of simulation tic methods that have been used for combinatorial optimization optimization problems. Conceptually, they mimic the behavior of ants to find shortest paths between their Table 5 contains a small subset of academic colony and food sources. Ants deposit pheromones implementations of SO algorithms, and classifies as they walk; and are more likely to choose paths them by type. Some of these are available with higher concentration of pheromones. This for download from the web, some have code phenomenon is incorporated in a pheromone update with suggested parameters in corresponding papers rule, which increases the pheromone content in themselves, and others are available upon request components of high-quality solutions, and causes from the authors. evaporation of pheromones in less favorable regions. Probability distributions are used to make the transition between each iteration. These methods 5 Comparison of algorithms differ from EDAs in that they use an iterative construction of solutions. As far as comparisons between algorithms are This and other algorithms that incorporate concerned, the literature does not yet provide a self-organization in biological systems are said to use comprehensive survey of the performance of different the concept of ‘swarm intelligence’. implementations and approaches on large test beds. In this regard, simulation optimization lags behind other optimization fields such as linear, integer, and nonlinear programming, global optimization and 3.8 Lipschitzian optimization even derivative-free optimization, where the first Lipschitzian optimization is a class of space- comprehensive comparison appeared in 2013 [166]. A partitioning algorithms for performing global op- study of prior comparisons in simulation optimization timization, where the Lipschitz constant is pre- is provided by [194], but these comparisons are specified. This enables the construction of global fairly dated, are inconclusive about which algorithms search algorithms with convergence guarantees. The perform better in different situations, and compare caveat of having prior knowledge of the Lipschitz only a small subset of available algorithms. One constant is overcome by the DIRECT (DIviding difficulty lies in the inherent difficulty of comparing RECTangles) algorithm [101] for deterministic solutions between algorithms over true black-box continuous optimization problems. An adaptation of simulations, as one does not usually know the this for noisy problems is provided in [47]. true optimal point and can only compare between noisy estimates observed by the solvers. Less impeding difficulties, but difficulties nonetheless, 4 Software include the need to interface algorithms to a common wrapper, the objective comparison with solvers that 4.1 Simulation optimization in commercial incorporate random elements as their results may not simulation software be reproducible, and lack of standard test simulations for purposes of benchmarking. Many discrete-event simulation packages incorporate The benchmarking of algorithms in mathematical some methodology for performing optimization. programming is usually done by performance profiles A comprehensive listing of simulation software, [50], where the graphs show the fraction of problems the corresponding vendors, and the optimization solved after a certain time. For derivative-free packages and techniques they use can be found in algorithms, data profiles are commonly used [141], Table 4. More details on the specific optimization where the fraction of problems solved after a routines can be found in [128]. OR/MS-Today, the certain number of iterations (function evaluations) online magazine of INFORMS, conducts a biennial or ‘simplex gradients’ is shown. The definition of survey of simulation software packages, the latest when a problem is ‘solved’ may vary—when the true of which is available at [150]. The survey lists global optimum is known, the solutions found within 43 simulation software packages, and 31 of these a certain tolerance of this optimal value may be called Simulation optimization: A review of algorithms and applications 16

Table 4: Simulation optimization packages in commercial simulation software

Optimization Vendor Simulation Optimization package software methodology supported AutoStat Applied Materials, AutoMod Evolutionary Inc. strategy

Evolutionary Imagine That, Inc. ExtendSim Evolutionary Optimizer strategy

OptQuest OptTek Systems, FlexSim, @RISK, Scatter search, Inc. Simul8, Simio, tabu search, neural SIMPROCESS, networks, integer AnyLogic, programming Arena, Crystal Ball, Enterprise Dynamics, ModelRisk SimRunner ProModel Corp. ProModel, Genetic algorithms MedModel, and evolutionary ServiceModel strategies

RISKOptimizer Palisade Corp. @RISK Genetic algorithm

WITNESS Lanner Group, Inc. WITNESS Simulated annealing, Optimizer tabu search, hill climbing GoldSim GoldSim Technology GoldSim Box’s complex Optimizer Group method

Plant Siemens AG Siemens PLM soft- Genetic algorithm Simulation ware Optimizer ChaStrobeGA N/A Stroboscope Genetic algorithm Global The MathWorks SimEvents (Matlab) Genetic algorithms, Optimization simulated annealing, toolbox pattern search Amaran, Sahinidis, Sharda and Bury 17

Table 5: Academic simulation optimization implementations

Algorithm Type Citation Year Continuous SPSA Stochastic approximation [185] 1992 SPSA 2nd Stochastic approximation [185] 1999 Order SKO Global response surface [93] 2006 CE method Cross-entropy [124] 2006 APS Nested partitioning [105] 2007 SNOBFIT Multi-start local response surface [97] 2008 CMA-ES Evolutionary strategy [80] 2011 KGCP Global response surface [177] 2011 STRONG Local response surface, trust region [32] 2011 GR Golden region search [106] 2011 SNM Direct search (Nelder-Mead) [31] 2012 DiceOptim Global response surface [169] 2012 Discrete KG Global response surface [57] 2009 COMPASS Neighborhood search (integer-ordered problems) [206] 2010 R-SPLINE Neighborhood search (integer-ordered problems) [198] 2012 Discrete and continuous MRAS Estimation of distribution [91, 92] 2005 NOMADm Mesh adaptive direct search [1] 2007 solutions, but when this optimum is not known, the exist a standard method to compare simulation solvers that find the best solution (within a tolerance) optimization algorithms on large test beds. Many for a problem, with respect to the other solvers papers perform several macroreplications and report being compared, may be said to have solved the the macroreplicate average of the best sample means problem. The latter metric may also be used when (along with the associated sample variance) at the function evaluations are expensive, and no solver is end of the simulation budget. The issue with this is able to reach within this tolerance given the limited that the performance of the algorithms with different simulation budget. simulation budgets is not seen, as in the case of In both of these cases, the output of the performance or data profiles. Other papers report simulations are deterministic, and so it is clear as the average number of evaluations taken to find a to which algorithms have performed better than sample mean that is within the global tolerance others on a particular problem. In simulation for each problem. Here, results are listed for each optimization, however, usually one does not know problem and one does not get an idea of overall the true solution for the black box system, nor does performance. In addition, the difference in sample one see deterministic output. All that one possesses variance estimates is not highlighted. As simulation are mean values and sample variances obtained from optimization develops, there is also a need for sample paths at different points. There does not methods of comparison of algorithms on test beds Simulation optimization: A review of algorithms and applications 18

with statistically significant number of problems. wide variety of simulation languages. This increased With regard to standardized simulation testbeds, use is reflected in the identification and application to our knowledge, the only testbed that provides of simulation and simulation optimization methods practical simulations for testing simulation opti- to diverse fields in science, engineering, and business. mization algorithms is available at www.simopt.org There also exist strong analogies between, and ideas [153]. At the point of writing this paper, that may be borrowed from recent progress in related just 20 continuous optimization problems were fields. All of these factors, along with the ever available from this repository. Most testing increasing number of publications and rich literature and comparisons happen with classical test prob- in this area, clearly indicate the interest in the field of lems in nonlinear optimization (many of which simulation optimization, and we have tried to capture have been compiled in [166] and available at this in this paper. http://archimedes.cheme.cmu.edu/?q=dfocomp), to With increased growth and interest in the which stochastic noise has been added. There is a field, there come also opportunities. Potential need for more such repositories, not only for testing of directions for the field of simulation optimization algorithms over statistically significant sizes of prob- are almost immediately apparent. Apart from the lem sets, but for comparison between different classes ability to handle simulation outputs from any well- of algorithms. The need for comparison is evident, defined probability distribution, the effective use of given the sheer number of available approaches to variance reduction techniques when possible, and solving simulation optimization problems, and the the improvement in theory and algorithms, there is lack of clarity and lack of consensus on which types a requirement to address (1) large-scale problems of algorithms are suitable in which contexts. with combined discrete/continuous variables; (2) As observed by several papers [61, 194, 88], there the ability to effectively handle stochastic and continues to exist a significant gap between research deterministic constraints of various kinds; (2) the and practice in terms of algorithmic approaches. effective utilization of parallel computing at the Optimizers bundled with simulation software, as linear algebra level, sample replication level, iteration observed in Section 4, tend to make use of algorithms level, as well as at the algorithmic level; (3) the which seem to work well but do not come with effective handling of multiple simulation outputs; provable statistical properties or guarantees of local (4) the incorporation of performance measures or global convergence. Academic papers, on the other other than expected values, such as risk; (5) hand, emphasize methods that are more sophisticated the continued consolidation of various techniques and prove convergence properties. One reason that and their potential synergy in hybrid algorithms; may contribute to this is that very few simulation (6) the use of automatic differentiation techniques optimization algorithms arising from the research in the estimation of simulation derivatives when community are easily accessible. We wholeheartedly possible; (7) the continued emphasis on providing encourage researchers to post their executable files, if guarantees of convergence to optima for local and not their source code. This could not only encourage global optimization routines in general settings; practitioners to use these techniques in practice, (8) the availability and ease of comparison of the but allow for comparisons between methods and performance of available approaches on different the development of standardized interfaces between applications; and (9) the continued reflection of simulations and simulation optimization software. sophisticated methodology arising from the literature in commercial simulation packages. 6 Conclusions References The field of simulation optimization has progressed significantly in the last decade, with several [1] Abramson, M.A.: NOMADm version 4.5 User’s new algorithms, implementations, and applications. Guide. Air Force Institute of Technology, Contributions to the field arise from researchers and Wright-Patterson AFB, OH (2007) practitioners in the industrial engineering/operations [2] Alkhamis, T.M., Ahmed, M.A., Tuan, V.K.: research, mathematical programming, statistics and Simulated annealing for discrete optimization machine learning, as well as the computer science with estimation. European Journal of communities. The use of simulation to model Operational Research 116, 530–544 (1999) complex, dynamic, and stochastic systems has only increased with computing power and availability of a Amaran, Sahinidis, Sharda and Bury 19

[3] Alrefaei, M.H., Andrad´ottir, S.: A simulated simulation. Journal of the Operational annealing algorithm with constant temperature Research Society 60(6), 735–746 (2009) for discrete stochastic optimization. Manage- [14] Ayvaz, M.T.: A linked simulation-optimization ment Science 45, 748–764 (1999) model for solving the unknown groundwater [4] Amaran, S., Sahinidis, N.V., Sharda, B., Bury, pollution source identification problems. Jour- S.J.: Simulation optimization: A review of nal of Contaminant Hydrology 117(1–4), 46–59 algorithms and applications. 4OR–A Quarterly (2010) Journal of Operations Research 12, 301–333 [15] Azadivar, F.: A tutorial on simulation (2014) optimization. In: J.J. Swain, D. Goldsman, [5] Ammeri, A., Hachicha, W., Chabchoub, H., R.C. Crain, J.R. Wilson (eds.) Proceedings of Masmoudi, F.: A comprehensive literature re- the 1992 Winter Simulation Conference, pp. view of mono-objective simulation optimization 198–204 (1992) methods. Advances in Production Engineering [16] Azadivar, J.: Simulation optimization method- & Management 6(4), 291–302 (2011) ologies. In: P.A. Farrington, H.B. Nembhard, [6] Anderson, E.J., Ferris, M.C.: A direct search D.T. Sturrock, G.W. Evans (eds.) Proceedings algorithm for optimization with noisy function of the 1999 Winter Simulation Conference, pp. evaluations. SIAM Journal on Optimization 93–100 (1999) 11, 837–857 (2001) [17] Balakrishna, R., Antoniou, C., Ben-Akiva, M., [7] Andrad´ottir, S.: Chapter 9: Simulation Koutsopoulos, H.N., Wen, Y.: Calibration of optimization. In: J. Banks (ed.) Handbook microscopic traffic simulation models: Methods of Simulation: Principles, Methodology, Ad- and application. Transportation Research vances, Applications, and Practice. John Wiley Record: Journal of the Transportation & Sons, New York (1998) Research Board 1999(1), 198–207 (2007) [8] Andrad´ottir, S.: An overview of simulation [18] Bangerth, W., Klie, H., Matossian, V., optimization via random search. In: S.G. Parashar, M., Wheeler, M.F.: An autonomic Henderson, B.L. Nelson (eds.) Handbooks in reservoir framework for the stochastic opti- Operations Research and Management Science: mization of well placement. Cluster Computing Simulation, vol. 13, chap. 20, pp. 617–631. 8(4), 255–269 (2005) Elsevier (2006) [19] Barton, R.R., Ivey, Jr., J.S.: Nelder- [9] Andrad´ottir, S.: Simulation optimization. Mead simplex modifications for simulation In: Handbook of Simulation: Principles, optimization. Management Science 42, 954– Methodology, Advances, Applications and 973 (1996) Practice, pp. 307–333. John Wiley & Sons [20] Barton, R.R., Meckesheimer, M.: Metamodel- (2006) based simulation optimization. In: S. Hender- [10] Andrad´ottir, S., Kim, S.H.: Fully sequential son, B. Nelson (eds.) Handbook in Operations procedures for comparing constrained systems Research and Management Science: Simulation via simulation. Naval Research Logistics 57(5), 13, pp. 535–574. Elsevier (2006) 403–421 (2010) [21] Bechhofer, R.E., Santner, T.J., Goldsman, [11] de Angelis, V., Felici, G., Impelluso, P.: D.M.: Design and Analysis of Experiments for Integrating simulation and optimisation in Statistical Selection, Screening, and Multiple health care centre management. European Comparisons. John Wiley & Sons (1995) Journal of Operational Research 150, 101–114 [22] van Beers, A.C., Kleijnen, J.P.C.: Kriging (2003) interpolation in simulation: A survey. Proceed- [12] Ang¨un, E.: Black box simulation optimization: ings of the 2004 Winter Simulation Conference Generalized response surface methodology. 1, 121–129 (2004) Ph.D. thesis, Tilburg University (2004) [23] Bertsimas, D., Tsitsiklis, J.: Simulated [13] Ang¨un, E., Kleijnen, J.P.C., Hertog, D.D., annealing. Statistical Science 8(1), 10–15 Gurkan, G.: Response surface methodology (1993) with stochastic constraints for expensive Simulation optimization: A review of algorithms and applications 20

[24] Bettonvil, B., del Castillo, E., Kleijnen, J.P.C.: Transactions on Automatic Control 41, 1227– Statistical testing of optimality conditions in 1231 (1996) multiresponse simulation-based optimization. [35] Chen, C.H., Lee, L.H.: Stochastic Simulation European Journal of Operational Research Optimization: An Optimal Computing Budget 199, 448–458 (2009) Allocation. System Engineering and Oper- [25] Bhatnagar, S.: Adaptive multivariate three- ations Research. World Scientific Publishing timescale stochastic approximation algorithms Company (2010) for simulation based optimization. ACM [36] Chen, C.H., Y¨ucesan, E., Dai, L., Chen, Transactions on Modeling and Computer H.C.: Optimal budget allocation for discrete- Simulation 15(1), 74–107 (2005) event simulation experiments. IIE Transactions [26] Bianchi, L., Dorigo, M., Gambardella, L.M., 42(1), 60–70 (2009) Gutjahr, W.J.: A survey on metaheuristics for [37] Chen, H., Schmeiser, B.W.: Retrospective stochastic combinatorial optimization. Natural optimization algorithms for stochastic root Computing 8(2), 239–287 (2009) finding. In: J. Tew, S. Manivannan, [27] Birge, J.R., Louveaux, F.: Introduction to D. Sadowski, A. Seila (eds.) Proceedings of Stochastic Programming, 2nd edn. Springer 1994 Winter Simulation Conference, pp. 255– (2011) 261 (1994) [28] Box, G.E.P., Wilson, K.B.: On the experimen- [38] Chick, S.E.: Subjective probability and tal attainment of optimum conditions. Journal bayesian methodology. In: S.G. Henderson, of the Royal Statistical Society XIII(1), 1–35 B.L. Nelson (eds.) Simulation, Handbooks in (1951) Operations Research and Management Science, vol. 13, pp. 225–257. Elsevier, Amsterdam [29] Carson, Y., Maria, A.: Simulation opti- (2006) mization: Methods and applications. In: S. Andrad´ottir, K.J. Healy, D.H. Winters, B.L. [39] Cho, J., Dorfman, K.D.: Brownian dynamics Nelson (eds.) Proceedings of the 1997 Winter simulations of electrophoretic DNA separations Simulation Conference, pp. 118–126 (1997) in a sparse ordered post array. Journal of Chromatography A 1217, 5522–5528 (2010) [30] Chang, K.H.: Stochastic trust region response surface convergent method for continuous [40] Cohn, D.A., Ghahramani, Z., Jordan, M.I.: simulation optimization. Ph.D. thesis, Purdue Active learning with statistical models. Journal University (2008) of Artificial Intelligence Research 4, 129–145 (1996) [31] Chang, K.H.: Stochastic Nelder-Mead simplex method–A new globally convergent direct [41] Collins, N.E., Eglese, R.W., Golden, B.L.: search method for simulation optimization. Simulated annealing–an annotated bibliogra- European Journal of Operational Research phy. American Journal of Mathematical 220, 684–694 (2012) Management Sciences 8, 209–308 (1988) [32] Chang, K.H., Hong, L.J., Wan, H.: [42] Conn, A.R., Gould, N.I.M., Toint, P.L.: Stochastic trust-region response-surface Trust-region methods. MOS-SIAM Series on method (STRONG)—A new response-surface Optimization (2000) framework for simulation optimization. [43] Conn, A.R., Scheinberg, K., Vicente, L.N.: INFORMS Journal on Computing 25(2), Introduction to derivative-free optimization. 230–243 (2013) SIAM, Philadelphia, PA (2009) [33] Chen, C.H.: An effective approach to smartly [44] de Boer, P.T., Kroese, D.P., Mannor, S., allocate computing budget for discrete event Rubinstein, R.Y.: A tutorial on the cross- simulation. In: Proceedings of the 34th IEEE entropy method. Annals of Operations Conference on Decision and Control, pp. 2598– Research 134, 19–67 (2005) 2605 (1995) [45] Deng, G.: Simulation-based optimization. [34] Chen, C.H.: A lower bound for the correct Ph.D. thesis, University of Wisconsin–Madison subset selection probability and its application (2007) to discrete event system simulations. IEEE Amaran, Sahinidis, Sharda and Bury 21

[46] Deng, G., Ferris, M.C.: Adaptation of the [57] Frazier, P., Powell, W., Dayanik, S.: The UOBYQA algorithm for noisy functions. In: knowledge-gradient policy for correlated nor- L.F. Perrone, F.P. Wieland, J. Liu, B.G. mal beliefs. INFORMS Journal on Computing Lawson, D.M. Nicol, R.M. Fujimoto (eds.) 21(4), 599–613 (2009) Proceedings of the 2006 Winter Simulation [58] Frazier, P.I.: Knowledge-gradient methods for Conference, pp. 312–319 (2006) statistical learning. Ph.D. thesis, Princeton [47] Deng, G., Ferris, M.C.: Extension of the University (2009) DIRECT optimization algorithm for noisy [59] Fu, M.C.: Optimization via simulation: A functions. In: S.G. Henderson, B. Biller, M.H. review. Annals of Operations Research 53, Hsieh, J. Shortle, J.D. Tew, R.R. Barton (eds.) 199–247 (1994) Proceedings of the 2007 Winter Simulation Conference, pp. 497–504 (2007) [60] Fu, M.C.: Optimization for simulation: Theory vs. practice. SIAM Journal on Computing [48] Dengiz, B., Akbay, K.S.: Computer simulation 14(3), 192–215 (2002) of a PCB production line: Metamodeling approach. International Journal of Production [61] Fu, M.C., Andrad´ottir, S., Carson, J.S., Economics 63(2), 195–205 (2000) Glover, F.W., Harrell, C.R., Ho, Y.C., Kelly, J.P., Robinson, S.M.: Integrating optimization [49] Dhivya, M., Sundarambal, M., Anand, L.N.: and simulation: Research and practice. In: Energy efficient computation of data fusion J.A. Joines, R.R. Barton, K. Kang, P.A. in wireless sensor networks using cuckoo-based Fishwick (eds.) Proceedings of the 2000 Winter particle approach (cbpa). International Journal Simulation Conference (2000) of Communications, Network and System Sciences 4(4), 249–255 (2011) [62] Fu, M.C., Glover, F.W., April, J.: Simulation Optimization: A review, new developments, [50] Dolan, E.D., Mor´e, J.J.: Benchmarking and applications. In: M.E. Kuhl, N.M. Steiger, optimization software with performance pro- F.B. Armstrong, J.A. Joines (eds.) Proceedings files. Mathematical Programming 91, 201–213 of the 2005 Winter Simulation Conference, pp. (2002) 83–95 (2005) [51] Dorigo, M., Blum, C.: Ant colony optimization [63] Fu, M.C., Hill, S.D.: Optimization of discrete theory: A survey. Theoretical Computer event systems via simulataneous perturbation Science 344(2–3), 243–278 (2005) stochastic approximation. IIE Transactions [52] Dorigo, M., St¨utzle, T.: Ant colony optimiza- 29(233–243) (1997) tion. MIT Press, Cambridge, MA, USA (2004) [64] Fu, M.C., Hu, J., Marcus, S.I.: Model-based [53] Driessen, L.T.: Simulation-based optimization randomized methods for global optimization. for product and process design. Ph.D. thesis, In: Proceedings of the 17th International Sym- Tilburg University (2006) posium on Mathematical Theory of Networks and Systems, pp. 355–363. Kyoto, Japan (1996) [54] Ernst, D., Glavic, M., Stan, G.B., Mannor, S., Wehenkel, L.: The cross-entropy method [65] Fu, M.C., Hu, J.Q.: Conditional Monte for power system combinatorial optimization Carlo: Gradient estimation and optimization problems. In: Power Tech, pp. 1290–1295. applications. Kluwer Academic Publishers IEEE (2007) (1997) [55] Ferris, M.C., Deng, G., Fryback, D.G., [66] Gendreau, M., Potvin, J.Y.: Tabu search. Kuruchittham, V.: Breast cancer epidemiol- In: Handbook of Metaheuristics, International ogy: Calibrating simulations via optimization. Series in Operations Research & Management Oberwolfach Reports 2 (2005) Science, vol. 146, second edn., pp. 41–60. Springer (2010) [56] Figueira, G., Almada-Lobo, B.: Hybrid simulation-optimization methods: A taxon- [67] Gerencs´er, L., Kozmann, G., V´ag´o, Z., omy. Simulation Modelling Practice and Haraszti, K.: The use of the spsa method in Theory 46, 118–134 (2014) ecg analysis. IEEE Transactions on Biomedical Engineering 49(10), 1094–1101 (2002) Simulation optimization: A review of algorithms and applications 22

[68] Gittins, J.C.: Multi-armed bandit allocation [81] Hansen, N.: The CMA evolution strategy: indices. Wiley-Interscience Series in Systems A comparing review. In: J.A. Lozano, and Optimization. John Wiley & Sons (1989) P. Larra˜naga, I. Inza, E. Bengoetxea (eds.) Towards a new evolutionary computation. [69] Glasserman, P.: Gradient Estimation via Advances on estimation of distribution algo- Perturbation Analysis. Kluwer Academic rithms, pp. 75–102. Springer (2006) Publishers (1991) [82] Healy, K., Schruben, L.W.: Retrospective [70] Glover, F.: Tabu search: A tutorial. Interfaces simulation response optimization. In: B.L. 20 (4), 77–94 (1990) Nelson, D.W. Kelton, G.M. Clark (eds.) [71] Glover, F., Hanafi, S.: Tabu search and finite Proceedings of the 1991 Winter Simulation convergence. Discrete Applied Mathematics Conference, pp. 954–957 (1991) 119(1–2), 3–36 (2002) [83] Hill, S.D., Fu, M.C.: Transfer optimization via [72] Glover, F., Laguna, M.: Tabu Search. Kluwer simultaneous perturbation stochastic approxi- Academic Publishers, Boston (1997) mation. In: C. Alexopoulos, K. Kang, W.R. Lilegdon, D. Goldsman (eds.) Proceedings of [73] Glover, F., Laguna, M.: Fundamentals of the 1995 Winter Simulation Conference, pp. scatter search and path relinking. Control and 242–249 (1995) Cybernetics 29(3), 653–684 (2000) [84] Ho, Y.C.: An explanation of ordinal opti- [74] Goldsman, D., Nelson, B.L.: Comparing mization: Soft computing for hard problems. systems via simulation. In: J. Banks Information Sciences 113, 169–192 (1999) (ed.) Handbook of Simulation: Principles, [85] Ho, Y.C., Cao, X.R.: Discrete Event Dynamic Methodology, Advances, Applications, and Systems and Perturbation Analysis. Kluwer Practice, chap. 8. John Wiley & Sons (1998) Academic Publishers (1991) [75] Gong, W.B., Ho, Y.C., Zhai, W.: Stochastic [86] Hochberg, Y., Tamhane, A.C.: Multiple comparison algorithm for discrete optimization Comparison Procedures. John Wiley & Sons, with estimation. SIAM Journal on Optimiza- New York (1987) tion 10, 384–404 (49) [87] Hong, L.J., Nelson, B.L.: Discrete optimization [76] Griewank, A., Walther, A.: Evaluating via simulation using COMPASS. Operations Derivatives: Principles and Techniques of Research 54(1), 115–129 (2006) Algorithmic Differentiation, 2nd edn. No. 105 in Other Titles in Applied Mathematics. SIAM, [88] Hong, L.J., Nelson, B.L.: A brief introduction Philadelphia, PA (2008). URL http://www. to optimization via simulation. In: M.D. ec-securehost.com/SIAM/OT105.html Rossetti, R.R. Hill, B. Johansson, A. Dunkin, R.G. Ingalls (eds.) Proceedings of the 2009 [77] G¨urkan, G., Ozge, A.Y., Robinson, S.M.: Winter Simulation Conference (2009) Sample path optimization in simulation. In: [89] Hooke, R., Jeeves, T.A.: Direct search solution J. Tew, S. Manivannan, D. Sadowski, A. Seila of numerical and statistical problems. Journal (eds.) Proceedings of 1994 Winter Simulation of the Association for Computing Machinery 8, Conference, pp. 247–254 (1994) 212–219 (1961) [78] Hajek, B.: Cooling schedules for optimal an- [90] Hsu, J.C.: Multiple Comparisons: theory and nealing. Mathematics of Operations Research methods. CRC Press, Boca Raton (1996) 13, 311–329 (1988) [91] Hu, J., Fu, M.C., Marcus, S.I.: Stochastic [79] Hall, J.D., Bowden, R.O., Usher, J.M.: Using optimization using model reference adaptive evolution strategies and simulation to optimize search. In: M.E. Kuhl, N.M. Steiger, F.B. a pull production system. Journal of Materials Armstrong, J.A. Joines (eds.) Proceedings of Processing Technology 61(1–2), 47–52 (1996) the 2005 Winter Simulation Conference, pp. [80] Hansen, N.: The CMA Evolution Strategy: A 811–818 (2005) tutorial. [92] Hu, J., Fu, M.C., Marcus, S.I.: A model http://www.lri.fr/~hansen/cmaesintro. reference adaptive search method for global html optimization. Operations Research 55(3), 549– 568 (2007) Amaran, Sahinidis, Sharda and Bury 23

[93] Huang, D., Allen, T.T., Notz, W.I., Zeng, [104] Kabirian, A.: Continuous optimization via N.: Global optimization of stochastic black-box simulation using golden region search. Ph.D. systems via sequential kriging meta-models. thesis, Iowa State University (2009) Journal of Global Optimization 34, 441–466 [105] Kabirian, A., Olafsson,´ S.: Allocation of (2006) simulation runs for simulation optimization. [94] Humphrey, D.G., Wilson, J.R.: A revised In: S.G. Henderson, B. Biller, M.H. Hsieh, simplex search procedure for stochastic simula- J. Shortle, J.D. Tew, R.R. Barton (eds.) tion response-surface optimization. INFORMS Proceedings of the 2007 Winter Simulation Journal on Computing 12(4), 272–283 (2000) Conference, pp. 363–371 (2007) [95] Hunter, S.R., Pasupathy, R.: Optimal [106] Kabirian, A., Olafsson,´ S.: Continuous sampling laws for stochastically constrained optimization via simulation using golden region simulation optimization on finite sets. IN- search. European Journal of Operational FORMS Journal on Computing 25(3), 527–542 Research (2011) (2013) [107] Kenne, J.P., Gharbi, A.: A simulation [96] Hutchison, D.W., Hill, S.D.: Simulation optimization approach in production planning optimization of airline delay with constraints. of failure prone manufacturing systems. Jour- In: B.A. Peters, J.S. Smith, D.J. Medeiros, nal of Intelligent Manufacturing 12, 421–431 M.W. Rohrer (eds.) Proceedings of the 2001 (2001) Winter Simulation Conference, pp. 1017–1022 [108] Khan, H.A., Zhang, Y., Ji, C., Stevens, (2001) C.J., Edwards, D.J., O’Brien, D.: Optimizing [97] Huyer, W., Neumaier, A.: SNOBFIT – Stable polyphase sequences for orthogonal netted noisy optimization by branch and fit. ACM radar. Signal Processing Letters, IEEE 13(10), Transactions on Mathematical Software 35, 1– 589–592 (2006) 25 (2008) [109] Kiefer, J., Wolfowitz, J.: Stochastic estimation [98] Irizarry, M.D.L.A., Wilson, J.R., Trevino, J.: of the maximum of a regression function. The A flexible simulation tool for manufacturing- Annals of Mathematical Statistics 23(3), 462– cell design, II: Response surface analysis and 466 (1952) case study. IIE Transactions (2001) [110] Kim, S.H.: Comparison with a standard via [99] Jacobson, S.H., Schruben, L.W.: Techniques fully sequential procedures. ACM Transactions for simulation response optimization. Opera- on Modeling and Computer Simulation 15(2), tion Research Letters 8, 1–9 (1989) 155–174 (2005) [100] Jia, Q.S., Ho, Y.C., Zhao, Q.C.: Comparison [111] Kim, S.H., Nelson, B.L.: Selecting the best of selection rules for ordinal optimization. system. In: S.G. Henderson, B.L. Nelson Mathematical and Computer Modelling 43(9– (eds.) Handbooks in Operations Research and 10), 1150–1171 (2006) Management Science: Simulation, chap. 17, pp. 501–534. Elsevier Science (2006) [101] Jones, D.R., Perttunen, C.D., Stuckman, B.E.: Lipschitzian optimization without the [112] Kim, S.H., Nelson, B.L.: Recent advances in Lipschitz constant. Journal of Optimization ranking and simulation. In: S.G. Henderson, Theory and Application 79, 157–181 (1993) B. Biller, M.H. Hsieh, J. Shortle, J.D. Tew, R.R. Barton (eds.) Proceedings of the 2007 [102] Jones, D.R., Schonlau, M., Welch, W.J.: Winter Simulation Conference, pp. 162–172 Efficient global optimization of expensive (2007) black-box functions. Journal of Global Optimization 13, 455–492 (1998) [113] Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization by simulated annealing. Science [103] Jung, J.Y., Blau, G., Pekny, J.F., Reklaitis, 220, 671–680 (1983) G.V., Eversdyk, D.: A simulation based optimization approach to supply chain manage- [114] Kleijnen, J.P.C.: Simulation and optimization ment under demand uncertainty. Computers & in production planning: A case study. Decision Chemical Engineering 28, 2087–2106 (2004) Support Systems 9, 269–280 (1993) Simulation optimization: A review of algorithms and applications 24

[115] Kleijnen, J.P.C.: Design and Analysis of [126] Larra˜naga, P., Lozano, J.A.: Estimation Simulation Experiments. Springer, New York of distribution algorithms: A new tool for (2008) evolutionary computation. Kluwer Academic Publishers (2002) [116] Kleijnen, J.P.C.: Kriging metamodeling in simulation: A review. European Journal of [127] Lau, T.W.E., Ho, Y.C.: Universal alignment Operational Research 192(3), 707–716 (2009) probabilities and subset selection for ordinal [117] Kleijnen, J.P.C., van Beers, W.C.M.: Robust- optimization. Journal of Optimization Theory ness of kriging when interpolating in random and Applications 93(3), 455–489 (1997) simulation with heterogeneous variances: Some [128] Law, A.M., Kelton, W.D.: Simulation experiments. European Journal of Operational modeling and analysis, 3rd edn. McGraw-Hill, Research 165, 826–834 (2005) Singapore (2000) [118] Kleijnen, J.P.C., Beers, W.C.M., van Nieuwen- [129] Lee, L.H., Pujowidianto, N.A., Li, L.W., Chen, huyse, I.: Expected improvement in efficient C.H., Yap, C.M.: Approximate simulation global optimization through bootstrapped budget allocation for selecting the best design kriging. Journal of Global Optimization 54(1), in the presence of stochastic constraints. IEEE 59–73 (2012) Transactions on Automatic Control 57(11), [119] Kleinman, N.L., Hill, S.D., Ilenda, V.A.: 2940–2945 (2012) Spsa/simmod optimization of air traffic delay [130] Li, Y.: A simulation-based evolutionary cost. In: Proceedings of the 1997 American approach to LNA circuit design optimiza- Control Conference, vol. 2, pp. 1121–1125 tion. Applied Mathematics and Computation (1997) 209(1), 57–67 (2009) [120] K¨ochel, P., Niel¨ander, U.: Simulation- [131] Lucidi, S., Sciandrone, M.: On the global based optimisation of multi-echelon inventory convergence of derivative-free methods for systems. International Journal of Production unconstrained minimization. SIAM Journal on Economics 93–94, 505–513 (2005) Optimization 13, 97–116 (2002) [121] Kolda, T.G., Lewis, R.M., Torczon, V.J.: Op- [132] Lutz, C.M., Davis, K.R., Sun, M.: Determining timization by direct search: New perspectives buffer location and size in production lines on some classical and modern methods. SIAM using tabu search. European Journal of Review 45, 385–482 (2003) Operational Research 106, 301–316 (1998) [122] Kothandaraman, G., Rotea, M.A.: [133] Mart´ı,R., Laguna, M., Glover, F.: Principles Simultaneous-perturbation-stochastic- of scatter search. European Journal of approximation algorithm for parachute Operational Research 169(2), 359–372 (2006) parameter estimation. Journal of Aircraft 42(5), 1229–1235 (2005) [134] Maryak, J.L., Chin, D.C.: Global random optimization by simulataneous perturbation [123] Kroese, D.P., Hui, K.P., Nariai, S.: Network stochastic approximation. IEEE Transactions reliability optimization via the cross-entropy on Automatic Control 53, 780–783 (2008) method. Reliability, IEEE Transactions on 56(2), 275–287 (2007) [135] Meketon, M.S.: Optimization in simulation: A survey of recent results. In: A. Thesen, [124] Kroese, D.P., Porotsky, S., Rubinstein, R.Y.: H. Grant, W.D. Kelton (eds.) Proceedings of The cross-entropy method for continuous the 1987 Winter Simulation Conference, pp. multi-extremal optimization. Methodology and 58–67 (1987) Computing in Applied Probability 8(3), 383– 407 (2006) [136] Merhof, D., Soza, G., Stadlbauer, A., Greiner, G., Nimsky, C.: Correction of susceptibility [125] Kulturel-Konak, S., Konak, A.: Simulation artifacts in diffusion tensor data using non- optimization embedded particle swarm opti- linear registration. Medical Image Analysis mization for reliable server assignment. In: 11(6), 588–603 (2007) B. Johansson, S. Jain, J. Montoya-Torres, J. Hugan, E. Y¨ucesan (eds.) Proceedings of the [137] Merton, R.C.: On the pricing of corporate 2010 Winter Simulation Conference, pp. 2897– debt: The risk structure of interest rates. The 2906 (2010) Journal of Finance 29(2), 449–470 (1974) Amaran, Sahinidis, Sharda and Bury 25

[138] Mishra, V., Bhatnagar, S., Hemachandra, N.: [149] Olafsson,´ S.: Metaheuristics. In: S. Henderson, Discrete parameter simulation optimization B. Nelson (eds.) Handbook in Operations algorithms with applications to admission Research and Management Science: Simulation control with dependent service times. In: 13, pp. 633–654. Elsevier (2006) Proceedings of the 46th IEEE Conference on [150] http://www.orms-today.org/surveys/ Decision and Control, pp. 2986–2991. New Simulation/Simulation.html Orleans, LA (2007) [151] Osorio, C., Bierlaire, M.: A simulation-based [139] Mockus, J.: Bayesian Approach to Global optimization approach to perform urban traffic Optimization. Kluwer Academic Publishers control. In: Proceedings of the Triennial (1989) Symposium on Transportation Analysis (2010) [140] Mockus, J., Tiesis, V., Zilinskas, A.: Towards Global Optimisation, vol. 2, chap. The [152] Pasupathy, R., Ghosh, S.: Simulation application of Bayesian Methods for Seeking optimization: A concise overview and im- the Extremum, pp. 117–128. North-Holland, plementation guide. Tutorials in operations Amsterdam (1978) research 10, 122–150 (2013) [141] Mor´e, J., Wild, S.: Benchmarking derivative- [153] Pasupathy, R., Henderson, S.G.: SIMOPT: free optimization algorithms. SIAM Journal on A library of simulation-optimization problems. Optimization 20, 172–191 (2009) In: S. Jain, R.R. Creasey, J. Himmelspach, K.P. White, e. M. Fu (eds.) Proceedings of the [142] Myers, R.H., Montgomery, D.C., Anderson- 2011 Winter Simulation Conference (2011) Cook, C.M.: Response Surface Methodology: Process and Product Optimization Using [154] Pasupathy, R., Kim, S.: The stochastic root Designed Experiments. Wiley Series in finding problem: Overview, solutions, and open Probability and Statistics. John Wiley & Sons questions. ACM Transactions on Modeling and (2009) Computer Simulation 21(3), 19:1–19:23 (2011) [143] Neddermeijer, H.G., Oortmarssen, G.J.V., [155] Peters, J., Vijayakumar, S., Schaal, S.: Piersma, N., Dekker, R.: A framework for Reinforcement learning for humanoid robotics. response surface methodology for simulation In: Third IEEE-RAS International Conference optimization. In: J.A. Joines, R.R. Barton, on Humanoid Robots, pp. 1–20. Karlsruhe, K. Kang, P.A. Fishwick (eds.) Proceedings of Germany (2003) the 2000 Winter Simulation Conference, pp. [156] Pflug, G.C.: Optimization of stochastic 129–136 (2000) models: The interface between simulation and [144] Nelder, J.A., Mead, R.: A simplex method for optimization. Kluwer Academic Publishers function minimization. Computer Journal 7, (1996) 308–313 (1965) [157] Powell, W.B.: http://www.castlelab. [145] Nelson, B.L.: Optimization via simulation princeton.edu/cso.htm (accessed October 23, over discrete decision variables. Tutorials in 2013) operations research 7, 193–207 (2010) [158] Powell, W.B., Ryzhov, I.O.: Optimal Learning. [146] Nelson, B.L., Goldsman, D.: Comparisons John Wiley & Sons (2012) with a standard in simulation experiments. Management Science 47(3), 449–463 (2001) [159] Prakash, P., Deng, G., Converse, M.C., Webster, J.G., Mahvi, D.M., Ferris, M.C.: [147] Nicolai, R., Dekker, R.: Automated response Design optimization of a robust sleeve antenna surface methodology for simulation optimiza- for hepatic microwave ablation. Physics in tion models with unknown variance. Quality Medicine and Biology 53, 1057–1069 (2008) Technology and Quantitative Management 6(3), 325–352 (2009) [160] Radac, M.B., Precup, R.E., Petriu, E.M., Preitl, S.: Application of ift and spsa to servo ¨ [148] O. Yal¸cinkaya, G.M.B.: Modelling and system control. IEEE Transactions on Neural optimization of average travel time for a Networks 22(12), 2363–2375 (2011) metro line by simulation and response surface methodology. European Journal of Operational Research 196, 225–233 (2009) Simulation optimization: A review of algorithms and applications 26

[161] Rall, L.B.: Automatic Differentiation: Tech- and machine learning. Springer-Verlag, New niques and Applications, Lecture Notes in York (2004) Computer Science, vol. 120. Springer, Berlin [172] Rubinstein, R.Y., Shapiro, A.: Discrete event (1981). DOI 10.1007/3-540-10861-0 systems: Sensitivity analysis and stochastic [162] Ramanathan, S.P., Mukherjee, S., Dahule, optimization by the score function method. R.K., Ghosh, S., Rahman, I., Tambe, S.S., John Wiley & Sons (1993) Ravetkar, D.D., Kulkarni, B.D.: Optimiza- [173] Sacks, J., Schiller, S.B., Welch, W.J.: Designs tion of continuous distillation columns using for computer experiments. Technometrics 31, stochastic optimization approaches. Transac- 41–47 (1989) tions of the Institution of Chemical Engineers 79, 310–322 (2001) [174] Safizadeh, M.H.: Optimization in simulation: [163] Rasmussen, C.E., Williams, C.K.I.: Gaussian Current issues and the future outlook. Naval Processes for Machine Learning. The MIT Research Logistics 37, 807–825 (1990) Press (2006) [175] Sahinidis, N.V.: Optimization under uncer- [164] Reeves, C.R.: Genetic algorithms for the tainty: State-of-the-art and opportunities. operations researcher. INFORMS Journal on Computers & Chemical Engineering 28(6–7), Computing 9(3), 231–250 (1997) 971–983 (2004) [165] Renotte, C., Vande Wouwer, A.: Stochastic [176] Schwartz, J.D., Wang, W., Rivera, D.E.: approximation techniques applied to parameter Simulation-based optimization of process con- estimation in a biological model. In: Intelligent trol policies for inventory management in Data Acquisition and Advanced Computing supply chains. Automatica 42, 1311–1320 Systems: Technology and Applications, 2003. (2006) Proceedings of the Second IEEE International [177] Scott, W., Frazier, P.I., Powell, W.: The Workshop on, pp. 261–265. IEEE (2003) correlated knowledge gradient for simulation [166] Rios, L.M., Sahinidis, N.V.: Derivative- optimization of continuous parameters using free optimization: A review of algorithms gaussian process regression. SIAM Journal on and comparison of software implementations. Optimization 21(3), 996–1026 (2011) 56 Journal of Global Optimization , 1247–1293 [178] Settles, B.: Active learning literature survey. (2013) Tech. rep., University of Wisconsin–Madison [167] Robbins, H., Monro, S.: A stochastic (2010) approximation method. The Annals of [179] Shapiro, A.: Asymptotic analysis of stochastic Mathematical Statistics 22(3), 400–407 (1951) programs. Annals of Operations Research 30, [168] Romero, P.A., Krause, A., Arnold, F.H.: 169–186 (1991) Navigating the protein fitness landscape with [180] Shapiro, A.: Simulation based optimization. gaussian processes. Proceedings of the National In: J.M. Charnes, D.J. Morrice, D.T. Brunner, Academy of Sciences (PNAS) 110(3) (2013). J.J. Swain (eds.) Proceedings of the 1996 DOI 10.1073/pnas.1215251110 Winter Simulation Conference, pp. 332–336 [169] Roustant, O., Ginsbourger, D., Deville, Y.: (1996) DiceKriging, DiceOptim: Two R packages ´ for the analysis of computer experiments by [181] Shi, L., Olafsson: Nested Partitions Op- kriging-based metamodeling and optimization. timization: Methodology and Applications, Journal of Statistical Software 51(1), 1–55 International Series in Operations Research & (2012) Management Science, vol. 109. Springer (2007) ´ [170] Rubinstein, R.: The cross-entropy method [182] Shi, L., Olafsson, S.: Nested partitions method for combinatorial and continuous optimization. for stochastic optimization. Methodology and Methodology and Computing in Applied Computing in Applied Probability 2, 271–291 Probability 1, 127–190 (1999) (2000) [171] Rubinstein, R.Y., Kroese, D.P.: The Cross- [183] Song, Y., Grizzle, J.W.: The extended Entropy Method: A unified approach to combi- kalman filter as a local asymptotic observer natorial optimization, Monte-Carlo simulation, for discrete-time nonlinear systems. Journal Amaran, Sahinidis, Sharda and Bury 27

of Mathematical Systems, Estimation, and [194] Tekin, E., Sabuncuoglu, I.: Simulation Control 5(1), 59–78 (1995) optimization: A comprehensive review on theory and applications. IIE Transactions 36, [184] Spall, J.C.: Multivariate stochastic approx- 1067–1081 (2004) imation using a simultaneous perturbation gradient approximation. IEEE Transactions on [195] Teng, S., Lee, L.H., Chew, E.P.: Multi- Automatic Control 37, 332–341 (1992) objective ordinal optimization for simulation optimization problems. Automatica 43(11), [185] Spall, J.C.: Chapter 7: Simultaneous 1884–1895 (2007) Perturbation Stochastic Approximation. In: Introduction to stochastic search and opti- [196] Trosset, M.W.: On the use of direct search mization: Estimation, simulation, and control. methods for stochastic optimization. Tech. Wiley-Interscience (2003) rep., Rice University, Houston, TX (2000) [186] Spall, J.C.: Introduction to stochastic search [197] Vande Wouwer, A., Renotte, ., Bogaerts, P., and optimization: Estimation, simulation, and Remy, M.: Application of spsa techniques in control. Wiley-Interscience (2003) nonlinear system identification. In: Proceed- ings of the European Control Conference, p. [187] Spall, J.C.: Feedback and weighting mecha- 2835 (2001) nisms for improving Jacobian estimates in the adaptive simultaneous perturbation algorithm. [198] Wang, H., Pasupathy, R., Schmeiser, B.W.: IEEE Transactions on Automatic Control Integer-ordered simulation optimization us- 54(6), 1216–1229 (2009) ing R-SPLINE: Retrospective search with piecewise-linear interpolation and neighbor- [188] Spall, J.C.: Stochastic optimization. In: J.E. hood enumeration. ACM Transactions on Gentle, W.K. H¨ardle, Y. Mori (eds.) Handbook Modeling and Computer Simulation 23, 17:1– of Computational Statistics: Concepts and 17:24 (2012) Methods, second edn., chap. 7, pp. 173–201. Springer (2012) [199] Wang, Q., Spall, J.C.: Discrete simultaneous perturbation stochastic approximation on loss [189] Srinivas, N., Krause, A., Kakade, S.M., Seeger, functions with noisy measurements. In: Pro- M.: Information-theoretic regret bounds for ceedings of the American Control Conference, gaussian process optimization in the bandit pp. 4520–4525. IEEE, San Francisco (2011) setting. IEEE Transactions on Information Theory 58(5), 3250–3265 (2012) [200] Whitley, D.: A genetic algorithm tutorial. Statistics and Computing 4, 65–85 (1994) [190] Stephens, C.P., Baritompa, W.: Global opti- mization requires global information. Journal [201] Xie, J., Frazier, P.I.: Sequential bayes-optimal of Optimization Theory and Applications 96, policies for multiple comparisons with a known 575–588 (1998) standard. Operations Research 61(5), 1174– 1189 (2013) [191] Swisher, J.R., Hyden, P.D., Jacobson, S.H., Schruben, L.W.: A survey of simulation [202] Xie, J., Frazier, P.I., Sankaran, S., Mars- optimization techniques and procedures. In: den, A., Elmohamed, S.: Optimization of J.A. Joines, R.R. Barton, K. Kang, P.A. computationally expensive simulations with Fishwick (eds.) Proceedings of the 2000 Winter gaussian processes and parameter uncertainty: Simulation Conference (2000) Application to cardiovascular surgery. In: 50th Annual Allerton Conference on Communica- [192] Syberfeldt, A., Lidberg, S.: Real-world tion, Control, and Computing (2012) simulation-based manufacturing optimization using cuckoo search. In: C. Laroque, [203] Xing, X.Q., Damodaran, M.: Assessment of J. Himmelspach, R. Pasupathy, O. Rose, simultaneous perturbation stochastic approxi- A. Uhrmacher (eds.) Proceedings of the 2012 mation method for wing design optimization. Winter Simulation Conference (2012) Journal of Aircraft 39, 379–381 (2002) [193] Tein, L.H., Ramli, R.: Recent advancements [204] Xing, X.Q., Damodaran, M.: Application of of nurse scheduling models and a potential simultaneous perturbation stochastic approxi- path. In: Proceedings of the 6th IMT-GT mation method for aerodynamic shape design Conference on Mathematics, Statistics and its optimization. AIAA Journal 43(2), 284–294 Applications, pp. 395–409 (2010) (2005) Simulation optimization: A review of algorithms and applications 28

[205] Xing, X.Q., Damodaran, M.: Inverse design of transonic airfoils using parallel simultaneous perturbation stochastic approximation. Jour- nal of Aircraft 42(2), 568–570 (2005) [206] Xu, J., Nelson, B.L., Hong, L.J.: Indus- trial strength COMPASS: A comprehensive algorithm and software for optimization via simulation. ACM Transactions on Modeling and Computer Simulation 20(1), 1–29 (2010) [207] Xu, J., Nelson, B.L., Hong, L.J.: An adaptive hypberbox algorithm for high-dimensional discrete optimization via simulation problems. INFORMS Journal on Computing 25(1), 133– 146 (2013) [208] Yan, D., Mukai, H.: Stochastic discrete optimization. SIAM Journal on Control and Optimization 30, 594–612 (1992) [209] Yang, X.S., Deb, S.: Engineering optimisation by cuckoo search. International Journal of Mathematical Modelling and Numerical Optimisation 1(4), 330–343 (2010) [210] Yeomans, J.S.: Solid waste planning under uncertainty using evolutionary simulation- optimization. Socio-Economic Planning Sci- ences 41, 38–60 (2007) [211] Yun, I., Park, B.: Application of stochastic optimization method for an urban corridor. In: L.F. Perrone, F.P. Wieland, J. Liu, B.G. Lawson, D.M. Nicol, R.M. Fujimoto (eds.) Proceedings of the 2010 Winter Simulation Conference, pp. 1493–1499 (2010)