MO2TOS: Multi-fidelity Optimization with Ordinal Transformation and Optimal Sampling Jie Xu, Si Zhang, Edward Huang, Chun-Hung Chen Department of System Engineering and Operations Research George Mason University, Fairfax, VA 22030 Loo Hay Lee Department of Industrial and Systems Engineering The National University of Singapore, Kent Ridge 119260, Singapore Nurcin Celik Department of Industrial Engineering The University of Miami, Coral Gables, FL 33146, USA Abstract Simulation optimization can be used to solve many complex optimization problems in au- tomation applications such as job scheduling and inventory control. We propose a new frame- work to perform efficient simulation optimization when simulation models with different fidelity levels are available. The framework consists of two novel methodologies: ordinal transformation (OT) and optimal sampling (OS). The OT methodology uses the low-fidelity simulations to transform the original solution space into an ordinal space that encapsulates useful information from the low-fidelity model. The OS methodology efficiently uses high-fidelity simulations to sample the transformed space in search of the optimal solution. Through theoretical analysis and numerical experiments, we demonstrate the promising performance of the Multi-fidelity Optimization with Ordinal Transformation and Optimal Sampling (MO2TOS) framework. 1 1 Introduction Scientists and engineers have long been using analytical and computational models to approximately describe and predict the behaviors of physical processes/systems and use these models to guide the design of engineering systems. Optimization methods can often be applied to these models to help identify designs that are not only viable but potentially achieve the best performance possible. Prototypes are then built to evaluate the actual performance of a few most promising designs (according to the model) and select the best one. In the past decades, the explosive growth of computing technology has revolutionized such long- standing engineering design practices. While still an indispensable process in engineering design, physical prototyping is giving ways to high-fidelity computer simulation models (in this paper, we use the term \simulation" to refer to any computational model that can be used to directly eval- uate the objective value of a candidate design, possibly with bias and/or noise), which can quite accurately evaluate the actual performance of a candidate design using a small fraction of time and resources that physical prototyping would require. Because of the accuracy of high-fidelity simulation models and their flexibility in modeling and analyzing complex systems that are intractable to traditional analytical methods, it is desirable to perform optimization directly with these simulation models. Simulation-based optimization, or simply simulation optimization (Chen and Lee 2011, Lee et al. 2010, Xu et al. 2010, 2014), refers to a class of methods that search for optimal or near optimal solutions by using simulation models to directly evaluate the objective values of candidate solutions. However, high-fidelity simulation models often have high computation cost and can be very time- consuming to run. As a result, when the time and the computing resources to perform optimization is limited, only a small number of candidate solutions can be evaluated via high-fidelity simulations in the simulation optimization process. Finding optimal or near optimal solutions using high- fidelity simulations may not be practical in many applications (Celik 2010). In contrast, models of lower-fidelity (e.g., analytical approximations, coarser-grained simulations) are much faster and can evaluate a large number of candidate solutions in a short amount of time. As a result, current industrial practice uses low-fidelity models to screen a large number of candidate solutions and select a few \top" solutions for evaluation by high-fidelity models. Such 2 a methodology has its intuitive appeal, is easy of use, and sometimes can achieve quite successful outcomes. However, it is heuristic in nature and may not be robust because low-fidelity model can be fraught with significant bias and variability. This paper offers a systematic, efficient, and robust procedure to exploit the benefits of both high- and low-fidelity models in optimization. We pro- pose a Multi-fidelity Optimization with Ordinal Transformation & Optimal Sampling (MO2TOS) framework and demonstrate its benefits through rigorous analysis and numerical experiments. While low-fidelity models may suffer from significant biases and variabilities, they often provide useful information on candidate solutions such as their relative quality. This is because low-fidelity models are often built on reasonable abstractions and simplifications of the underlying physical processes. The MO2TOS framework exploits this property and proposes an Ordinal Transformation (OT) approach to transform the original decision space into a new one-dimensional space where all solutions are positioned according to their ordinal ranks using the low-fidelity model. To see the benefit of transforming the solution space into a one-dimensional ordinal space, it is important to remember that the original solution space can be high-dimensional, have multiple locally optimal solutions spread far apart, and include a mix of integer-valued and categorical variables. After OT, the new solution space is one-dimensional, likely to be well-behaved and display some trend, and greatly simplifies further optimization using just a small number of high-fidelity simulations. We then propose an optimal sampling (OS) method to search the transformed space for a (near) optimal solution. OS adaptively samples the transformed space taking into account the quality of the low-fidelity model and thus achieves both efficiency and robustness. There has been related work on the optimization of complex systems with multi-fidelity simula- tions. The Multi-Fidelity Sequential Kriging Optimization (MFSKO) procedure constructs kriging models to approximate the difference in simulation output between models of consecutive fidelity levels (Huang et al. 2006). MFSKO then sequentially determines the next solution to simulate and the level of fidelity for that simulation with an objective to maximize the expected improvement in the quality of the best solution found so far. The Value-based Global Optimization (VGO) proce- dure (Moore 2012) aims to maximize the value of a sampling decision and also uses a kriging model to predict the performance of a new solution. Compared to MFSKO, there are also some significant technical differences in how kriging models are used in determining the next solution to simulate and the fidelity level to use. In March and Willcox (2012), the authors used a radial basis func- 3 tion (RBF) interpolation to create a surrogate model of the high-fidelity simulation model in the neighborhood of a trust region. The RBF surrogate model is then used to predict the high-fidelity simulation results for new solutions and select the next solution for high-fidelity simulation. All these earlier methods basically create a surrogate model using an interpolation method (kriging or RBF) to correct the bias of the low-fidelity model and perform the optimization using the \corrected" low-fidelity model. While they have been shown to work reasonably well on a number of engineering design problems, the performance of these methods depends critically on the quality and applicability of the interpolation method. It is well known that interpolation methods such as kriging and RBF would require a large number of design points to perform well when the underlying response surface is highly nonlinear and multi-modal, and/or the dimension of the solution space is high. In many complex system design problems, there are a mix of discrete and categorical decision variables. This would present additional challenges to these interpolation methods. In comparison, MO2TOS is a very general and flexible framework and has the following im- portant advantages: 1) MO2TOS handles a mix of discrete and categorical decision variables in a high-dimensional solution space; 2) MO2TOS is a general framework and is not restricted to any specific interpolation technique such as kriging; and 3) MO2TOS has the potential to provide both efficiency and robustness through the new OT and OS methodologies, and can effectively perform optimization even when low-fidelity models may not be good. The rest of the paper is organized as follows. In Section 2, we illustrate the principles and benefits of MO2TOS using a machine allocation example in the context of flexible manufacturing systems. In Section 3, we present a mathematical model and analysis to show the benefits of OT and propose an OS strategy. We also present a simulation optimization algorithm under the general MO2TOS framework in Section 3. Section 4 presents numerical results. We conclude the paper and point out future research directions in Section 5. 2 A MOTIVATIONAL CASE STUDY In this paper, we assume that there is one high-fidelity model that generate accurate predictions of the system and represents the \ground truth". We also have a low-fidelity model whose results 4 provide approximations to the high-fidelity model with unknown bias. We now use a flexible manufacturing system as an example to illustrate the basic principles and potential benefits of MO2TOS. While this illustrative example is relatively simple, the ideas are equally applicable to more general problems. There are two types of