Pdfs) Is Discussed

Combustion and Flame 183 (2017) 88–101

Contents lists available at ScienceDirect

Combustion and Flame

journal homepage: www.elsevier.com/locate/combustflame

A general probabilistic approach for the quantitative assessment of LES combustion models

∗ Ross Johnson, Hao Wu, Matthias Ihme

Department of Mechanical Engineering, Stanford University, Stanford, CA 94305, United States

a r t i c l e i n f o a b s t r a c t

Article history: The Wasserstein metric is introduced as a probabilistic method to enable quantitative evaluations of LES Received 9 March 2017 combustion models. The Wasserstein metric can directly be evaluated from scatter data or statistical re-

Revised 5 April 2017 sults through probabilistic reconstruction. The method is derived and generalized for turbulent reacting Accepted 3 May 2017 flows, and applied to validation tests involving a piloted turbulent jet flame with inhomogeneous inlets. It Available online 23 May 2017 is shown that the Wasserstein metric is an effective validation tool that extends to multiple scalar quan- Keywords: tities, providing an objective and quantitative evaluation of model deficiencies and boundary conditions Wasserstein metric on the simulation accuracy. Several test cases are considered, beginning with a comparison of mixture- Model validation fraction results, and the subsequent extension to reactive scalars, including temperature and species mass Statistical analysis fractions of CO and CO 2 . To demonstrate the versatility of the proposed method in application to multi-

Quantitative model comparison ple datasets, the Wasserstein metric is applied to a series of different simulations that were contributed Large-eddy simulation to the TNF-workshop. Analysis of the results allows to identify competing contributions to model deviations, arising from uncertainties in the boundary conditions and model deﬁciencies. These applications demonstrate that the Wasserstein metric constitutes an easily applicable mathematical tool that reduce multiscalar combustion data and large datasets into a scalar-valued quantitative measure. ©2017 The Combustion Institute. Published by Elsevier Inc. All rights reserved.

1. Introduction sults from Favre and Reynolds averaging, conditional data, probability density functions, and scatter data. These data provide im- A main challenge in the development of models for turbu- portant information for model evaluation. lent reacting flows is the objective and quantitative evaluation of Significant progress has been made in simulating turbulent the agreement between experiments and simulations. This chal- flows. This progress can largely be attributed to adopting high- lenge arises from the complexity of the flow-field data, involv- fidelity large-eddy simulation (LES) for the prediction of unsteady ing different thermo-chemical and hydrodynamic quantities that turbulent flows, and establishing forums for collaborative compar- are typically provided from measurements of temperature, speci- isons of benchmark flames that are supported by comprehensive ation, and velocity. This data is obtained from various diagnos- databases [7] . tics techniques, including nonintrusive methods such as laser spec- In spite of the increasing popularity of LES, the valida- troscopy and particle image velocimetry, or intrusive techniques tion of combustion models follows previous steady-state RANS- such as exhaust-gas sampling or thermocouple measurements [1– approaches, comparing statistical moments (typically mean and 4] . Instead of directly measuring physical quantities, they are typ- root-mean-square) and conditional data along axial and radial lo- ically inferred from measured signals, introducing several correc- cations in the flame. Qualitative comparisons of scatter data are tion factors and uncertainties [5] . Depending on the experimen- commonly employed to examine whether a particular combustion tal technique, these measurements are generated from single-point model is able to represent certain combustion-physical processes data, line measurements, line-of-sight absorption, or multidimen- in composition space that, for instance, are associated with extinc- sional imaging at acquisition rates ranging from single-shot to tion, equilibrium composition, or mixing conditions. In other in- high-repetition rate measurements to resolve turbulent dynamics vestigations, error measures were constructed by weighting mo- [6] . This data is commonly processed in the form of statistical re- ments and scalar quantities for evaluating the sensitivity of model coefficients in subgrid models and for uncertainty quantification [8,9] . It is not uncommon that models match particular measure- ∗ Corresponding author. ment quantities (such as temperature or major species products) E-mail addresses: [email protected] (R. Johnson), [email protected] (H. in certain regions of the flame, while mispredicting the same Wu), [email protected] (M. Ihme). http://dx.doi.org/10.1016/j.combustflame.2017.05.004 0010-2180/© 2017 The Combustion Institute. Published by Elsevier Inc. All rights reserved. R. Johnson et al. / Combustion and Flame 183 (2017) 88–101 89 quantities in other regions or showing disagreements for other Since multivariate distributions are commonly used to represent flow-field data at the same location. Further, the comparison of in- the thermo-chemical states in turbulent flows, this method is use- dividual scalar quantities makes it difficult to consider dependen- ful for the quantitative assessment of differences between a nu- cies and identify correlations between flow-field quantities. Faced merical simulation and experimental data of turbulent flames. with this dilemma, such comparisons often only provide an incon- The dissimilarity between two points in the thermo-chemical clusive assessment of the model performance, and limit a quantita- state space can typically be quantified by its Euclidean distance. tive comparison among different modeling approaches. Therefore, a In the case of a 1D state space, e.g., temperature, the Euclidean need arises to develop a metric that enables a quantitative assess- distance is simply the absolute value of the difference. This met- ment of combustion models, fulfilling the following requirements: ric serves the purpose well for the comparison among two de- terministic measurements, which can indeed be fully represented 1. provide a single metric for quantitative model evaluation; by points in the state space. However, the Euclidean distance falls 2. combine single and multiple scalar quantities in the validation short when measurements are made on random data, e.g., states in metric, including temperature, mixture fraction, species mass a turbulent flow, in which case a single data point can no longer fractions, and velocity; represent the measurement and shall be replaced by a distribution 3. incorporate scatter data from single-shot, high-speed, and si- of possible outcomes. Consequently, the dissimilarity between two multaneous measurements, and enable the utilization of statis- random variables shall be quantified by the “distance” between the tical data; corresponding probability distributions. 4. enable the consideration of dependencies between measure- Among many definitions of the “distance” between distributions ment quantities; and [11,12] , the Wasserstein metric (also known as the Mallows met- 5. ensure that the metric satisfies conditions on non-negativity, ric) is of interest in this study. Several other methods for quan- identity, symmetry, and triangular inequality. tifying the difference between two distributions have been pro- By addressing these requirements, the objective of this work is posed. However, those methods are either limited to evaluating the to introduce the probabilistic Wasserstein distance [10] as a met- equality of marginal distributions (e.g., Kolmogorov–Smirnov test) ric for the quantitative evaluation of combustion models. Com- or are incomplete in that they fail to satisfy essential metric prop- plementing currently employed comparison techniques, this met- erties (e.g., Kullback–Leibler divergence). The interested reader is ric possesses the following attractive properties. First, this metric referred to the survey by Gibbs and Su [11] for a detailed com- directly utilizes the abundance of data from unsteady simulation parison between the Wasserstein metric and other candidates. The techniques and high repetition rate measurement methods. Rather Wasserstein metric represents a natural extension of the Euclidean than considering low-order statistical moments, this metric is for- distance, which can be recovered as the distribution reduces to a mulated in distribution space. As such, it is thereby directly ap- Dirac delta function. As such, this metric provides a measure of the plicable to scatter data that are obtained from transient simula- difference between distributions that resembles the classical idea tions and high-speed measurements without the need for data re- of “distance”. The resulting value can be judged and interpreted in duction to low-order statistical moment information. Second, this a similar fashion as the arithmetic difference between two deter- metric is able to synthesize multidimensional data into a scalar- ministic quantities. valued quantity, thereby aggregating model discrepancies for indi- The Wasserstein metric has been used as a tool for measur- vidual quantities. Third, the resulting metric utilizes a normaliza- ing the “distance” between distributions or histograms in the con- tion, thereby enabling the objective comparison of different sim- text of content-based image retrieval [12] , hand-gesture recogni- ulation approaches. Fourth, this method is directly applicable to tion [13] , and analysis of 3D surfaces [14] , etc. These applications sample data that are generated from scatter plots, instantaneous were first proposed by Rubner et al. [12] , under the name of the simulation results or reconstructed from statistical results. Fifth, Earth Mover’s Distance (EMD), which is in fact the Wasserstein the Wasserstein metric enables the consideration of conditional metric of order one, W1 . More recent applications favor the 2nd and multiscalar data, and is equipped with essential properties of Wasserstein metric by way of Brenier’s theorem [14,15] , which metric spaces. Finally, this metric can be regarded as a companion shows the uniqueness of the optimal solution for the squared- tool to previously established methods for validation of LES and distance and its connection to differential geometry through which instantaneous CFD-simulations. more efficient algorithms have been constructed. The remainder of this paper is structured as follows. This section is primarily concerned with the Wasserstein metric Section 2 formally introduces the Wasserstein metric and builds a for discrete distributions in the Euclidean space. This allows us to mathematical foundation for this method. This method is demon- focus on the concepts that are directly related to the practical cal- strated in application to the Sydney piloted jet flame with inho- culation and estimation of this metric, and avoid the usage of mea- mogeneous inlets, and experimental configuration and simulation sure theory without creating much ambiguity. The definition of the setup are described in Section 3 . Results are presented in Section 4 . Wasserstein metric for general probability measures with more for- To introduce the metric, we first consider a scalar distribution of mal treatment of the probability theory is discussed in Appendix A . mixture fraction and establish a quantitative comparison of ex- The interested reader is also referred to books by Villani [16] , sur- periments and simulations against presumed closure models. This veys by Urbas [17] , and the more recent lectures by McCann and is then extended to multiscalar data, involving temperature and Guillen [18] for further details. species mass fractions. Subsequently, we examine the utility of applying the Wasserstein metric to statistical results, rather than 2.1. Preliminaries: Monge–Kantorovich transportation problem scatter data, while Section 5 explores how the Wasserstein metric could add value for directly comparing multiple simulations The Wasserstein metric is motivated by the classical optimal to assess different LES modeling approaches. This work finishes in transportation problem first proposed by Monge in 1781 [19] . The Section 6 with conclusions. optimal transportation problem of Monge considers the most efficient transportation of ore from n mines to n factories, each of 2. Methodology which produces and consumes one unit of ore, respectively. These mines and factories form two finite point sets in the Euclidean In this section, the methodology of quantifying the dissimilarity space, which are denoted by M and F. The cost of transporting ∈ M ∈ F between two multivariate distributions (joint PDFs) is discussed. one unit of ore from mine xi to factory y j is denoted by 90 R. Johnson et al. / Combustion and Flame 183 (2017) 88–101

cij , which is chosen by Monge to be the Euclidean distance. The The pth Wasserstein metric between discrete distributions f and transport plan can be expressed in the form of an n × n matrix g is defined to be the p th root of the optimal transport cost for the γ , of which the element ij represent the amount of ore trans- corresponding Monge–Kantorovich problem: ported from xi to yj . The central problem in optimal transporta- / n n 1 p tion is to find the transport plan that minimizes the total cost of p W ( f, g) = min γ | x − y | transportation, which is the sum of the costs on n × n available p ij i j = = n n γ i 1 j 1 routes between factories and mines, i.e., = = ij cij . The orig- i 1 j 1 (5) inal problem formulated by Monge requires transporting the ore n n γ = , γ = , γ ≥ . in its entirety. Therefore, is constrained to be a permutation ma- subject to ij fi ij g j i, j 0 trix, and can only take binary values of 0 or 1. Formally, the Monge j=1 i =1 transport problem can be expressed as The constraints in Eq. (5) ensure that the total mass transported n n from xi and the total mass transported to yj match fi and gj , re- min γ c ij ij spectively. i =1 j=1 The Wasserstein metric for continuous distributions is pre- (1) n n sented in Appendix A . The estimation and calculation for the con- γ = γ = , γ ∈ { , } . subject to ij ij 1 ij 0 1 tinuous case and for higher-dimensional problems are discussed in i =1 i = j the next section. The transport problem in Monge’s formulation can be solved relatively efficient by the Hungarian algorithm proposed by Kuhn 2.3. Non-parametric estimation of W through empirical distributions [20] . p In 1942, Kantorovich reformulated Monge’s problem by relaxing Two major difficulties arise in obtaining the Wasserstein metric the requirement that all the ore from a given mine goes to a single of two multivariate distributions of thermo-chemical states. One is factory [21,22] . As a result, the transport plan is modified from be- that the multivariate distributions are not readily available from ei- ing a permutation matrix to a doubly stochastic matrix. With this, ther experiments or simulations. Instead, a series of samples drawn Monge’s problem is written in the following form: from these distributions is provided. The other is that there is no

n n easy way of calculating W p for continuous multivariate distribu- min γ c ij ij tions, especially for those of high dimensions. To overcome these i =1 j=1 problems, a non-parametric estimation of W is devised using em- (2) p n n pirical distributions. subject to γij = γij = 1 , γij ≥ 0 . An empirical distribution is a random discrete distribution i =1 i = j formed by a sequence of independent samples drawn from a given distribution of interest. Let (X , . . . , X ) b e a set of n independent This relaxation eliminates the difficulties of Monge’s formulation 1 n random samples obtained from a continuous multivariate distribu- in terms of obtaining certain desirable mathematical properties, tion f . The empirical (or fine-grained [24] ) distribution f is defined such as existence and uniqueness of the optimal transport plan. to be The Monge–Kantorovich transportation problem in Eq. (2) can be n solved as the solution to a general linear programming prob- 1 ( ) = δ( − X ) , lem, although more efficient algorithms exists by exploiting special fn x x i (6) n structures of the problem. i =1 which is a discrete distribution with equal weights. The empir- 2.2. Wasserstein metric for discrete distributions ical distribution is random as it depends on random samples, X i . The samples may be obtained from experimental measure- The Monge–Kantorovich problem in Eq. (2) not only gives the ments or generated from a given distribution using, for instance, solution to the optimal transport problem, but also provides a way an acceptance-rejection method [25] . Calculation of W p between of quantifying the dissimilarity between the distributions of mines two empirical distributions is identical to the method discussed in and factories by evaluating the optimal transport cost (normalized Section 2.2 . Its procedure and cost is independent of the dimen- by total mass). The Wasserstein metric [23] follows directly from sionality of the distributions. this observation. The empirical distribution converges to the original distribution. Consider replacing the mines and factories by two general dis- Most importantly, in the context of this study, is the convergence crete distributions, whose probability mass functions are: in the Wasserstein metric [26] . More specifically, the Wasserstein

metric between empirical and original distributions converges to n n

( ) = δ( − ) , ( ) = δ( − ) , zero in probability. Given the metric property of Wp: f x fi x xi g y g j y yi (3) i =1 j=1 W p ( f n , g n ) − W p ( f, g) ≤ W p ( f n , f ) + W p ( g n , g) , (7) where n f = n g = 1 and δ( · ) denotes the Kronecker delta i =1 i j=1 i which ensures that the Wasserstein metric between two empirical function: distributions also converges in probability to that of the actual dis- 1 if x = 0 , tribution. In addition, the mean rates of convergence for empirical δ( ) = x (4) distributions have also been established [27,28] , giving the upper 0 if otherwise . p( , ) , bound on the expectation, E Wp fn f as n increases. For the

The unit cost of transport between xi and yj is deﬁned to be the similar reason, the convergence rate for the non-parametric esti- = | − | p pth power of the Euclidean distance, i.e. cij xi y j . In addition, mation follows. The exact rates depend on the dimensionality and the “mass” of probabilities fi and gj is no longer restricted to be regularity conditions of the distributions. For details on the conver- unity and the possible outcomes of the distributions, i.e. xi and yj , gence rate for more general cases, the interested reader is referred are points in an Euclidean space whose dimension is not restricted. to the work by Fournier and Guillin [28] . R. Johnson et al. / Combustion and Flame 183 (2017) 88–101 91

In the case of d -dimensional distributions that have sufficiently normal distributions are uncorrelated. More specifically, if many moments, we have: g(x ) ≡ 1 , ⎧ −1 / 2 (11) ⎨ n if d < 4 , ( ) = , , . . . , , 2, 2, . . . , 2 , T x [x1 x2 xd x1 x2 xd ] 2 −1 / 2 E W ( f n , f ) ≤ C × n log (1 + n ) if d = 4 , (8) 2 ⎩ the SMLD becomes −2 /d > , n if d 4 d ( − μ ) 2 1 xk k ( ) = − , fSML x exp (12) where the value of C depends on the distribution f and is indepen- πσ2 σ2 = 2 2 k dent of n . By combining this result with Eq. (7) , we obtain the fol- k 1 k 2 lowing convergence rate for the non-parametric estimation of W2 : where μ = x and σ2 = x 2 − x . k k k k k After obtaining the SMLDs, the sets of samples can be drawn ⎧ from them and the Wasserstein metric can be directly calculated. − / ⎪ 1 2 ⎨ n ∗ if d < 4 , In addition, the metric property implies the following inequality 2 − / ( , ) − ( , ) ≤ × 1 2 E W2 fn gn W2 f g C n ∗ log (1 + n ∗ ) if d = 4 , | ( , ) − ( , ) | ≤ ( , ) + ( , ) . ⎪ Wp fSML gSML Wp f g Wp fSML f Wp gSML g (13) ⎩ − / 2 d > , n∗ if d 4 In other words, the difference in the Wasserstein metrics of the (9) actual distributions, f and g , and that of the reconstructed coun- , terparts, fSML and gSML are bounded by the error of the SMLD re- = ( , ) where n ∗ min n n . With this, all necessary results that ensure construction, which themselves can be quantified by the Wasser- ( , ) the soundness of using Wp fn gn as an estimator for Wp (f, g) are stein metric. In the current study, the set of constraints are lim- presented. ited to the marginal first and second moments, which are typically In addition to the rate of convergence, statistical inference and reported in the literature. More accurate reconstructions can be further quantification of uncertainties for the non-parametric esti- obtained by including high-order and cross moments. In addition, mation of W p can also be performed. For instance, the magnitude statistics of other forms may also be considered. For instance, β- of the uncertainty in Eq. (9) and the corresponding confidence in- distributions can be recovered for constraints in the form of ln (x ) terval can be estimated via the method of bootstrap [29,30] . In and ln (1 − x ), which is potentially more appropriate for conserved particular, it was shown that the m -out- n bootstrap [31] can be scalars such as mixture fraction. applied to the Wasserstein metric [32,33] . 2.5. Calculation procedure 2.4. Statistically most likely reconstruction of distributions In the present study, the 2nd Wasserstein metric is used. The computation of the metric can be realized by any general-purpose So far, the evaluation of the Wasserstein metric using sample linear programming tool. For the present analysis, the program by data as empirical distribution function has been described. How- Pele and Werman [37] is used. It calculates the Wasserstein metric ever, the usage of the Wasserstein metric is not limited to results as a flow-min-cost problem (a special case of linear programming) reported in such fashion. In the following, the computation of the using the successive shortest path algorithm [38] . A pseudo code Wasserstein metric from statistical results will be discussed. This of the corresponding algorithm is given in Algorithm 1 , and Fig. 1 versatility is important to the metric, given the fact that the con- provides an illustrative example for the evaluation of the Wasser- ventional practice of reporting only the statistics, predominantly stein metric. The source code for the evaluation of the Wasserstein first and second moments, is still the prevailing one. metric is provided in Appendix B . The procedure of applying the Wasserstein metric to statistical results is to first reconstruct the multivariate distribution from statistical results. An empirical distribution is then sampled to com- Algorithm 1: Pseudo code for evaluating the Wasserstein met- pute the Wasserstein metric following the method described in ric. Section 2.5 . The reconstruction of a continuous PDF from a set of Input : Two sets of d-dimensional data representing empirical known statistical models can be performed using the concept of distribution functions: X , Y, with lengths n and n the statistically most likely distribution (SMLD) [34,35] . The SMLD from scatter data or sampled from continuous of a d -dimensional random variable is defined to be the distribu- distribution functions tion that maximizes the relative entropy, given a prior distribution Preprocessing : Normalize X and Y by standard deviation of g(x), under a given set of constraints: one data set, σx for i = 1 : n do ( ) f x f (x ) = argmax f (x ) ln dx for j = 1 : n do SML ( ) f (·) R d g x Evaluate pair-wise distance matrix d 2 c , = = X , − Y , ; subject to f (x ) dx = 1 , i j k 1 k i k j R d end end ( ) ( ) = . T x f x dx t (10) R d Compute Wasserstein metric and transport matrix as solution to minimization problem of Eq. (5) using the shortest path Here, T ( x ) is the set of statistical moments that are selected as algorithm by Pele & Werman [37] with input ci, j constraints, and t is the vector of corresponding values obtained Output : Wasserstein metric: W 2 from the data. The type of the so obtained distribution is dictated by the form of the constraints [36] . For instance, if the multivariate distribution is constructed using only the first and second mo- In this context, it is important to note that the input data to ments with a uniform prior, a multivariate normal distribution is the Wasserstein metric are normalized to enable a direct compar- obtained. In addition, if only the marginal second moments are ison and enable a physical interpretation of the results. A natural given while the cross moments are not, the obtained multivariate choice is to normalize each sample-space variable by its respective 92 R. Johnson et al. / Combustion and Flame 183 (2017) 88–101

Fig. 2. Schematic of the piloted turbulent burner with inhomogeneous inlets.

Fig. 1. Illustrative example of computing the Wasserstein matrix from two distri- relevant information, such as space-time correlations, can be fac- bution functions: (a) The continuous PDFs are first sampled by empirical distribu- tored in via the extension of the phase space for the PDFs as per- tion functions (EDF, shown by discrete peaks, which are here down-sampled for formed by Muskulus and Verduyn–Lunel [39] . illustrated purposes). The Wasserstein metric is then evaluated from the samples Comparable samples need to be drawn from numerical simu- X Y i =1 , ... , 4 and i =1 , ... , 5 with (b) coefficients of transport matrix , obtained from the lations and experimental data to ensure a consistent comparison solution to the minimization problem of Eq. (5) . between the two distributions. This can be achieved by matching the sampling locations and frequencies between the two sources of standard deviation that is computed from the reference data set data. Furthermore, experimental uncertainties can also be consid- (for instance, the experimental measurements). ered by either convolving the simulation data with the error dis- Suppose we have two sets of data with sample sizes of n and tribution or through Bayes deconvolution of the experimental data n , respectively. Each sample represents a point in the thermo- [40,41] . chemical (sub)-space, e.g., x = [ Z, T , Y , . . .] . The empirical distri- H2 O bution can be constructed from each data set following Eq. (6) , 3. Configuration = / = / where fi 1 n and gi 1 n . The Wasserstein metric is then computed following the definition in Eq. (5) . The worst time complex- In the present study, the utility of the Wasserstein metric is ity of the algorithm is O (n + n ) 3 log (n + n ) . Note that the di- evaluated in an application to simulation results of a piloted tur- mension of the thermo-chemical (sub)-space affects only the pair- bulent jet flame with inhomogeneous inlet. The experimental con- wise distance between data points but not the definition or calcu- figuration is described in the next section, the computational setup lation of the metric. is summarized in Section 3.2 , and statistical simulation results that are utilized in evaluating Wasserstein metric are presented in 2.6. Remarks on the interpretation and usage of the Wasserstein Section 3.3 . metric W p 3.1. Experimental setup The Wasserstein metric is a natural extension of the Euclidean distance to statistical distributions. It enables the comparison be- The piloted turbulent jet flame with inhomogeneous inlets con- tween two multi-dimensional distributions via a single metric sidered in this work was experimentally investigated by Meares while taking all information presented by the distributions into et al. [42,43] and Barlow et al. [44] . The burner is schematically consideration. The definition of W p in Eq. (5) can be viewed as shown in Fig. 2 . The reactants are supplied through an injector that the weighted average of the pair-wise distances between samples consists of three tubes; the inner fuel-supply tube with a diame- of two distributions. In the case of one-dimensional distributions, ter of 4 mm is recessed by a length L r with respect to the burner the obtained value of the metric shares the same unit as the sam- exit plane. This inner tube is placed inside an outer tube (with a ple data. For instance, if two distributions of temperature are con- diameter of D J = 7 . 5 mm) of ambient air; depending on the recess sidered, the corresponding Wasserstein metric in units of Kelvin height, varying levels of mixture inhomogeneity can be achieved. can be interpreted as the average difference between the values The flame is stabilized by a pilot stream, exiting through an annu- of temperature from the two distributions. In the case of multi- lus with outer diameter of 18 mm. The burner is placed inside a dimensional distributions, each dimension is normalized before wind tunnel, providing co-flowing air at a bulk velocity of 15 m/s. pair-wise distances are calculated. The choice of the normalization The present study considers the operating condition FJ-5GP- method is application-specific. In the present study, the marginal Lr75-57 . The fuel of methane is supplied through the inner tube standard deviation is chosen. The so obtained W p represents the ( FJ ). The pilot flame is a gas mixture consisting of five compo- averaged difference, which is proportional to the marginal standard nents (5GP: C2 H2 , H2 , CO2 , N2 , and air to match the product deviations, between samples from the two distributions. As such, a gas composition and equilibrium temperature of the stoichiomet- = . value of Wp 0 5 can be interpreted as a difference between sim- ric CH4 /air mixture). The bulk velocity of the unburned pilot mix- ulation and experimental data at the level of 0.5 standard devia- ture is 3.72 m/s. The inner tube of the fuel stream is recessed from tion. Although not considered in this study, additional turbulent- the jet exit plane by L r = 75 mm, and the bulk jet velocity is set to R. Johnson et al. / Combustion and Flame 183 (2017) 88–101 93

57 m/s ( Lr75-57 ), corresponding to 50% of the experimentally mea- simulated data presented in this section is not expected to repli- sured blow-off velocity. The recess results in a partially premixed cate the experimental results perfectly. Instead, the data serves the reactant-gas mixture, which is relevant to modern gas-turbine ap- purpose to determine where in the flame and for which species plications [45,46] . the model behaves well, as well as regions in which the model could be improved. Quantitative results using the Wasserstein val- 3.2. Computational setup and mathematical model idation metric will be presented in Section 4 , and these results should match the data interpretation developed in this section. The computational domain is discretized using a three- Comparisons between radial profiles of experimental data and dimensional structured mesh in cylindrical coordinates, and in- simulations are provided in Fig. 3 . The comparisons are made at cludes the upstream portion of the burner to represent the par- four distinct axial locations ( x/D J = { 1 , 5 , 10 , 15 } ), with four scalar tial mixing of reactants and flame stabilization. The computational quantities, which include mixture fraction ( Z ), temperature ( T ), and domain extends to 20DJ in axial direction and 15DJ in radial di- species mass fractions of CO2 and CO. The solid lines represent rection, and is discretized using 1.6 million control volumes. In- simulation results, while the symbols correspond to data collected flow conditions for the fuel/air jet, the pilot, and the coflowing from experimental data. In the following, these four scalars are air stream are obtained from separate simulations. The pilot flame used as quantities of interest for the Wasserstein metric to em- is treated by prescribing the scalar profile from the corresponding body the accuracy of the simulations in modeling mixing, heat re- chemistry table, with the mixture stoichiometry, temperature, and lease, fuel conversion, and emissions. Radial profiles for mixture mass flow rate representing the experimental setting. An improved fraction and temperature are in overall good agreement with mea- flame stability is experimentally observed with the inhomogeneous surements. Discrepancies are largely confined to regions in the jet inlet conditions. This can be attributed to the upstream premixed core and shear-layer, where simulations underpredict scalar mix- combustion of the near-stoichiometric fluid in the jet reacting with ing. A shift in the peak location for temperature and CO2 mass the hot pilot. Local extinction and re-ignition was found to be not fraction profiles at the intermediate axial locations, x/D J = { 5 , 10 } relevant under these operating conditions [43] . is apparent. This can be related to discrepancies in the mixing pro- To model the turbulent reacting flow-field, a flamelet/progress file. Results for mean CO-profiles are presented in the last row of variable (FPV) model is employed [47,48] , in which the thermo- Fig. 3 , showing that the simulation underpredicts this intermedi- chemical quantities are expressed in terms of a reaction-transport ate product, which can be attributed to shortcomings of the FPV- manifold that is constructed from the solution of steady-state non- combustion model [52] . premixed flamelet equations [49] . This model requires the solution Scatter data and mixture-fraction conditioned data from experi- of transport equations for the filtered mixture fraction, residual ments and simulations are shown in Fig. 4 . This data is sampled at mixture fraction variance, and filtered reaction progress variable. a subset of the axial locations, while using Z , T , and mass fractions

These modeled equations take the following form: of CO2 and CO as the same four quantities of interest. Scatter data

res are frequently examined to assess the agreement of the thermo- ρDt Z = ∇ · ( ρα∇ Z ) + ∇ · τ , (14a) Z chemical state space that is accessed by the model and experi- ment. While this direct comparison provides insight about shifts in the composition proﬁles, as seen for mass fractions of CO and CO , 2 2 res res 2 ρD Z = ∇ · ( ρα∇ Z ) + ∇ · τ − 2 ρu Z · ∇ Z − ρχ , t Z 2 Z such comparisons are mostly of qualitative nature. By utilizing the (14b) Wasserstein metric, these scatter data and statistical results will be used in the next section to obtain an objective measure for the

res quantitative assessment of the agreement between measurements ρD C = ∇ · ( ρα∇ C ) + ∇ · τ + ρω˙ , (14c) t C C and simulations. in which the turbulent fluxes are modeled by a gradient trans- χres 4. Results for application of Wasserstein metric port assumption [50], and the residual scalar dissipation rate Z is evaluated using spectral arguments [51] . With the solution of Eqs. (14a)-(14c) , all thermo-chemical quantities are then ex- To introduce the Wasserstein metric as a quantitative valida- pressed in terms of Z , Z 2 , and C , and a presumed PDF-closure is tion tool, in the following we consider two test cases. The first test used to model the turbulence-chemistry interaction. For this, the case, presented in Section 4.1 , focuses on the analysis of a single- marginal PDF for mixture fraction is described by a presumed β- scalar experimental results in which mixture fraction data is con- PDF, and the conditional PDF of the reaction progress variable is sidered at individual points in the flame. Previous work has shown modeled as a Dirac-delta function. that the mixture fraction can reasonably be approximated by A recently performed analysis of the model compliance showed a β-distribution [57] , and this test case examines this premise by that the FPV-approach only provides an incomplete description of applying the Wasserstein metric to experimental data and modeled the interaction between the partially premixed mixture and the β-distribution that is obtained from a maximum likelihood estima- hot pilot and discrepancies in the prediction of carbon monoxide tion (MLE) of the measurements. This one-dimensional test case is [52] . Therefore, the present simulation is intended for the purpose intended to present the capabilities of the Wasserstein metric in a of demonstrating the merit of employing the Wasserstein metric simplified context. as a quantitative validation measure and to identify discrepancies The second test, presented in Section 4.2 , considers the quanti- of the model through direct comparisons against experiments. Ex- tative validation of LES modeling results against experimental data. tended flamelet models have been developed to describe the com- For this, the Wasserstein metric will be employed to incorporate multiple thermo-chemical quantities, including Z , T , Y , and Y , plex flame topology and turbulence-chemistry interaction appear- CO2 CO ing in this configuration [53–56] , and the performance of other thereby contracting information about the model accuracy for pre- models will be examined in Section 5 . dicting mixing, fuel conversion, and emissions into a single validation measure. Several locations in the flame will be considered to 3.3. Statistical results and scatter data evaluate potential model deficiencies, demonstrating the merit of the Wasserstein metric as multidimensional validation tool. Before we examine the Wasserstein metric, this section summa- In these test cases, scatter data and empirical distributions that rizes statistical results between simulations and experiments. The are reconstructed from statistical moment information, using the 94 R. Johnson et al. / Combustion and Flame 183 (2017) 88–101

Fig. 3. Comparison of radial proﬁles between experimental measurements and simulations at x/D J = { 1 , 5 , 10 , 15 } . Variables considered include mixture fraction, temperature, and species mass fractions of CO 2 and CO. method presented in Section 2.4 , are considered to examine the of interest. Superimposed over each plot is a maximum likelihood accuracy of both methods. estimation of this data using the β-distribution [58] , whose probability distribution function reads

4.1. Conserved scalar results (a + b) − ( − ) f (x ; a, b) = x a 1(1 − x ) b 1 , (15) (a )(b) Previous investigations have shown that the evolution of con- where ( · ) is the gamma function. Suppose that there are n in- served scalars in two-stream systems can be approximated by dependent samples drawn from a β-distribution, whose values are a β-distribution [57] , and a Dirichlet-distribution as a multivari- x , x , . . . , x , the method of maximum likelihood estimates the ate generalization of the β-distribution provides a description of 1 2 n parameters a and b by finding the arguments of the maxima for turbulent scalar mixing in multistream flows [53] . This section ex- the logarithmic likelihood function, amines the accuracy of representing the conserved scalar by a presumed PDF using the Wasserstein metric as a quantitative metric. n , = ( ( ; , )). To simplify the analysis, this study focuses on data collected at a b arg max log f xi a b (16) a,b i =1 the axial locations introduced previously ( x/D J = { 1 , 5 , 10 , 15 } ) and a set of four radial positions located at r/D J = { 0 , 0 . 5 , 0 . 85 , 1 . 2 } . Although the best-fit β-distribution provides a reasonable approx- The axial positions are spaced uniformly, whereas the radial po- imation for the data, there still are noticeable differences between sitions represent key locations in the burner geometry. Specifically, the experimental data and the MLE-fit. These differences are quan- these four radial locations correspond approximately to the center titatively expressed through the Wasserstein metric, and the com- of the fuel stream, the outer edge of the air stream, the midpoint puted values are reported at the top of each histogram in Fig. 5 . of the pilot, and the outer edge of the pilot, respectively. Although The quantitative evaluation of the Wasserstein metric shows these sixteen measurement locations were chosen for the analysis, that largest deviations between experimental and presumed dis- it is possible to apply the Wasserstein metric at any location in the tributions occur within the fuel jet near the nozzle exit. These flame and generate similar results. quantitative results are corroborated with an interpretation of the PDFs for mixture fraction, represented by the histograms in histograms. The largest deviations between the MLE β-distribution Fig. 5 , provide the experimental distribution for all of the points and measurements occur near the nozzle inlet, corresponding to R. Johnson et al. / Combustion and Flame 183 (2017) 88–101 95

Fig. 4. Qualitative comparison of scatter profiles between measurements and simulations at x/D J = { 1 , 5 } . Variables including temperature, and species mass fractions of CO 2 and CO are plotted against mixture fraction. Conditional mean results are laid over the scatter data. regions of high turbulence and strong mixing. Note that the magni- ing fine-grained information about the simulation accuracy, model tude of the Wasserstein metric, W2 (Z), is provided in natural units deficiencies in predicting certain scalar quantities, and the impact of mixture fraction, which is bounded to the interval [0, 1]. As of inconsistencies of boundary conditions on the simulation. These such, the metric provides an integral physical interpretation of the three fundamental features will be emphasized and built upon differences between both distributions. as Section 4.2 considers the multidimensional validation of a LES In the next step, we employ the Wasserstein metric to evaluate combustion model using experimental data. the accuracy of the presumed β-distribution along several radial profiles. For this, a total of 86 uniformly spaced points are con- 4.2. Multiscalar results sidered along the four axial measurement stations used previously. β At each point, a -distribution is constructed from the measure- Having provided a comparison for the Wasserstein metric ap- ments using the maximum likelihood estimate, from which calcu- plied to mixture fraction as a single scalar, this second test case lations of the Wasserstein metric are performed subsequently. Re- will demonstrate the application of the Wasserstein metric to mul- sults are presented in Fig. 6 , and they show that the agreement of tiple scalars in the form of joint scalar distributions. This property β the -distribution with the experimentally determined PDFs im- allows for multiple, simultaneous error calculations, which provide proves with increasing axial and radial distance. This observation a multifaceted, quantitative validation. Here, the Wasserstein met- corroborates the findings from the point-wise analysis in Fig. 5 , ric is used to compare simulation results with experimental data and agrees with physical expectation that with increasing down- for the piloted jet flame with inhomogeneous inlets as discussed stream distance the mixture composition approaches a homoge- in Section 3 . < neous state. The low absolute error values of W2 0.02 compared To quantitatively assess a combustion simulation, a multiscalar β to the [0,1] range of mixture fraction shows that, overall, the - Wasserstein metric can be evaluated that takes into account d distribution provides an adequate representation of the experimen- scalar quantities. In the present cases, four scalar quantities are tally determined mixture fraction data. considered, namely mixture fraction, temperature, and species While this test case qualitatively affirms that the mixture frac- mass fractions of CO2 and CO, and evaluations are performed at β tion PDF follows a -distribution, it also demonstrates three key different axial locations in the jet flame. traits of the Wasserstein metric. First, the quantitative nature of / = − , The W2 -metrics at x DJ 10 for three two-scalar cases: Z T the Wasserstein metric allows for direct comparisons between sev- Z − Y , and Z − Y are shown in Fig. 7 . The W -metric of sim- CO2 CO 2 eral distributions, simultaneously. For example, it is much easier ilar magnitude and radial profile are found in the cases of Z − T / = to compare the four distributions at x DJ 1 using the Wasser- and Z − Y , in which there is little discrepancy between the CO2 stein metric results, as opposed to analyzing their differences in results obtained from the scatter data and SMLD reconstruction. Fig. 5 directly. The Wasserstein metric also provides a compari- − The Z YCO case exhibits a much higher level of W2 . There is son in distribution space. It therefore contains information about also greater difference between the values from two sources of all moments, and is not limited to low-order moments such as experimental results. We then examine how the W2 -metric is af- mean and root-mean square quantities. Finally, the Wasserstein fected by increasing the number of scalars that is included in its metric is applicable to any location in the flow, thereby provid- evaluation. For this, we consider results at the same axial loca- 96 R. Johnson et al. / Combustion and Flame 183 (2017) 88–101

Fig. 5. Distributions of mixture fraction from measurements and presumed β-distribution at different locations in the piloted turbulent jet ﬂame with inhomogeneous inlets. The Wasserstein metric is calculated for each location, showing that the modeling performance improves with increasing axial and radial distance. For reference, values for mean ( μ) and standard deviation ( σ ), computed from the experimental data, are provided.

Fig. 6. Computed Wasserstein metric for mixture fraction between measurements and presumed β-distribution obtained from MLE, along radial directions at x/D J = { 1 , 5 , 10 , 15 } .

/ = − , − , − Fig. 7. Wasserstein metric, evaluated at x DJ 10: Z T Z YCO2 and Z YCO . The W2 -metrics are calculated based on scatter data (solid line) and SMLD-reconstructed data (dash-dotted line). R. Johnson et al. / Combustion and Flame 183 (2017) 88–101 97

/ = − , − − , − − − Fig. 8. Wasserstein metric, evaluated for four scalars at x DJ 10: Z, Z T Z T YCO2 and Z T YCO2 YCO . The plots demonstrate deviations in W2 -calculations based on data sources: scatter data (solid line) and SMLD-reconstructed data (dash-dotted line).

(a) Scatter data. (b) SMLD reconstruction.

Fig. 9. Calculations of Wasserstein metric, from (a) scatter data and (b) SMLD-reconstructed data that are evaluated from measurements and LES computations. The stacked results draw attention to the contributions from each dimension of comparison. The W 2 -metric is calculated along radial profiles at x/D J = { 1 , 5 , 10 , 15 } , and is non- dimensionalized using the standard deviation of experimental data. Superimposed on the floor of each plot are mean experimental temperature fields to illustrate where the error calculations lie within the flame.

tion of x/D J = 10 , and results are presented in Fig. 8 . The result of Wasserstein metric. It can be seen that main contributions to the Z − Y is repeated for clarity. Since the Wasserstein metric mea- deviations arise from the mixture fraction at the jet core, and CO2 sures differences in distribution, it allows for a direct evaluation approximately equal distributions from temperature and CO mass of how differences arising from uncertainties in boundary condi- fractions in the shear layer. This information can be used to guide tions or modeling errors manifest in the flow field. The values for corrections in the boundary conditions and the selection of the the Wasserstein metric increases as more quantities are considered combustion model. Since the transport matrices are different be- (from left to right in Fig. 8 ). This is to be expected as the pair-wise tween cases of different dimensionalities, no direct correspondence distances becomes larger with the inclusion of additional dimen- can be established between the stacking in Fig. 9 and the lower di- sions. mensional results in Fig. 8 . Note also that by increasing the num- A comparison of the Wasserstein metric, evaluated from the ber of scalar quantities in the evaluation of the Wasserstein metric, SMLD-reconstruction, shows that this statistical reconstruction correlations and scalar interdependencies are taken into account. technique qualitatively and quantitatively captures the results ob- Figure 9 reveals the sources and locations of model errors. For tained from the scatter data. However, the main deviation made by example, considering x/D J = 10 , most of the error is concentrated ≤ ≤ the SMLD-reconstruction arise when including YCO , which indicates in the region between 0.5 r/DJ 1. The largest contributors to deficiencies of the uncorrelated normal distribution in representing this error are YCO and T. There is additional, but less significant, the actual joint distribution including CO and possible strong cor- error in the central jet region from Z , which can be attributed to relations between CO and other quantities. deficiencies in the boundary conditions. Finally, the model error Having built the multiscalar Wasserstein metric, results for cal- drops off from all sources on the edge of the domain. Similar anal- culations at different axial locations in the flame are displayed in yses could be conducted for the other axial locations, but this ex-

Fig. 9. The figure on the left represents a W2 -comparison made us- ample demonstrates the usefulness of the Wasserstein metric as a ing scatter data, while the figure on the right represents a com- validation tool. Information from these calculations could be used parison made using SMLD-reconstructed data. Although scatter to identify model limitations, isolate regions where mesh refine- data provides a more representative conclusion, SMLD data, re- ment is needed, and where further measurements are required. constructed from mean and variance data, may be more practical Three further advantages of using the Wasserstein metric in based on the existing data reported from simulation results. Since combustion validation are outlined below. One of the method’s the data has been normalized by the standard deviation, as ex- benefits is that it isolates sensitivity to boundary conditions. The plained in Section 2.5 , the error metric is unitless. This property, Wasserstein metric calculations for x/D J = 1 provide some indi- along with the stacking of error contributions, allows for simple cation of the error at the boundary conditions. As results down- comparisons between species, as well as comparisons between ax- stream are examined, it can be seen that the error features from ial locations. The stacking represents error contributions that are the inlet are convecting and diminishing. One example of this be- computed based on each scalar’s contribution in pair-wise dis- havior is the mixture fraction error in the jet region, which is tances and the transport matrix obtained from the four-scalar a significant source of error for x/D J = 1 , but a minor source of 98 R. Johnson et al. / Combustion and Flame 183 (2017) 88–101 error downstream at x/D J = 15 . By comparing slices at the inlet ing the simulation results without loss of information, and a 5% and slices downstream, one can identify how boundary conditions level of uncertainty is estimated for the multiscalar Wasserstein introduce errors to the combustion model. Although some error metric. Next, each data set was normalized by the experimental features are diffusing, others are forming downstream. standard deviation, as set forth in Section 2.5 . Subsequently, the A second benefit to this approach is the detection of modeling multiscalar Wasserstein metric was evaluated from each data set, errors not arising from boundary conditions. These new peaks arise and results are presented in Fig. 10 . from deficiencies in the combustion model. For instance, the error From this direct comparison, the following observations can be for Y , just inside r/D = 1 is modest at the inlet, peaks sharply at drawn. First, the multiscalar Wasserstein metric, W (Z, T , Y , Y ) , CO J 2 CO2 CO x/D J = 5 , and reduces at x/D J = 10 , before peaking sharply again shows cumulative contributions of each scalar quantity to the at x/D J = 15 . This error represents a deficiency in the CO model- overall error that is invoked by the model prediction. As dis- ing, and it could be targeted for improvement in future versions of cussed in Section 2.2 , the analysis is performed using properly the combustion model. In this way, the Wasserstein metric helps non-dimensionalized quantities, so that the W2 -metric provides a identify regions of modeling error and highlights potential areas direct representation of the error; a numerical value of W 2 = 1 for model improvement. corresponds to an error of one standard deviation with respect Finally, a third benefit to the Wasserstein metric validation ap- to the measurement. From this follows the second observation proach is the seamless transition to multiscalar quantities and the that this decomposition of the W2 -metric provides direct infor- contraction of multidimensional information into a single scalar mation about relative contributions of each quantity that is in- value. Any number of additional species could be added to these cluded in its construction. Since the Wasserstein metric is con- plots, without a significant increase in computational cost. The re- structed from joint scatter data, it takes into account correlations sulting higher-dimensional calculations would offer an even more among scalar quantities, and can therefore be employed in isolat- detailed comparison of the simulated and experimental data, and ing causalities in model predictions. For instance, it can be seen provide more insight into boundary condition and combustion that simulations from Institutions 1 and 2 exhibit a relatively modeling error. Condensing multiple validation plots into a com- small error contribution from mixture fraction and temperature, pact metric makes the validation process easier to interpret, and but a larger contribution from CO mass fraction. In contrast, In- provides greater understanding about the effectiveness of combus- stitutions 3 and 4 show a bias towards larger errors in the mix- tion models. ture fraction data, whereas Institutions 9 and 10 show an approximately equally distributed contribution from all four quantities

5. Quantitative comparison of different simulation results to the W2 -metric. Such quantitative information can be useful in isolating model deficiencies and guiding the formulation of model In this section, we seek to demonstrate the capability of the extensions. Wasserstein metric as a tool for quantitative comparisons of simu- A third observation can be drawn by considering the axial evo- lation results that are generated using different modeling strate- lution of the W2 -metric. Specifically, from Fig. 10 it can be seen gies. The direct application of this metric has the potential for that most simulations show a reduction in the error with increas- eliminating the need for subjective model assessments. It can also ing axial distance. This trend can be explained by the equilibration provide a direct evaluation of strengths and weaknesses of certain of the combustion, reduction of the mixing, and decay of the tur- modeling approaches and can guide the need for further experi- bulence, which is typical for canonical jet flames. In turn, signif- mental data to constrain the model selection. icant deviations from this anticipated trend can hint at potential In this study, we resort to experimental and computational deficiencies in the model setup, or physically interesting combus- data of the piloted turbulent burner with inhomogeneous inlets tion behavior (such as local extinction, multistream mixing, flame that were collectively reported at the 13th International Work- lifting, or local vortex break-down) that requires further experi- shop on Measurement and Computation of Turbulent Nonpremixed mental investigation or refinement of the modeling procedure, im- Flames (TNF) [59] . By complying with the TNF-spirit of open scien- proving mesh-resolution, or adjustments of boundary conditions. tific collaborations and acknowledging that results reported at this As example, the simulation from Institution 5 in Fig. 10 shows sig- venue are a work-in-progress, we removed any reference to the nificant deviation of the mixture fraction at the second measure- original authorship in the following analysis. In this context, we ment location ( x/D J = 5 ), suggesting that results at this measure- would like to emphasize that the present investigation is not in- ment location require further analysis. tended to provide any judgment about a particular modeling strat- The fourth observation arises from the utility of the Wasser- egy. Such an endeavor requires a concerted community effort, by stein metric in separating contributions from boundary conditions which the herein proposed Wasserstein metric could come to use and combustion-model formulations. For instance, dominant dis- as a quantitative measure. It is noted that several of these TNF- crepancies at the first measurement location, x/D J = 1 , can most contributions have been extended since, and we would like to re- likely be attributed to uncertainties in the boundary conditions. fer to publications [52,60–65] , which provide further details on the Contribution of different combustion models can then be tested to validation and analysis of simulation results, as well as descrip- examine model developments, and guide model selections for sim- tion of combustion models and computational setups of individual ulating particular flame configurations. Simulation results reported groups. from this comparative study show a rather wide variation in model In this study, we concentrate on the flame configuration FJ-5GP- accuracy, ranging between W 2 = 0 . 8 for the most accurate simula- Lr75-57 that was discussed in Section 3.1 . The following analysis tion to values of W 2 = 3 , with largest errors occurring at the first concentrates on comparisons of scatter plots of mixture fraction, reported measurement station. temperature, and species mass fractions of CO2 and CO at four ax- These results show that the application of the Wasserstein met- ial locations x/D J = { 1 , 5 , 10 , 15 } . ric to a large set of simulation data provides for a quantitative In order to apply the Wasserstein metric to this extensive set of assessment of different simulation results. The practical utility of modeling results, we proceeded as follows. First, we downsampled this metric lies in the direct assessment of the model perfor- the scatter data by randomly selecting 50 0 0 points, each point mance and in tracking the model convergence as continuous im- containing information for Z, T , Y , and Y . This downsampling provements on the model formulation, boundary conditions, and CO2 CO procedure was performed to achieve reasonable runtimes. The cho- measurements are provided. In regard to providing a verbal eval- sen number of samples was found to be appropriate for represent- uation of the accuracy of a particular model, the Wasserstein R. Johnson et al. / Combustion and Flame 183 (2017) 88–101 99

( , , , ) Fig. 10. Quantitative comparison of multiscalar Wasserstein metric, W2 Z T YCO 2 YCO from ten anonymized LES-calculations, presented at the 13th TNF-workshop [59]. The decomposition of multiscalar calculations allows contributions from each variable at each axial location to become visible. This quantitative validation analysis enables models to be compared objectively. The four bar-graphs from each contribution correspond to axial locations of x/D J = { 1 , 5 , 10 , 15 } . (Results used in this figure were included with permission from TNF-contributors.) metric can be utilized as practically useful model performance in- Acknowledgments dex for model ranking [66] . The Wasserstein metric provides a way of ranking models, which we believe is an initial step towards ad- Financial support through NASA’s Transformational Tools and vancing the quantitative evaluation of combustion models. Technologies Program with Award No. NNX15AV04A is gratefully acknowledged. Resources supporting this work were provided by the NASA High-End Computing (HEC) Program through the NASA 6. Conclusions Advanced Supercomputing (NAS) Division at Ames Research Cen- ter and the National Energy Research Scientific Computing Center, This paper addresses the need for the validation of numeri- a DOE Office of Science User Facility supported by the Office of cal simulations by considering multiple scalar quantities (veloc- Science of the U.S. Department of Energy under Contract No. DE- ity, temperature, composition), different data presentations (sta- AC02-05CH11231 . We would like to thank Benoit Fiorina, Mélody tistical results, scatter data, and conditional results), and various Cailler, and contributors to the 2016 TNF Workshop for permitting data acquisitions (pointwise, 1D, and planar measurements at dif- us to use the computational data presented in Fig. 10 for analy- ferent spatial locations). To this end, the Wasserstein metric is pro- sis. To acknowledge that these results were a work-in-progress and posed as a general formulation to facilitate quantitative and ob- honor TNF-policy, we anonymized individual contributions. jective comparisons between measurements and simulations. This metric is a probabilistic measure and is applicable to empirical dis- Appendix A. Wasserstein metric for general probability tributions that are generated from scatter data or statistical results measures using probabilistic reconstruction. The Wasserstein metric was derived for turbulent-combustion Let ( M , d ) denote a complete separable metric space equipped applications and essential convergence properties where examined. with distance functions d , on which there are two probability mea- μ ν The resulting method was demonstrated in application to LES of a sures and with finite pth moments. Following Kantorovich’s μ ν partially premixed turbulent jet flame, and used to categorize er- formulation, we have the p th Wasserstein metric between and rors arising from deficiencies in the specification of boundary con- defined as 1 /p ditions and intrinsic limitations of combustion models. This inves- (μ, ν) = ( , ) p γ ( , ) , tigation was followed by applying the Wasserstein metric to dif- Wp inf d x y d x y (A.1) γ ∈ (μ, ν) M×M ferent simulations that were contributed to the TNF-workshop in μ ν γ × order to demonstrate the versatility of this method in establishing where ( , ) is a set of admissible measures on M M, whose μ ν an objective evaluation of large data sets. marginals are and .

In building upon previous statistical validation approaches, the Speciﬁcally for continuous distributions deﬁned on the Eu- = R d ( , ) = | − | , μ Wasserstein metric offers greater insight by condensing multiscalar clidean space with M and d x y x y the measures ν differences between data into a single quantitative error measure. and can be represented by their probability density functions de-

The Wasserstein metric gives an accurate indication of errors over noted by f and g. Thus, we can rewrite the definition in Eq. (A.1) as the whole flame, and its interpretation can assist in identifying 1 /p p causalities of discrepancies in numerical simulations, arising from W p (μ, ν) = inf d(x, y ) h (x, y ) dx dy , (A.2) ( , ) boundary conditions, complex flow-field structures, or model lim- h ∈ G f g R d R d itations. The method is easy to apply, and its major benefit lies where G ( f , g ) is a set of joint probability density functions, whose in the seamless extension to multiscalar analyses, thereby taking marginal density functions satisfy into account interdependencies of combustion-physical processes. When applied to combustion validation, the Wasserstein met- h (x, y ) dy = f (x ) , (A.3) R d ric can concisely communicate shortcomings of current models, and assist in the development of more accurate LES combustion models. h (x, y ) dx = g(y ) . (A.4) R d 100 R. Johnson et al. / Combustion and Flame 183 (2017) 88–101

The formulation in Eq. (A.2) of the Wasserstein metric entails [8] A.M. Kempf , B.J. Geurts , J.C. Oefelein , Error analysis of large-eddy simulation two different interpretations. The ﬁrst interpretation is very close of the turbulent non-premixed Sydney bluff-body ﬂame, Combust. Flame 158 (12) (2011) 2408–2419 .

to the origin of the optimal transportation problem. If each distri- [9] M. Khalil , G. Lacaze , J.C. Oefelein , H.N. Najm , Uncertainty quantiﬁcation in LES bution is viewed as a pile of “dirt” distributed in the Euclidean of a turbulent bluff-body stabilized ﬂame, Proc. Combust. Inst. 35 (2) (2015) space according to the probability, the metric is the minimum 1147–1156 .

[10] V.I. Bogachev, Measure theory, Springer, 2007. amount of work required to turn one pile into the other. The trans- [11] A.L. Gibbs , F.E. Su , On choosing and bounding probability metrics, Int. Stat. Rev. port plan h ( x , y ) represents the density of mass to be transported 70 (3) (2002) 419–435 . from x to y . The second interpretation views h ( x , y ) as the joint dis- [12] Y. Rubner , C. Tomasi , L.J. Guibas , Earth mover’s distance as a metric for image

tribution of x and y whose marginal distributions match f and g . retrieval, Int. J. Comput. Vision 40 (2) (2000) 99–121. [13] Z. Ren , J. Yuan , Z. Zhang , Robust hand gesture recognition based on ﬁn- The Wasserstein metric is the minimal expectation of the distance ger-earth mover’s distance with a commodity depth camera, 19th ACM Inter- between x and y among all such distributions. national Conference on Multimedia (2011), pp. 1093–1096 .

For the special case of one-dimensional distributions on the [14] Z. Su, Y. Wang, R. Shi, W. Zeng, J. Sun, F. Luo, X. Gu, Optimal mass transport for shape matching and comparison, IEEE Trans. Pattern. Anal. Mach. Intell. 37

real line, the Wasserstein metric possesses many useful properties (11) (2015) 2246–2259 . [67,68] . Let F and G be the cumulative distribution functions for [15] F. deGoes , D. Cohen-Steiner , P. Alliez , M. Desbrun , An optimal transport ap- one-dimensional distributions μ and ν, while F −1 and G −1 being proach to robust reconstruction and simpliﬁcation of 2d shapes, Comput. Graph. Forum 30 (5) (2011) 1593–1602 .

their corresponding inversions. The Wasserstein metric can then be [16] C. Villani , Optimal transport: old and new, A Series of Comprehensive Studies written in explicit form as in Mathematics, 338, Springer, 2008 . [17] J. Urbas, Mass transfer problems, vol. 41, Sonderforschungsbereich Nichtlineare / 1 1 p −1 −1 p Partielle Differentialgleichungen Bonn: Vorlesungsreihe, Sonderforschungs- W p (μ, ν) = | F (x ) − G (x ) | dx . (A.5) bereich 256 (1998) http://www.worldcat.org/title/mass-transfer-problems/

0 oclc/75884472 .

Furthermore, when μ and ν are marginal empirical distributions [18] R.J. McCann, N. Guillen, Five lectures on optimal transportation: geometry, regularity and applications, Analysis and Geometry of Metric Measure Spaces,

with the same number of samples, the relationship in Eq. (A.5) can Lecture Notes of The Séminaire de Mathématiques Supérieure (SMS), Montréal, be further simpliﬁed as 2012, pp. 145–180 . [19] G. Monge , Mémoire sur la théorie des déblais et des remblais, De l’Imprimerie n 1 /p 1 ∗ ∗ Royale, 1781. (μ, ν) = | − | p , Wp xi yi (A.6) [20] H.W. Kuhn, The Hungarian method for the assignment problem, Nav. Res. Lo- n i gist. Q. 2 (1-2) (1955) 83–97. [21] L.V. Kantorovich , G.S. Rubinstein , On a space of completely additive functions, ∗ ∗ where xi and yi are xi and yi in sorted order. Vestnik Leningrad. Univ 13 (7) (1958) 52–59. [22] L.V. Kantorovich , On a problem of Monge, J. Math. Sci. 133 (4) (2006) 1383 . [23] R.M. Dudley , Probabilities and metrics: convergence of laws on metric spaces,

Appendix B. Sample code for the evaluation of the with a view to statistical testing, vol. 45, Aarhus Universitet, Matematisk Insti- multidimensional Wasserstein metric tut, 1976 . [24] S.B. Pope , Turbulent ﬂows, Cambridge University Press, Cambridge, 20 0 0 . [25] R.Y. Rubinstein , D.P. Kroese , Simulation and the Monte Carlo method, John Wi- Matlab sample code is provided for three different test- ley & Sons, Inc., 2008 . cases, involving the evaluation of the Wasserstein metric for [26] P.J. Bickel , D.A. Freedman , Some asymptotic theory for the bootstrap, Ann. Stat. a single scalar quantity ( Z ), joint scalars ( Z − T ), and multi (1981) 1196–1217 . [27] J. Horowitz , R.L. Karandikar , Mean rates of convergence of empirical measures scalars ( Z − T − Y − Y ). The code, available at: https:// CO2 CO in the Wasserstein metric, J. Comput. Appl. Math. 55 (3) (1994) 261–273 . github.com/IhmeGroup/WassersteinMetricSample , is easily adapt- [28] N. Fournier , A. Guillin , On the rate of convergence in Wasserstein distance of able to other conditions. The implementation of the sample the empirical measure, Probab. Theory Relat. Fields 162 (3-4) (2015) 707–738 .

code mirrors the procedure laid out in Section 2.5 . The script [29] B. Efron, Bootstrap methods: another look at the jackknife, Ann. Stat. (1979) 1–26 . sampleWasserstein.m is the main program, while calcW2.m [30] M.R. Chernick , W. González-Manteiga , R.M. Crujeiras , E.B. Barrios , Bootstrap is the function that calculates the Wasserstein metric. methods, Springer, 2011 .

Inputs to calcW2.m , provided as example, include experi- [31] P.J. Bickel, J.J. Ren, The M out of N bootstrap and goodness of ﬁt tests with double censored data, Robust Statistics, Data Analysis, and Computer Intensive

mental and simulated data samples for the Sydney piloted jet Methods, Springer (1996), pp. 35–47 .

ﬂame with inhomogeneous inlet, at x/D J = 10 , r/D J = 0 . 6 , as well [32] M. Sommerfeld, A. Munk, Inference for empirical Wasserstein distances on ﬁ- as a list of comparison species. Outputs include the calculated nite spaces, 2016, arXiv: 1610.03287 . [33] L. Dümbgen , On nondifferentiable functions and the bootstrap, Probab. Theory

Wasserstein metric (W2 ), transport matrix, and data visualiza- Relat. Fields 95 (1) (1993) 125–140 . tions (for one-dimensional and two-dimensional cases, only). Addi- [34] S.B. Pope , A rational method of determining probability distributions in turbu- tional cases can be calculated by modifying the species_index lent reacting flows, J. Non-Equilib. Thermodyn. 4 (1979) 309–320 . [35] M. Ihme , H. Pitsch , Prediction of extinction and reignition in non-premixed input, where the available species include Z , T , Y , and CO2 turbulent flames using a flamelet/progress variable model. 1. A priori study

YCO . and presumed PDF closure, Combust. Flame 155 (2008) 70–89 . [36] J.H.C. Lisman , M.C.A. VanZuylen , Note on the generation of most probable fre- quency distributions, Stat. Neerl. 26 (1) (1972) 19–23 . References [37] O. Pele , M. Werman , Fast and robust earth mover’s distances, IEEE Interna- tional Conference on Computer Vision (2009), pp. 460–467 .

[1] M.V. Heitor , A.L.N. Moreira , Thermocouples and sample probes for combustion [38] R.K. Ahuja, Network ﬂows, Technische Hochschule Darmstadt, 1988 (Ph.D. the-

studies, Progress in Energy and Combustion Science 19 (3) (1993) 259–278 . sis).

[2] A.C. Eckbreth , Laser diagnostics for combustion temperature and species, Gor- [39] M. Muskulus, S. Verduyn-Lunel, Wasserstein distances in the analysis of time

don and Breach, 1996 . series and dynamical systems, Phys. D 240 (1) (2011) 45–58.

[3] K. Kohse-Höinghaus , R.S. Barlow , M. Aldén , J. Wolfrum , Combustion at the fo- [40] B. Efron, Empirical bayes deconvolution estimates, Biometrika 103 (1) (2016)

cus: laser diagnostics and control, Proc. Combust. Inst. 30 (1) (2005) 89–123 . 1.

[4] R.S. Barlow , Laser diagnostics and their interplay with computations to under- [41] N. Laird, Nonparametric maximum likelihood estimation of a mixing distribu-

stand turbulent combustion, Proc. Combust. Inst. 31 (2007) 49–75 . tion, J. Am. Stat. Assoc 73 (364) (1978) 805–811.

[5] R.S. Barlow , J.H. Frank , A.N. Karpetis , J.-Y. Chen , Piloted methane/air jet ﬂames: [42] S. Meares, A.R. Masri, A modiﬁed piloted burner for stabilizing turbulent

transport effects and aspects of scalar structure, Combust. Flame 143 (2005) ﬂames of inhomogeneous mixtures, Combust. Flame 161 (2) (2014) 484–495.

433–449 . [43] S. Meares, V.N. Prasad, G. Magnotti, R.S. Barlow, A.R. Masri, Stabilization of

[6] M. Aldén , J. Bood , Z. Li , M. Richter , Visualization and understanding of combus- piloted turbulent ﬂames with inhomogeneous inlets, Proc. Combust. Inst. 35

tion processes using spatially and temporally resolved laser diagnostic tech- (2) (2015) 1477–1484.

niques, Proc. Combust. Inst. 33 (1) (2011) 69–97 . [44] R.S. Barlow, S. Meares, G. Magnotti, H. Cutcher, A.R. Masri, Local extinction

[7] R.S. Barlow, Web site for the international workshop on measurement and and near-ﬁeld structure in piloted turbulent CH4/air jet ﬂames with inhomo-

computation of turbulent nonpremixed ﬂames (TNF), 1996 . http://www.ca. geneous inlets, Combust. Flame 162 (10) (2015) 3516–3540. sandia.gov/TNF/abstract.html . R. Johnson et al. / Combustion and Flame 183 (2017) 88–101 101

[45] P. Moin , S.V. Apte , Large-eddy simulation of realistic gas turbine combustors, [57] A.W. Cook , J.J. Riley , A subgrid model for equilibrium chemistry in turbulent AIAA J. 44 (4) (2006) 698–708 . flows, Phys. Fluids 6 (8) (1994) 2868 . [46] L. Esclapez , P.C. Ma , E. Mayhew , R. Xu , S. Stouffer , T. Lee , H. Wang , M. Ihme , [58] J. Rice , Mathematical statistics and data analysis, Nelson Education, 2006 . Fuel effects on lean blow-out in a realistic gas turbine combustor, Combust. [59] R.S. Barlow, 13th International Workshop on Measurement and Computation Flame 181 (2017) 82–99 . of Turbulent Nonpremixed Flames (TNF), 2016 . http://www.sandia.gov/TNF/ [47] C.D. Pierce , P. Moin , Progress-variable approach for large-eddy simulation of 13thWorkshop/TNF13.html . non-premixed turbulent combustion, J. Fluid Mech. 504 (2004) 73–97 . [60] P. Wang , F. Zieker , R. Schießl , N. Platova , J. Fröhlich , U. Maas , Large eddy sim- [48] M. Ihme , C.M. Cha , H. Pitsch , Prediction of local extinction re-ignition effects ulations and experimental studies of turbulent premixed combustion near ex- in non-premixed turbulent combustion using a flamelet/progress variable ap- tinction, Proc. Combust. Inst. 34 (1) (2013) 1269–1280 . proach, Proc. Combust. Inst. 30 (20 05) 793–80 0 . [61] F. Proch , A.M. Kempf , Numerical analysis of the Cambridge stratified flame se- [49] N. Peters , Laminar diffusion flamelet models in non-premixed turbulent com- ries using artificial thickened flame LES with tabulated premixed flame chem- bustion, Prog. Energy Combust. Sci. 10 (3) (1984) 319–339 . istry, Combust. Flame 161 (10) (2014) 2627–2646 . [50] T. Poinsot , D. Veynante , Theoretical and numerical combustion, R.T. Edwards, [62] H. Wang , P. Zhang , A unified view of pilot stabilized turbulent jet flames for Inc., Philadelphia, PA, 2001 . model assessment across different combustion regimes, Proc. Combust. Inst. [51] M. Ihme , H. Pitsch , Prediction of extinction and reignition in non-premixed 36 (2) (2017) 1693–1703 . turbulent flames using a flamelet/progress variable model. 2. A posteriori [63] K. Kleinheinz , T. Kubis , P. Trisjono , M. Bode , H. Pitsch , Computational study study with application to Sandia flames D and E, Combust. Flame 155 (2008) of flame characteristics of a turbulent piloted jet burner with inhomogeneous 90–107 . inlets, Proc. Combust. Inst. 36 (2) (2017) 1747–1757 . [52] H. Wu , M. Ihme , Compliance of combustion models for turbulent reacting flow [64] B.A. Perry , M.E. Mueller , A.R. Masri , A two mixture fraction flamelet model for simulations, Fuel 186 (2016) 853–863 . large eddy simulation of turbulent flames with inhomogeneous inlets, Proc. [53] Y. Chen , M. Ihme , Large-eddy simulation of a piloted premixed jet burner, Combust. Inst 36 (2) (2017) 1767–1775 . Combust. Flame 160 (2013) 2896–2910 . [65] K. Luo , J. Yang , Y. Bai , J. Fan , Large eddy simulation of turbulent combustion by [54] M. Ihme , Y.C. See , LES flamelet modeling of a three-stream MILD combustor: a dynamic second-order moment closure model, Fuel 187 (2017) 457–467 . analysis of flame sensitivity to scalar inflow conditions, Proc. Combust. Inst 33 [66] C.J. Willmott , Some comments on the evaluation of model performance, Bull. (2010) 1309–1317 . Am. Meteorol. Soc. 63 (11) (1982) 1309–1313 . [55] H. Wu , Y.C. See , Q. Wang , M. Ihme ,A Pareto-efficient combustion framework [67] S. Bobkov , M. Ledoux , One-dimensional empirical measures, order statistics with submodel assignment for predicting complex flame configurations, Com- and Kantorovich transport distances, Mem. Amer. Math. Soc. (2016) . preprint bust. Flame 162 (2015) 4208–4230 . [68] S.D. Cohen , L.J. Guibas , The earth mover’s distance: Lower bounds and invari- [56] Q. Wang , M. Ihme , Regularized deconvolution method for turbulent combus- ance under translation, CS-TR-97-1597, Department of Computer Science, Stan- tion modeling, Combust. Flame 176 (2017) 125–142 . ford University, 1997 .