SANDIA REPORT SAND20054302 Unlimited Release Printed August 2005 nd Experiment: and Matthew F. Barone Prepared by Sandia National Laboratories Albuquerque, New Mexico 871 85 and Livermore, California 94550 Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear Security Administration under Contract DE-AC04-94AL85000. Approved for public release; further dissemination unlimited. @ Sandia National laboratories Issued by Sandia National Laboratories, operated for the United States Department of Energy by Sandia Corporation. NOTICE: This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government, nor any agency thereof, nor any of their employees, nor any of their contractors, subcontractors, or their employees, make any warranty, express or implied, or assume any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represent that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government, any agency thereof, or any of their contractors or subcontractors. The views and opinions expressed herein do not necessarily state or reflect those of the United States Government, any agency thereof, or any of their contractors. Printed in the United States of America. This report has been reproduced directly from the best available copy. Available to DOE and DOE contractors from U.S. Department of Energy Office of Scientific and Technical Information P.O. Box 62 Oak Ridge, TN 37831 Telephone: (865)576-8401 Facsimile: (865)576-5728 E-Mail: [email protected] Online ordering: ht@://www.osti.gov/ridge Available to the public from U.S. Department of Commerce National Technical Information Service 5285 Port Royal Rd Springfield, VA 22 16 1 Telephone: (800)553-6847 Facsimile: (703)605-6900 E-Mail: [email protected] Online order: ht@://www.ntis.~ovkel~/orde~ethods.asp?~oc=7-4-O#on~ine 2 SAND2005-4302 Unlimited Release Printed August 2005 Measures of Agreement Between Computation and Experi men t : Val idat ion Metri cs William L. Oberkampf Validation and Uncertainty Quantification Department Matthew F. Barone Aerosciences and Compressible Fluid Dynamics Department Sandia National Laboratories P. 0. Box 5800 Albuquerque, New Mexico, 871 85-0828 Abstract With the increasing role of computational modeling in engineering design, performance estimation, and safety assessment, improved methods are needed for comparing computational results and experimental measurements. Traditional methods of graphically comparing computational and experimental results, though valuable, are essentially qualitative. Computable measures are needed that can quantitatively compare computational and experimental results over a range of input, or control, variables and sharpen assessment of computational accuracy. This type of measure has been recently referred to as a validation metric. We discuss various features that we believe should be incorporated in a validation metric and also features that should be excluded. We develop a new validation metric that is based on the statistical concept of confidence intervals. Using this fundamental concept, we construct two specific metrics: one that requires interpolation of experimental data and one that requires regression (curve fitting) of experimental data. We apply the metrics to three example problems: thermal decomposition of a polyurethane foam, a turbulent buoyant plume of helium, and compressibility effects on the growth rate of a turbulent free-shear layer. We discuss how the present metrics are easily interpretable for assessing computational model accuracy, as well as the impact of experimental measurement uncertainty on the accuracy assessment. 3 Acknowledgements The authors thank Jon Helton, consultant to Sandia, for the enlightening discussions and insights. We also thank Tim Trucano, Basil Hassan, Marty Pilch, Laura Swiler, and Tony Giunta of Sandia for reviewing an earlier version of the paper and making a number of suggestions for improvement. The cooperation of Fred Blottner, Kevin Dowding, Sheldon Tieszen, and Tim O’Hern of Sandia in sharing recent results used in the example problems is greatly appreciated. Contents 1. Introduction ................................................................................................. 8 2 . Review of the Literature ................................................................................... 9 2.1 Review of the Terminology and Processes ......................................................... 9 2.2 Review of Approaches ............................................................................... 12 3 . Construction of Validation Metrics ..................................................................... 14 3.1 Recommended Features of Validation Metrics ................................................... 14 3.2 Perspectives of the Present Approach ............................................................. 16 4 . Validation Metric for One Condition ................................................................... 17 4.1 Development of the Equations ..................................................................... 17 4.2 Example: Thermal Decomposition of Foam ...................................................... 22 5 . Validation Metric Using Interpolation ................................................................. 25 5.1 Development of the Equations ..................................................................... 25 5.2 Global Metrics ........................................................................................ 27 5.3 Example: Turbulent Buoyant Plume .............................................................. 28 6 . Validation Metric Requiring Regression ............................................................... 34 6.1 Development of the Equations ..................................................................... 34 6.2 Solution of the Equations ........................................................................... 36 6.3 Example: Compressible Free-Shear Layer ....................................................... 37 7 . Conclusions and Future Work ........................................................................... 44 References .................................................................................................... 46 5 Figures Figure 1 Validation. calibration. and prediction ......................................................... 11 Figure 2 Schematic of foam decomposition experiment .............................................. 23 Figure 3 Statistical and validation-metric results of foam decomposition .......................... 25 Figure 4 Experimental setup for measurements of the helium plume ............................... 29 Figure 5 Experimental measurements of time-averaged vertical velocity along the centerline for the helium plume .................................................... 30 Figure 6 Experimental sample mean with 90% confidence interval and computational result for vertical velocity in the helium plume ............................ 31 Figure 7 Validation metric result and 90% confidence interval for centerline velocity ...........32 Figure 8 Estimated error and true error in the model with 90% confidence interval ..............33 Figure 9 Example of various confidence regions for the case of two regression parameters .... 37 Figure 10 Flow configuration for the turbulent free-shear layer ...................................... 38 Figure 1 1 Experimental data for compressibility factor versus convective Mach number ....... 39 Figure 12 Three-dimensional 90% confidence region for the regression fit to the shear-layer experimental data .......................................................... 42 Figure 13 Comparison of the simulation result with the experimental data, nonlinear regression curve. and 90% simultaneous confidence interval ................ 43 Figure 14 Validation metric result and 90% confidence interval for 0 ............................. 44 Table Table 1 Experimental data for foam decomposition .................................................. 23 Nomenclature C = confidence level chosen, C = lOO(1- a)% l:lavg l:lavg = average confidence indicator normalized by the absolute valcle of the estimated experimental mean over the range of the experimental data, see Eq. (22) E = true error of the computational model as compared to the true mean of the - experimental measurements, y, - p E = estimated error of the computational model as compared to the estimated mean of the - experimental measurements, y, - y, N IHg= average of the absolute value of the relative estimated error over the range of the experimental data, see Eq. (21) = maximum of the absolute value of the relative estimated error over the range of the experimental data, see Eq. (23) = F probability distribution, where v, is the first parameter specifying the number of degrees of freedom, v, is the second parameter specifying the number of degrees of freedom, and 1 - a is the quantile for the confidence interval chosen = number of sample (experimental) measurements = sample (estimated)