+ki AECL EACL UNCERTAINTY ANALYSIS GUIDE T.H. Andres Whiteshell Laboratories Pinawa, Manitoba ROE IL0 2002 AECL-12103 A i@ AECL EACL GUIDE D’ANALYSE D’INCERTITUDE par T. H. Andres Le present guide s’applique a l’estimation de l’incertitude dont les quantites sont calculees par des programmes informatiques d’analyse et de conception scientifiques qui entrent dans le cadre du guide d’assurance qualid du logiciel (AQL) d’EACL. 11&unit les methodes rationnelles du guide AQL et trois autres sources differentes : a) la mtthodologie d’evaluation CSAU @de &aling, &plicability, and Qncertainty); b) le Guide pour 1 ‘expression de I ‘incertitude de mesure de 1’ISO et c) la methode d’analyse du risque SVA @sterns _Yariability &alysis). Ce rapport d&it la meilleure facon d’estimer et d’exprimer les incertitudes aleatoires et systematiques en quantites calculees. L’incertitude aleatoire dans la sortie du modele provient d’incertitudes d’entree. On peut rep&enter de diverses facons la propagation de cette incertitude au moyen d’un modele informatique, comprenant les calculs exacts, les approximations de series et les mtthodes de Monte Carlo. Les incertitudes systematiques proviennent de l’elaboration du modble informatique lui-meme, par le biais de simplifications et de mesures de prudence, par exemple. On doit estimer et combiner ces dernieres avec les incertitudes aleatoires pour determiner l’incertitude combinee dans une sortie du modele. Ce rapport porte egalement sur la facon dont on doit employer les incertitudes dans la validation du code, pour determiner si les experiences et les simulations concordent et si le code satisfait ou non a la tolerance prescrite pour cette application. Laboratoires de Whiteshell Pinawa (Manitoba) ROE 1LO 2002 AECL-12103 1 Rsi AECL EACL UNCERTAINTY ANALYSIS GUIDE T.H. Andres ABSTRACT This guide applies to the estimation of uncertainty in quantities calculated by scientific, analysis and design computer programs that fall within the scope of AECL’s software quality assurance (SQA) manual. The guide weaves together rational approaches from the SQA manual and three other diverse sources: (a) the CSAU (Code Scaling, bplicability, and IJncertainty) evaluation methodology; (b) the IS0 Guide for the Expression of Uncertainty in Measurement; and (c) the SVA (Systems variability Analysis) method of risk analysis. This report describes the manner by which random and systematic uncertainties in calculated quantities can be estimated and expressed. Random uncertainty in model output can be attributed to uncertainties of inputs. The propagation of these uncertainties through a computer model can be represented in a variety of ways, including exact calculations, series approximations and Monte Carlo methods. Systematic uncertainties emerge from the development of the computer model itself, through simplifications and conservatisms, for example. These must be estimated and combined with random uncertainties to determine the combined uncertainty in a model output. This report also addressesthe method by which uncertainties should be employed in code validation, in order to determine whether experiments and simulations agree, and whether or not a code satisfies the required tolerance for its application. Whiteshell Laboratories Pinawa, Manitoba ROE 1LO 2002 AECL-12103 ii ACKNOWLEDGEMENTS Bruce McDonald, as Program Manager for this and related activities, has given me a free hand to work out the techniques described here. As substantial development work was required, this opportunity was very important to the eventual production of this guide. Discussions with a number of analysts have helped to clarify my thinking. The list includes Amad Abdul-Razzak, Bert Carlucci, Ray Dickson and Maw-Rong Lin. Romney Duffey first focused our attention on the CSAU methodology. Harve Sills raised a number of interesting issues, while employing some aspectsof the techniques proposed here in the MMIR project. Because of his use of response surfaces, I investigated them for this report, and found them to be more useful than expected in some circumstances. Jason Pascoe of Ontario Power Generation pointed out the need for code accuracy estimation, as opposed to just uncertainty analysis. Laverne Wojciechowski gave me valuable feedback on the readability of the text. In many cases, the techniques described in this guide have their roots in the years I spent working in the Nuclear Fuel Waste Management Program. The ideas were influenced during that time by insights and discussions with co-workers, primarily Bruce Goodwin, Ted Melnyk, Dennis LeNeveu, Steve Oliver and Chuck Kitson. TABLE OF CONTENTS 1. SCOPE OF THIS GUIDE . .. .. .. .. .. .. I 2. DEFINITIONS . .. .. .. .. .. .. .. .. .2 3. BACKGROUND .................................................................................................................... 4 3.1 CSAU-Code Scaling, Applicability and Uncertainty .................................................. 5 3.2 GUM95-Guide to the Expression of Uncertainty in Measurement ............................. 6 3.3 SVA-Systems Variability Analysis .............................................................................. 6 3.4 SQA-AECL’s Software Quality Assurance Manual .................................................... 7 4. UNCERTAINTY ANALYSIS OF MEASURED VALUES (MEASURANDS) ................ 11 4.1 Attributes of Measurands ............................................................................................. 11 4.2 Random and Systematic Errors of Measurands........................................................... .12 4.3 Random and Systematic Uncertainties of Measurands ................................................ 16 4.4 Establishing Intervals with a Given Coverage Probability ........................................... 18 5. THEORY OF UNCERTAINTY ANALYSIS OF SIMULATED VALUES (SIMULANDS) ................................................................................................................... .28 5.1 Objective of Uncertainty Analysis of Simulands ......................................................... 28 5.2 Sources of Uncertainty in Simulation ......................................................................... ,30 5.2.1 Stochastic Uncertainty.. ....................................................................................... .30 5.2.2 Uncertainty in Inputs ........................................................................................... .32 5.2.3 Model Uncertainty.. ............................................................................................. .33 5.3 Representation of Input Uncertainties .......................................................................... 34 5.4 Monte Carlo Uncertainty Analysis of a Simuland ....................................................... 37 5.5 Theoretical Uncertainty Analysis of a Simuland.. ....................................................... .42 5.6 Perturbation Uncertainty Analysis of a Simuland ....................................................... .48 6. RECOMMENDED APPROACH FOR UNCERTAINTY ANALYSIS OF SIMULANDS.. .................................................................................................................... .54 6.1 Summary ...................................................................................................................... 54 6.2 Evaluate a Point Estimate j( k, f, 2,. .) .................................................................... .55 6.3 Perform a Small Set of Randomized Runs.. ................................................................ .57 6.4 Replicate Randomized Runs with All Parameters Varying ......................................... 68 I iv 7. CONSIDERATIONS FOR RANDOMIZED SAMPLING .................................................. 73 7.1 Summary ...................................................................................................................... 73 7.2 Sampling Requirements ............................................................................................... 75 7.3 Simple Random Sampling ............................................................................................ 77 7.4 2-Level Fractional Factorial Designs ........................................................................... 79 7.5 Latin Hypercube and Fractional Factorial / Latin Hypercube Designs ........................84 7.6 Low Discrepancy (Quasi-Random) Sequences............................................................ 87 7.7 Statistics with Non-Random Samples .......................................................................... 92 8. VALIDATION USING UNCERTAINTY ANALYSIS ...................................................... 99 8.1 What is Validation7....................................................................................................... 99 8.2 Comparing Two Uncertain Quantities ...................................................................... ,100 8.3 Comparing Simulations to Experimental Data for a Single Value ............................102 8.4 Comparing a Set of Observations .............................................................................. .112 8.5 Assessment of Code Accuracy.. ................................................................................ .116 9. CONCLUSIONS . .. .. .. .. .. .. .. .. .. 121 10. REFERENCES . .. .. .. .. .. .. .. .. 122 APPENDIX A SAMPLE DESIGNS . .. .. .. .. .. .. .. 128 APPENDIX B PROPERTIES OF PROBABILITY DISTRIBUTIONS . .. ..I...................... 136 V LIST OF FIGURES Figure 4.4/l.