Uncertainty and uncertainty analysis methods Keith R. Hayes February 2011 Report Number: EP102467 Issues in quantitative and qualitative risk modeling with application to import risk assessment ACERA project (0705) Enquiries should be addressed to: Keith R Hayes CSIRO Mathematics, Informatics and Statistics GPO Box 1538 Hobart, Tasmania, Australia, 7001 Castray Esplanade, Hobart, Australia, 7000 Telephone : +61 3 6232 5260 Fax : +61 3 6232 5485 Email : [email protected] Distribution List Client (1) Publications Officer (X) Stream Leader (1) Authors (1) Copyright and Disclaimer c CSIRO To the extent permitted by law, all rights are reserved and no part of this publication covered by copyright may be reproduced or copied in any form or by any means except with the written permission of CSIRO. Important Notice CSIRO advises that the information contained in this publication comprises general statements based on scientific research. The reader is advised and needs to be aware that such information may be incomplete or unable to be used in any specific situation. No reliance or actions must therefore be made on that information without seeking prior expert professional, scientific and technical advice. To the extent permitted by law, CSIRO (including its employees and consul- tants) excludes all liability to any person for any consequences, including but not limited to all losses, damages, costs, expenses and any other compensation, arising directly or indirectly from using this publication (in part or in whole) and any information or material contained in it. Contents 1 Introduction..................................... 8 1.1 Project background...............................8 1.2 Report structure and outline...........................8 1.3 Fundamental issues............................... 10 1.4 Notation..................................... 11 2 Risk and uncertainty................................. 14 2.1 What is risk?................................... 14 2.2 Why and how is risk assessed?......................... 17 2.3 What is uncertainty?............................... 21 2.3.1 Linguistic uncertainty.......................... 23 2.3.2 Variability................................ 25 2.3.3 Epistemic uncertainty.......................... 26 3 Uncertainty analysis................................. 28 3.1 Linguistic uncertainty.............................. 28 3.2 Epistemic uncertainty and variability...................... 29 3.2.1 Methods overview............................ 29 3.2.2 Model uncertainty............................ 32 3.2.3 Completeness.............................. 38 3.2.4 Scenario uncertainty........................... 38 3.2.5 Subjective judgement.......................... 38 3.2.6 Measurement error............................ 42 3.2.7 Parametric uncertainty and sampling uncertainty............ 42 3.3 Dependence................................... 48 4 Uncertainty analysis methods............................ 60 4.1 Analytical methods............................... 60 4.2 Probabilistic methods.............................. 62 4.2.1 Monte Carlo simulation......................... 62 4.2.2 Probability boxes and probability bounds analysis........... 66 4.3 Graphical methods................................ 73 4.3.1 Bayesian networks............................ 73 4.3.2 Loop analysis.............................. 77 4.3.3 Fuzzy cognitive maps.......................... 81 4.4 Non-probabilistic methods........................... 85 4.4.1 Fuzzy set theory............................. 85 4.4.2 Interval analysis............................. 86 4.4.3 Info-gap theory............................. 88 5 Uncertainty analysis in practise........................... 93 5.1 Who’s using what?............................... 93 5.2 Methods evaluation............................... 96 1 5.2.1 Assumptions............................... 96 5.2.2 Technical complexity.......................... 97 5.2.3 Software availability........................... 99 5.2.4 Decision utility.............................. 99 6 Discussion and recommendations.......................... 102 6.1 Uncertainty and qualitative risk assessment................... 102 6.2 Honest quantitative risk assessment....................... 103 6.3 Uncertainty, statistics and quantitative risk assessment............. 104 6.4 A recommended strategy for uncertainty analysis............... 106 2 Acknowledgements In completing this report I have been assisted and encouraged by friends, colleagues and collab- orators. Simon Barry (CSIRO) and Gareth Peters (UNSW) have patiently guided me through modern statistical methods, and commented on various aspects of the report. Scott Ferson intro- duced me to Probability Bounds Analysis and provided me with a copy of his pbox R libraries. The PBA analysis presented in this report would not have been possible without his generosity. My Hobart team - Jeff Dambacher (CSIRO) and Geoff Hosack (CSIRO) - have been a constant source of support and inspiration. They also introduced me to the concept of qualitative mod- eling. Jeff contributed to Appendix B of this report and Geoff contributed to the discussion on fuzzy cognitive maps in Section 4.3. I would like to thank Greg Hood (Bureau of Rural Science) and Petra Kuhnert (CSIRO) for their encouragement, comments along the way and programming advice. Alan Walsh (ANU) and Xunguo Lin (CSIRO) acted as sounding boards for half-baked ideas and helped clarify my thoughts. Brent Henderson (CSIRO), Yakov Ben-Haim (Technion), Scott Ferson (Applied Biomathematics), James Franklin (UNSW) and two anonymous reviewers provided comments that have improved the report. Lynda Watson (CSIRO) kindly assisted with proof reading. This report is the product of the Australian Center of Excellence for Risk Analysis (ACERA) and CSIRO. In preparing this report, the author acknowledges the financial and other sup- port provided by the Australian Government Department of Agriculture, Fisheries and Forestry (AGDAFF) and the University of Melbourne. 3 Executive summary This report reviews uncertainty and uncertainty analysis methods in risk assessment, with a specific focus on issues related to import risk assessment. The report is motivated by the avail- ability of qualitative and quantitative methods for import risk assessment. It examines how the challenges posed by uncertainty influence choices between these two approaches. The project’s terms of reference are to summarise and categorise the different sources of uncertainty in risk assessment problems, and review the practicality and applicability of a range of treatment meth- ods. The report is intended for scientists and managers involved in, or contemplating the use of, qualitative or quantitative risk assessment. Whilst the report focusses on import risk as- sessment, readers from other application domains will find that much of the information and analysis presented here is relevant to them. Uncertainty is a term used to encompass many concepts. It has been described, defined and categorised in many different ways, using different names for the same things and occasion- ally the same name for different things. This report identifies four basic sources of uncertainty: uncertainty that arises through the vagarious nature of language (linguistic uncertainty), the un- certainty created by our limited understanding of natural systems (epistemic uncertainty), the uncertainty created by the irreducible variation in these systems (variability), and finally the un- certainty associated with our value systems and management decisions (decision uncertainty). The report examines in detail the various sources of linguistic uncertainty, epistemic uncertainty and variability in scientific endeavors and risk-related problems. It summarises probabilistic, non-probabilistic and graphical methods for treating and propagating these sources of uncer- tainty through risk assessment under the headings of five basic strategies: ignore it, eliminate it, envelope it, average over it or factorise it. It also examines the related problem of dependency that occurs when arithmetic operations are performed with random variables. The principal impediment to uncertainty analysis within qualitative risk assessment is that vari- ability and epistemic uncertainty are confounded with each other and with linguistic uncertainty. Separating the three sources of uncertainty requires, as a minimum, that linguistic uncertainty is eliminated from the problem as far as possible. Fuzzy sets and possibility theory provide a mechanism that was specifically designed to eliminate two important sources of linguistic uncertainty (vagueness and ambiguity). These sources of uncertainty, however, can also be eliminated with probability theory via carefully implemented elicitation methods and probabil- ity bounds analysis. This approach has the additional advantages of: a) being able to minimise other well known heuristics and biases in human perception and judgements of uncertain events; and, b) couching its analysis within the realms of probability theory which is likely to be more familiar to decision makers than evidence or possibility theory. Furthermore, there are a range of issues with qualitative approaches to risk assessment that relate to the science-quality criteria of transparency, repeatability and falsifiability, and the decision- utility criteria of precision and accuracy, namely: • qualitative risk assessment predictions cannot be (in)validated with observations, and un- certainty cannot be coherently propagated through risk functions