EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES The Precautionary Principle (with Application to the Genetic Modification of Organisms) Nassim Nicholas Taleb⇤, Rupert Read§, Raphael Douady‡, Joseph Norman†,Yaneer Bar-Yam† ⇤School of Engineering, New York University †New England Complex Systems Institute ‡ Institute of Mathematics and Theoretical Physics, C.N.R.S., Paris §School of Philosophy, University of East Anglia F Abstract—The precautionary principle (PP) states that if an action PP states that if an action or policy has a suspected risk or policy has a suspected risk of causing severe harm to the public of causing severe harm to the public domain (such as domain (affecting general health or the environment globally), the action general health or the environment), and in the absence should not be taken in the absence of scientific near-certainty about its of scientific near-certainty about the safety of the action, safety. Under these conditions, the burden of proof about absence of harm falls on those proposing an action, not those opposing it. PP is the burden of proof about absence of harm falls on those intended to deal with uncertainty and risk in cases where the absence proposing the action. It is meant to deal with effects of of evidence and the incompleteness of scientific knowledge carries absence of evidence and the incompleteness of scientific profound implications and in the presence of risks of "black swans", knowledge in some risky domains.1 unforeseen and unforeseable events of extreme consequence. We believe that the PP should be evoked only in This non-naive version of the PP allows us to avoid paranoia and extreme situations: when the potential harm is systemic paralysis by confining precaution to specific domains and problems. Here we formalize PP, placing it within the statistical and probabilistic (rather than localized) and the consequences can involve structure of “ruin” problems, in which a system is at risk of total failure, total irreversible ruin, such as the extinction of human and in place of risk we use a formal"fragility" based approach. In these beings or all life on the planet. problems, what appear to be small and reasonable risks accumulate The aim of this paper is to place the concept of inevitably to certain irreversible harm. Traditional cost-benefit analyses, precaution within a formal statistical and risk-analysis which seek to quantitatively weigh outcomes to determine the best structure, grounding it in probability theory and the policy option, do not apply, as outcomes may have infinite costs. Even properties of complex systems. Our aim is to allow high-benefit, high-probability outcomes do not outweigh the existence of low probability, infinite cost options—i.e. ruin. Uncertainties result in decision makers to discern which circumstances require sensitivity analyses that are not mathematically well behaved. The PP the use of the PP and in which cases evoking the PP is is increasingly relevant due to man-made dependencies that propagate inappropriate. impacts of policies across the globe. In contrast, absent humanity the biosphere engages in natural experiments due to random variations with only local impacts. 2DECISION MAKING AND TYPES OF RISK Our analysis makes clear that the PP is essential for a limited set Taking risks is necessary for individuals as well as for de- of contexts and can be used to justify only a limited set of actions. cision makers affecting the functioning and advancement We discuss the implications for nuclear energy and GMOs. GMOs of society. Decision and policy makers tend to assume all represent a public risk of global harm, while harm from nuclear energy is comparatively limited and better characterized. PP should be used to risks are created equal. This is not the case. Taking into prescribe severe limits on GMOs. account the structure of randomness in a given system can have a dramatic effect on which kinds of actions are, or are not, justified. Two kinds of potential harm must be September 4, 2014 considered when determining an appropriate approach to the role of risk in decision-making: 1) localized non- spreading impacts and 2) propagating impacts resulting 1INTRODUCTION in irreversible and widespread damage. He aim of the precautionary principle (PP) is to prevent decision makers from putting society as a 1. The Rio Declaration on Environment and Development presents T it as follows: "In order to protect the environment, the precautionary whole—or a significant segment of it—at risk from the approach shall be widely applied by States according to their capabil- unexpected side effects of a certain type of decision. The ities. Where there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing Corresponding author: N N Taleb, email [email protected] cost-effective measures to prevent environmental degradation." 1 EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES 2 Traditional decision-making strategies focus on the Standard Risk Management Precautionary Approach localized harm systemic ruin case where harm is localized and risk is easy to calculate nuanced cost-benefit avoid at all costs from past data. Under these circumstances, cost-benefit statistical fragility based analyses and mitigation techniques are appropriate. The statistical probabilistic non-statistical variations ruin potential harm from miscalculation is bounded. convergent probabibilities divergent probabilities On the other hand, the possibility of irreversible and recoverable irreversible widespread damage raises different questions about the independent factors interconnected factors evidence based precautionary nature of decision making and what risks can be reason- thin tails fat tails ably taken. This is the domain of the PP. bottom-up, tinkering top-down engineered Criticisms are often levied against those who argue evolved human-made for caution portraying them as unreasonable and pos- Table 1: Two different types of risk and their respective sibly even paranoid. Those who raise such criticisms characteristics compared are implicitly or explicitly advocating for a cost benefit analysis, and necessarily so. Critics of the PP have also Probability of Ruin expressed concern that it will be applied in an overreach- 1.0 ing manner, eliminating the ability to take reasonable risks that are needed for individual or societal gains. 0.8 While indiscriminate use of the PP might constrain appropriate risk-taking, at the same time one can also 0.6 make the error of suspending the PP in cases when it is vital. 0.4 Hence, a non-naive view of the precautionary princi- ple is one in which it is only invoked when necessary, 0.2 and only to prevent a certain variety of very precisely de- fined risks based on distinctive probabilistic structures. Exposure 2000 4000 6000 8000 10 000 But, also, in such a view, the PP should never be omitted when needed. Figure 1: Why Ruin is not a Renewable Resource. No The remainder of this section will outline the differ- matter how small the probability, in time, something ence between the naive and non-naive approaches. bound to hit the ruin barrier is about guaranteed to hit it. 2.1 What we mean by a non-naive PP Risk aversion and risk-seeking are both well-studied occur within a system, even drastic ones, fundamentally human behaviors. However, it is essential to distinguish differ from ruin problems: a system that achieves ruin the PP so that it is neither used naively to justify any act cannot recover. As long as the instance is bounded, e.g. of caution, nor dismissed by those who wish to court a gambler can work to gain additional resources, there risks for themselves or others. may be some hope of reversing the misfortune. This is The PP is intended to make decisions that ensure not the case when it is global. survival when statistical evidence is limited—because Our concern is with public policy. While an individual it has not had time to show up —by focusing on the may be advised to not "bet the farm," whether or not he adverse effects of "absence of evidence." does so is generally a matter of individual preferences. Table 1 encapsulates the central idea of the paper and Policy makers have a responsibility to avoid catastrophic shows the differences between decisions with a risk of harm for society as a whole; the focus is on the aggregate, harm (warranting regular risk management techniques) not at the level of single individuals, and on global- and decisions with a risk of total ruin (warranting the systemic, not idiosyncratic, harm. This is the domain of PP). collective "ruin" problems. Precautionary considerations are relevant much more 2.2 Harm vs. Ruin: When the PP is necessary broadly than to ruin problems. For example, there was a precautionary case against cigarettes long before there The purpose of the PP is to avoid a certain class of what, was an open-and-shut evidence-based case against them. in probability and insurance, is called “ruin" problems Our point is that the PP is a decisive consideration for [1]. A ruin problem is one where outcomes of risks ruin problems, while in a broader context precaution is have a non-zero probability of resulting in unrecoverable not decisive and can be balanced against other consid- losses. An often-cited illustrative case is that of a gambler erations. who loses his entire fortune and so cannot return to the game. In biology, an example would be a species that has gone extinct. For nature, "ruin" is ecocide: an 3WHY RUIN IS SERIOUS BUSINESS irreversible termination of life at some scale, which could The risk of ruin is not sustainable. By the ruin theorems, be planetwide.