Global Challenges Foundation

Global Catastrophic Risks 2016

© Global Challenges Foundation/Global Priorities Project 2016

GLOBAL CATASTROPHIC RISKS 2016 THE GLOBAL CHALLENGES FOUNDATION works to raise awareness of the The views expressed in this report are those of the authors. Their Global Catastrophic Risks. Primarily focused on climate change, other en- statements are not necessarily endorsed by the affiliated organisations. vironmental degradation and politically motivated violence as well as how these threats are linked to poverty and rapid population growth. Against this Authors: background, the Foundation also works to both identify and stimulate the Owen Cotton-Barratt*† development of good proposals for a management model – a global gover- Sebastian Farquhar* nance – able to decrease – and at best eliminate – these risks. John Halstead* Stefan Schubert* THE GLOBAL PRIORITIES PROJECT helps decision-makers effectively prior- Andrew Snyder-Beattie† itise ways to do good. We achieve his both by advising decision-makers on programme evaluation methodology and by encouraging specific policies. We * = The Global Priorities Project are a collaboration between the Centre for Effective Altruism and the Future † = The Future of Humanity Institute, University of Oxford of Humanity Institute, part of the University of Oxford.

Graphic design: Accomplice/Elinor Hägg

Global Challenges Foundation in association with

4 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 5 Contents Definition: Global

Foreword 8 Introduction 10 Catastrophic Risk Executive summary 12 1. An introduction to global catastrophic risks 20 – risk of events or 2. What are the most important global catastrophic risks? 28 Catastrophic climate change 30 processes that would Nuclear war 36 Natural pandemics 42 Exogenous risks 46 lead to the deaths of Emerging risks 52 Other risks and unknown risks 64 Our assessment of the risks 66 approximately a tenth of 3. Risk factors and interactions between risks 72 Drivers of individual risks 74 the world’s population, or Shared risk factors and interactions between risks 78 4. Do institutions collectively underinvest in global catastrophic risk? 82 have a comparable impact. Market and political failures 84 Which actors can help reduce global catastrophic risk? 88 5. What can the world do to reduce global catastrophic risk? 94 Endnotes 100 Acknowledgements 107 Contact info 107

6 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 7 FOREWORD

Dear Reader!

early four years ago tion from other important risks. And in favor of one or more organizations when the Global above all, the defense against various that have a mandate to decide on how Challenges Foundation forms of political violence requires to mitigate GCRs. was established, we a grotesquely large share of public Would this be possible? My count- decided on a direction resources. Each day, the world spends er question is whether there are any Nwith two parallel strategies. The first over SEK 40 billion on defence alternatives? To continue relying on is increasing the knowledge about expenditure – money that would be multilateral negotiations increases Global Catastrophic Risks (GCRs), needed to fight poverty and prevent the probability that decisions and which with our terminology means catastrophic risks. actions are insufficient and executed threats that can eliminate at least 10% My personal opinion is that in order too late. This means that the likeli- of the global population. This knowl- to drastically minimize GCRs we hood of GCRs continues to escalate. edge is an important prerequisite for must develop a model where a major- I hope that this publication can the Foundation’s second strategy: to ity of the world’s nations, with strong deepen the understanding of GCRs encourage debates and proposals as support from leading nations, can and that these insights provide a to how we can effectively and fairly make binding decisions which can fertile ground for both debates and reduce – and preferably eliminate – be enforced in an effective and fair proposals on how we can develop a these catastrophic risks. way. This would imply that individ- better way of managing and address- This publication, the Foundation’s ual nations waive their sovereignty ing these risks. Annual Report for 2016, is the re- this. In addition to the risks involved sult of a collaboration between the in the Annual Report for 2016, the Foundation and the Future of Hu- Foundation actively works with envi- manity Institute (FHI) and the Global ronmental degradation, weapons of Stockholm, April 2016 Priorities Project at Oxford University mass destruction, population growth in the U.K., which has now lasted for (that exacerbates several risks), and over two years. A big group of re- political violence which is behind searchers at the FHI, commissioned many of the world’s current problems. by the Foundation, summarized Political violence comes in many where research, focused on charting forms. Various kinds of weapons of some of the greatest global risks, cur- mass destruction represent poten- Laszlo Szombatfalvy rently stands. tially devastating weaponry. Further, Founder of Global Challenges Foundation In addition to describing the risks, political violence creates uncon- their effects and their likelihood of trolled migration and we receive occurring, this year’s Annual Report repeated reminders that there is takes one step further and try’s to also “digital violence” in the form of show how different risks relate to one cyber-attacks. Together, this takes up another, what can be done to combat a significant amount of space on the the risks and who can and should do political agenda, thus stealing atten-

8 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 9 INTRODUCTION

Global catastrophic risks pose a pressing challenge

his report addresses one caught out by consequences that fol- some risks. Smart risk management of the most important low from the technology more rapidly means being realistic in weighing issues of our age – glob- than climate change has. these factors against each other. This al catastrophic risk. As a global community, we need report offers an excellent background Over the last decades, to win the race between the growing to the underlying issues of global Tbehavioural psychology has taught power of our technology and the catastrophic risks, and is an outstand- us that, as a species, we are bad at wisdom with which we manage it. ing starting point for policy-makers assessing scope. Issues that affect ten This requires a nuanced approach developing an interest in the area or people do not intuitively seem ten towards technological developments, researchers considering how their times more important than those that acknowledging both that technol- own work might be brought into the affect one person. Global catastrophic ogy carries huge potential to make study of global catastrophic risks. risks are one area where our scope lives better and also that it carries insensitivity might prove the most dangerous. These risks can’t just be treated as problem for the future, even though we might well expect them not to ma- terialise this year or the next. At the of our universe itself – the beauty Future of Life Institute, my team and of which has inspired my own deep Max Tegmark I have been calling for global leaders curiosity in cosmology. Co-founder of the Future of Life Institute to address critical global risk issues This technological power is an Professor of Physics at MIT including nuclear weapons, biotech- enormous force for good, but carries nology and artificial intelligence. This its own risks. Although consuming builds on existing risk reduction work fossil fuels was critical in creating the led by institutions such as the United thriving and wonderful civilization Nations. we live in today, we’ve come to learn Over the last centuries, humanity that there are potentially catastroph- has achieved incredible things. New ic long-term consequences from medical technologies save millions of climate change. Other technologies, lives every year. Agricultural science more powerful than combustion en- allows billions to be fed who might gines, might also offer huge benefits otherwise not exist. And we have and carry unforeseen risks. If we fail begun to explore the very foundations to manage this risk well, we might be

10 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 11 EXECUTIVE SUMMARY

Executive Summary We believe the global

ost generations generations. Reducing these risks is community should work never experience a therefore both a global and an inter- global catastrophe. generational public good. However, the idea of The ever-evolving landscape of such catastrophes technology and society compounds together to harness new Mis not fanciful: plagues have killed these challenges. Technological over 10% of world’s population and and economic forces can create new we came close to nuclear war several global catastrophic risks, such as times in the 20th century. anthropogenic climate change and tools to address global Despite their scale, the risks of the 20th century’s nuclear arms race. global catastrophes receive limited But technology can also reduce risk, attention. One reason is that many for example through better vaccines catastrophic risks. of these risks are unlikely in any or clean energy. given decade. But even when the We believe the global community probability is low, the sheer magni- should work together to harness new tude of an adverse outcome warrants tools to address global catastrophic taking these risks seriously. A global risks. It is possible that, collectively, catastrophic risk not only threatens we significantly under-invest in glob- everyone alive today, but also future al catastrophic risk reduction.

12 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 13 EXECUTIVE SUMMARY

What are the biggest threats?

he global catastrophic risks in this report can be divided into T two categories. Some are on- going and could potentially occur in any given year. Others are emerging and may be very unlikely today but will become significantly more likely in the coming decades. The most significant ongoing risks are natural pandemics and nuclear war, whereas the most significant emerging risks are catastrophic climate change and risks stemming from emerging tech- nologies. Even where risks remain in the future, there are things we can do today to address them. The Spanish influenza pandemic of 1918 may have killed as much as 5% of the world population. Some outbreaks since then infected over a third of the world’s population (e.g., pandemic influenza), whereas others killed over half of people infected (e.g., Ebola or SARS). If a disease were to emerge that was as transmis- sible as the flu and as lethal as Ebola, the results could be catastrophic. Fortunately, this rarely transpires, but it is possible that it could, for ex- ample with the H5N1 influenza virus. The invention of nuclear weap- ons ushered in a new era of risks created by human action. A large nuclear war between major powers would likely kill tens or hundreds of millions in the initial conflict, and perhaps many more if a nuclear win- ter were to follow. During the Cuban

14 Global Catastrophic Risks 2016 EXECUTIVE SUMMARY

Missile Crisis, President Kennedy is expected or if positive environ- FIGURE 1. OUR ASSESSMENT OF GLOBAL CATASTROPHIC RISKS estimated the chance of nuclear mental feedback loops occur. conflict as “between one in three and Catastrophic risks from emerging even”. Tensions have eased some- technology are less well understood. what since the Cold War, but could Emerging technologies promise sig- recur. Moreover, accidents or mis- nificant benefits, but a handful could calculation with nuclear weapons also create unprecedented risks to continue to pose a risk. civilisation and the biosphere alike. Climate change is a well-known an- Biotechnology could enable the thropogenic risk. Even if we succeed creation of pathogens far more dam- in limiting emissions, scientists ex- aging than those found in nature, Natural pandemic pect significant climate change to oc- while in the longer run, artificial cur. This could bring a host of global intelligence could cause massive challenges including environmental disruption. degradation, migration, and the The relative likelihood and ur- possibility of resource conflict. But gency of the different risks matters Unknown risks Nuclear war this is not the worst-case scenario. when deciding how to respond. Even Although it receives far less atten- though the level of uncertainty is tion, scientists also acknowledge the extreme, rational action requires possibility of catastrophic climate explicit assessments of how much Engineered pandemic change. There is a small likelihood attention the different risks deserve, that warming could even exceed 6 °C, and how likely they are. The views of leaving large swathes of the planet the authors on these vexed ques- dramatically less habitable. This tions, based on our reading of the Higher likelihood over next 5 years could occur if emissions are not cut scientific evidence, are summarised sufficiently, if the sensitivity of the in the following table. More informa- climate system is different from what tion can be found in Chapter 2. Catastrophic Asteroid impact climate change

Supervolcanic Catastrophic eruption disruption from AI

Failure of geo-engineering Lower likelihood over next 5 years Low High APPROPRIATE LEVEL OF ATTENTION

16 Global Catastrophic Risks 2016 EXECUTIVE SUMMARY

Humanity can respond to these risks Research communities

or each of the risks in this To reduce the risk of global ca- report, we consider actions tastrophe caused by nuclear war: should further investigate F available to avoid or mitigate • The international community the risk, and which actors are best- should continue the policy of placed or responsible for taking that nuclear non-proliferation, and nu- action. For the most significant risks, clear states can continue to reduce the possible risks from some of the most promising opportu- stockpiles. nities are listed here. • Nuclear-weapon states should con- tinue to work to reduce the chance To reduce the risk of global ca- of accidental launch or escalation. emerging capabilities tastrophe caused by pandemic: • The World Health Organisation, To reduce the risk of global ca- nation states, and other bodies tastrophe caused by emerging in biotechnology and should increase their planning for technologies: extremely bad pandemics. • Research communities should • The global health community further investigate the possible should improve developing world risks from emerging capabilities in artificial intelligence, capacity for response, for example biotechnology and artificial intel- by ensuring that vaccine produc- ligence, and possible solutions. tion facilities are well-distributed • Policymakers could work with around the world. researchers to understand the and possible solutions. issues that may arise with these To reduce the risk of global ca- new technologies, and start to lay tastrophe caused by climate groundwork for planned adaptive change: risk regulation. • Research communities should in- crease their focus on understand- To reduce global catastrophic risk ing the pathways to and likelihood in a cross-cutting way: of catastrophic climate change, • Research communities should fo- and possible ways to respond. cus greater attention on strategies • Nations should continue to imple- and technologies for resilience to ment and improve mechanisms and recovery from global catastro- for emissions abatement such as phe, for example by developing carbon taxes or tradable emissions alternate food sources. quotas, as for non-catastrophic • Nations should work to incorpo- climate change. rate the interests of future gener- ations into their decision-making frameworks.

18 Global Catastrophic Risks 2016 CHAPTER 1 – AN INTRODUCTION TO GLOBAL CATASTROPHIC RISKS

Chapter 1 The most deadly An Introduction to event of the 20th century Global Catastrophic Risks was probably the Spanish influenza pandemic of ver the course of history, times been uncomfortably high. the world has suffered Throughout the Cold War, the disasters of such magni- threat of nuclear warfare loomed 1918-1920 which killed tude that human civil- large. The United States and the So- isation itself has been viet Union possessed tens of thou- Othreatened. Warfare and pandem- sands of high yield nuclear warheads, between 2.5% and 5% of ics have caused especially significant and their retaliatory strike systems damage. Originating in 541-542, the were programmed to respond to any initial outbreak of the ‘Great Plague of attack within minutes. The world has the world population… Our Justinian’ killed 25-33 million people come close to the nightmare scenar- – between 13% and 17% of the world io on a number of occasions. Perhaps population at the time.1 The plague the narrowest escape came on 27th had trans-generational consequenc- October 1962. Two Russian B-59 sub- focus here is on even more es: many historians believe that it marine commanders off the coast weakened the Byzantine Empire at of Cuba gave the order to launch a a crucial time, undermining its at- nuclear strike against the United extreme possibilities which tempts to reconquer Europe.2 States, on the mistaken assump- In recent times, humanity tion that war had already start- has not endured catastroph- ed. The launch of a nuclear receive less attention. ic events on the propor- As torpedo required the con- tionate scale of Plague of many sent of all three officers Justinian. However, the as on board; the second risk of global catastro- in command, Vasili end of the Cold War, the risk remains. machine intelligence which could be- phe, which is deter- Arkhipov, was alone In addition, ongoing economic and have in a manner incompatible with mined by the po- in refusing permis- technological developments bring, human values. 17% 3 tential damage of sion. alongside their benefits, a range of However, our governments and in- of the world’s population was the event and While the nu- new unprecedented anthropogen- stitutions, whose primary focus is killed in the initial outbreak of the its probabili- ‘Great Plague of Justinian’. That equals clear threat ic risks: for example, catastrophic understandably on more day-to-day ty of occur- 25-33 million people has receded climate change, pandemics of glob- concerns, may systematically be ne- ring, has at since the al proportions, and the potential for glecting global catastrophic risks.

20 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 21 CHAPTER 1 – AN INTRODUCTION TO GLOBAL CATASTROPHIC RISKS

1.1. Defining Global FIGURE 1.1. QUALITATIVE RISK CATEGORIES 8 Catastrophic Risk

lobal catastrophes are events None of the various humanitarian or processes that would inflict disasters of the 20th century killed G serious damage to humanity more than 10% of the world popula- on a global scale, such as all-out nu- tion. Around 1% of the world popu- clear war or a pandemic killing hun- lation died in the First World War, dreds of millions. The severity of a while up to 3% died in the Second risk is a function of its scope (the size World War.5 The most deadly event of the population at risk), intensity of the 20th century was probably the (how badly this population would be Spanish influenza pandemic of 1918- affected), and probability (how likely 1920 which killed 50 - 100 million SCOPE the disaster is to occur).4 people – between 2.5% and 5% of the A fatal car crash is a personal ca- world population.6 Although these tastrophe: a small number of unfor- were huge tragedies, our focus here tunate victims suffer a severe harm. is on even more extreme possibilities Global warming Depletion of the Most genocides are examples of local which receive less attention. In Chap- Nuclear war by 0.001 C° ozone layer catastrophes: thousands or millions ter 2 we give an overview of the ma- Global of people within a country or region jor risks of global catastrophe, and in lose their lives. The focus of this re- Chapter 3 we look at the factors which port is catastrophic risks with global may increase or decrease these risks. scope. We define a global catastrophe Limited historical evidence makes as a possible event or process that, it very difficult to provide a definitive were it to occur, would end the lives list of past global catastrophes. There Minute increase Recession in one National of approximately 10% or more of the have been at least two in the past two country genocide Local in regional air pollution global population, or do comparable millennia – the Plague of Justinian damage. Extinction risks are a sub- and the Black Death. Some scholars set of global catastrophic risks, which have argued that more than 10% of would end the human race. the world population lost their lives It is important to put the scale of in pre-industrial wars, though this is global catastrophic risks in context. heavily disputed.7 Loss of one hair Loss of one arm Fatal car crash Personal

Imperceptible Damaging Catastrophic SEVERITY

22 Global Catastrophic Risks 2016 CHAPTER 1 – AN INTRODUCTION TO GLOBAL CATASTROPHIC RISKS

1.2. Why Global The UK’s Stern Review Catastrophic Risks matter on the Economics of lthough the chance of dy- Review would imply a 9.5% chance ing in a car crash is small, we of human extinction within the next A each take steps to mitigate hundred years.12 Climate Change suggested the risk such as wearing seat belts and Reducing these risks has obvious driving safely. National governments humanitarian benefits for those alive take steps to mitigate the risk of rare today. But we should also consider natural disasters, such as earthquakes the welfare of future generations. A a 0.1% chance of human and hurricanes. Similarly, it is im- global catastrophe could reduce the portant that the global communi- standards of living for many genera- ty works to reduce the risk of cata- tions to come, while outright human extinction each year. strophic events which would have a extinction denies existence to all fu- global scope. ture generations. Many leading mor- It is easy to be misled by the appar- al philosophers have argued that the ently low probabilities of catastroph- welfare of these future generations is If this estimate is ic events. The UK’s Stern Review on of utmost importance.13 the Economics of Climate Change Global catastrophic risks are also suggested a 0.1% chance of human likely to be politically neglected.14 extinction each year, similar to some Global catastrophic risk reduction is correct, a typical person rough estimates of accidental nucle- a global public good, as even a large ar warfare.8 At first glance, this may country would only capture a small seem like an acceptable level of risk.10 portion of the total benefit of risk mit- is more than five times Moreover, small annual probabil- igation. Moreover, it is an intergen- ities compound significantly over erational public good, as many of the the long term. The annual chance of beneficiaries are future people who dying in a car accident in the Unit- have no voice in the political process. as likely to die in an ed States is 1 in 9,395. However, this For these reasons and others, nation- translates into an uncomfortably al and international actors are likely high lifetime risk of 1 in 120.11 Using to underinvest in risk reduction. the annual 0.1% figure from the Stern extinction event as in a car crash.

24 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 25 CHAPTER 1 – AN INTRODUCTION TO GLOBAL CATASTROPHIC RISKS

1.3. Why Global Catastrophic 1.4. What can be done?

hen dealing with global policy-makers, industries, research Risks are especially relevant today catastrophic risks we can- communities, and citizens can take W not generally rely on his- preemptive steps to limit global cat- ow more than ever before, most important future risks may be at torical experience or trial and error. astrophic risks. This report outlines global catastrophic risks de- present unknown. Just as in the early Given the severity of global catastro- the key features of the world’s most N serve attention. Prior to the 20th century it would have been im- phes, learning from experience would significant global catastrophic risks 20th century, the main global cata- possible to predict nuclear weapons, be extremely costly or, in the event and identifies, in Chapter 5, some strophic risks that humankind faced catastrophic climate change, or bio- of human extinction, impossible. But strategies for limiting them. were natural pandemics and conven- technology risks, it may be that many tional warfare. However, economic of the future leading global cata- and technological development have strophic risks are not yet within sight. brought a range of new anthropogen- Moreover, to reduce these new an- ic risks. thropogenic risks we may need lev- The first of these new risks was nu- els of international coordination that clear weapons, which gave states un- existing institutions are not designed precedented destructive power and to produce, something we discuss in emerged very rapidly: the bombings Chapter 4. A good illustration of this of Hiroshima and Nagasaki came only is the threat of catastrophic climate 1.5. How to read this report six years after Einstein’s letter to Roo- change. Atmospheric concentrations sevelt warning of the dangers of nu- of greenhouses gases are now at their hapter 2 gives a comprehen- Chapter 3 discusses the causes of clear fission.15 Other anthropogenic highest level for hundreds of thou- sive overview over the main global catastrophic risks. These in- risks might also mature quickly giv- sands of years,18 and if the interna- C global catastrophic risks, from clude both factors that increase the ing us little time to prepare. Advances tional community fails to take strong catastrophic climate change and nu- likelihood or impact of individu- in certain kinds of biotechnology, for action soon there is a worryingly high clear war to risks associated with al risks, as well as factors that affect example, might at some point in the chance of warming in excess of 6°C emerging technologies such as bio- multiple risks, such as poor gover- next few decades give states, or even (compared to pre-industrial levels) technology, artificial intelligence and nance. terrorist groups, the capacity to cre- by the end of the century.19 But be- geo-engineering. We discuss the po- Chapter 4 discusses why we cur- ate devastating designer pathogens.16 cause unilateral action is costly for tential impact and the likelihood of rently collectively underinvest in Likewise, experts warn of the lon- any state, and the benefits are felt by each risk, as well as the main actions global catastrophic risk. It also dis- ger-term risks associated with power- everyone regardless of their contribu- to limit them. Finally, we attempt to cusses what actors are best placed to ful machine intelligence, which may tion, action on greenhouse gas emis- compare the risks in terms of how overcome this neglect. prove hard to control safely.17 sions has been slow in coming. likely they are and in terms of how Finally, in Chapter 5 we briefly dis- Indeed, experience over the last much attention we ought to pay them cuss a number of concrete steps to re- century suggests that many of the at present. duce global catastrophic risk.20

26 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 27 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

Chapter 2 Splitting the atom What are the Most brought the promise of Important Global clean power, but also Catastrophic Risks? led to the nuclear bomb,

or most of human histo- sion. The burning of fossil fuels has which has brought ry, humanity has had to brought huge improvements in hu- contend with a relatively man welfare, but unless strong action narrow range of global cat- is taken soon, there is an unaccept- astrophic risks. Super-vol- able chance that our children and humanity to the brink of Fcanic eruptions and large asteroid im- grandchildren will face catastrophic pacts were possible but very unlikely, global warming. Rapid developments so only natural pandemics and ex- in biotechnology could enable scien- treme conventional warfare seriously tists to develop new therapies to re- catastrophe on more than threatened the complete destruction duce the global burden of disease and or permanent stagnation of human feed a growing population, but might civilisation. also in the future give malicious one occasion. Today, thanks to economic and groups the capacity to synthesise dev- technological progress, global living astating pathogens. standards have never been higher, This chapter surveys currently the but unfortunately, for the same rea- most important global catastrophic son, we face a number of new anthro- risks by examining expert scientific pogenic global catastrophic risks. opinion on the two determinants of Some of these appear to be at least as risk: potential impact and likelihood. threatening to human civilisation as There is also a brief discussion of ac- natural pandemics and convention- tions available to limit each of these al warfare. Splitting the atom brought risks, which is summarised in Chap- the promise of clean power, but also ter 5. In the final section, we give a led to the nuclear bomb, which has comparative assessment of the differ- brought humanity to the brink of ca- ent catastrophic risks. tastrophe on more than one occa-

28 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 29 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

2.1. Catastrophic climate change s a result of human activi- ty or strong positive feedback loops in ty since the Industrial Rev- the carbon cycle. For example, global A olution, atmospheric con- warming might cause the melting of centrations of Greenhouse Gases arctic permafrost, which would re- (GHGs) are now at their highest level lease substantial amounts of meth- for hundreds of thousands of years,21 ane – a potent GHG – into the atmo- which has caused global surface and sphere.22 This process could itself ocean warming. Continued increas- trigger other positive feedback loops. es in GHG emissions are very likely Catastrophic warming could also oc- to cause future warming. The eventu- cur even without these feedback ef- al level of warming depends on total fects, if climate sensitivity turns out GHG emissions and on the sensitiv- to be higher than median estimates. ity of the climate to GHG emissions. Alternatively, it might occur simply The reports of the Intergovernmental because we are less able to coordinate Panel on Climate Change (IPCC) focus internationally to reduce emissions on the most likely levels of warming than we expect. given a particular emissions path- It is impossible to say with confi- way. It is widely agreed that the neg- dence exactly what level of warming ative consequences of the most likely would bring about global catastrophe, levels of warming will be substantial. in the sense we are interested in here. However, it is important to take ac- The IPCC states: count the ‘tail risk’ of lower probabili- “Global climate change risks are ty, but potentially catastrophic, levels high to very high with global mean of warming. Of course, what we refer temperature increase of 4°C or more to in this section as ‘non-catastroph- above preindustrial levels in all rea- ic climate change’ would still have se- sons for concern, and include severe vere consequences, but it would not and widespread impacts on unique constitute a global catastrophe, on and threatened systems, substan- our definition. tial species extinction, large risks to global and regional food security, and POTENTIAL IMPACT the combination of high temperature OF THE CATASTROPHE and humidity compromising normal Increasing GHG emissions could po- human activities, including growing tentially trigger catastrophic climate food or working outdoors in some ar- change due to high climate sensitivi- eas for parts of the year. The precise

30 Global Catastrophic Risks 2016 31 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

levels of climate change sufficient to most likely to follow a medium-high trigger tipping points (thresholds for emissions scenario, and there is some Governments need to abrupt and irreversible change) re- chance we will follow a high emis- main uncertain, but the risk associ- sions scenario.27 However, the situ- ated with crossing multiple tipping ation may have changed as a result take into account lower points in the earth system or in inter- of the December 2015 Paris Agree- linked human and natural systems ment.28 increases with rising temperature.”23 In estimating the impact of differ- The latest IPCC report focuses on ent emissions scenarios, the IPCC probability but extreme the impacts of warming of 1°C to 4°C does not set out the probability of all above pre-industrial levels, and sug- possible levels of warming, but in- gests that the impacts corresponding stead only sets out the likely range of to more extreme levels of warming warming, where ‘likely’ is defined as levels of warming. Indeed, are relatively unstudied.24 However, having a greater than 66% chance of it is likely that damages increase sig- occurring.29 However, in order to pri- nificantly at higher temperatures,25 oritise resources effectively govern- these may constitute and possible that warming of 6°C or ments need to take into account the more above pre-industrial levels may whole probability distribution, in- be catastrophic. Warming of this mag- cluding lower probability but extreme nitude is, for example, likely to ren- levels of warming. Indeed, these may the majority of the der most of the tropics substantially constitute the majority of the expect- less habitable than at present.26 ed (probability-weighted) costs of cli- mate change. LIKELIHOOD OF Some scholars have provided esti- probability-weighted THE CATASTROPHE mates of the probability of extreme The probability of catastrophic cli- warming. The economists Ger- mate change depends on the lev- not Wagner and Martin Weitz- costs of climate change. el of GHGs in the atmosphere man have inferred estimates and on the sensitivity of the of the probability of warming climate to cumulative GHGs. of more than 6°C relative to According to a report by pre-industrial levels from King et al for the Cen- the IPCC figures. They tre for Science Policy argue that even on a at the University of low-medium emis- Cambridge, if ma- sions scenario, there jor countries and $40 is at least a 3% regions contin- per tonne chance of even- ue with cur- Is the recommended global price tual 6°C warm- rent plans and of carbon by some leading economists. ing (with projects, the Current price is significantly less. significant world is uncertain-

32 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 33 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

ty). On the medium-high emissions approximately linear relationship FIGURE 2.1. THE CHANCE OF EXTREME CLIMATE CHANGE scenario, the chance could be around between the total amount of carbon The probability of warming of 6°C for different atmospheric concentrations of greenhouse gases.39 10%.30 These figures are of course emitted and the resulting tempera- speculative, but they do provide some ture increase.34 The majority of the reason to believe that the probabili- carbon still underground is in the ty of catastrophic climate change is form of coal.35 If we were to avoid non-negligible, unless strong action is burning the remaining global coal taken on GHG emissions. reserves we would likely avoid cata- strophic levels of climate change.36 MAIN ACTIONS AVAILABLE States could commit to building no TO LIMIT THE RISK new coal-fired power stations without Probability of warming >6°C There are three main ways to reduce carbon capture and sequestration to Percent the risks from climate change: adap- limit the fraction of the global coal re- tation to climate change, abatement serves which are burned. 20 of GHG emissions, and geo-engineer- Geo-engineering – the deliberate ing. It is likely to be very costly if not use of technology to alter the world’s impossible to to avoid many of the climate – in the form of Carbon Di- impacts of non-catastrophic climate oxide Removal (CDR) or Solar Radia- change by adaptation alone,31 and tion Management (SRM), could also adapting to warming of 6°C or more is help to reduce the risk of catastroph- 15 likely to be even more costly and dif- ic climate change, as a complement ficult. to GHG abatement. CDR techniques, Turning to abatement, most econ- such as carbon sequestration or iron omists agree that the best way to re- fertilisation of the oceans, would re- duce GHG emissions is to impose a move CO2 from the atmosphere and carbon tax or a cap and trade sys- thereby help us move towards net 10 tem.32 At present, the global price of neutral or net negative emissions.37 carbon is approximately $4 per tonne, SRM techniques, such as the injection whereas according to Wagner and of sulphates into the stratosphere, Weitzman, to fully price in the exter- cause global cooling by reflecting nalities from catastrophic climate sunlight. The benefits and risks of change, a price of at least $40 could geo-engineering are discussed in 5 be required.33 more detail in section 2.5.38 Scientists have demonstrated an

0 400 450 500 550 600 650 700 750 800 CO2e concentration Parts per million

34 Global Catastrophic Risks 2016 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

2.2. Nuclear war

he invention of nuclear weap- Russia has 4,300, with yields ranging ons gave humanity the tech- from 50 to 800 kilotons.42 For compar- T nical capacity to cause dev- ison, the Little Boy bomb dropped on astation on a hitherto unseen scale. Hiroshima had a 15 kiloton yield.44 Although there have been no nucle- The damage from nuclear war can ar attacks since the Second World be divided into two main categories. War, we have come close to inadver- Firstly, there is the damage from the tent and intentional nuclear war on blast, fire and radiation. A 1979 re- a number of occasions. A worrying port by the U.S. Office of Technolo- possible consequence of nuclear war gy Assessment estimated that, in an is a nuclear winter with global climat- all-out war between the US and Rus- ic implications. While the chance of sia involving thousands of nuclear nuclear war may appear to have de- weapons, 35-77% of the U.S. popula- clined since the end of the Cold War, tion and 20-40% of the Soviet popula- tensions between nuclear states per- tion would die within the first 30 days sist. Reducing the likelihood of nu- of the attack, and millions would die clear war is a serious ongoing global globally in the following years due to challenge. the radioactive dust cloud.45 The pro- Despite these risks, some have ar- portionate death toll today is likely to gued that the deterrence effects of be lower because nuclear arsenals at the nuclear bomb have been, and will that time were five times larger.46 be, very valuable in terms of ensur- The second category of damage is ing global peace.40 How to make the a possible nuclear winter, which is trade-off between lower likelihood caused by the burning of cities, in- of nuclear or conventional conflict is dustrial facilities and other flamma- unclear. ble materials, sending smoke into the atmosphere. Scientists have applied POTENTIAL IMPACT modern climate models to predict the OF THE CATASTROPHE scale of nuclear winter, though these The scale of the damage done by nu- predictions are uncertain. According clear war obviously depends on the to one model, an all-out exchange of scale of the nuclear war itself. In 2014, 4,000 nuclear weapons would release there were 9,920 stockpiled nuclear 150 teragrams of smoke, leading to a warheads in the world (down from a 8°C fall in global temperature.47 Due peak of 65,000 in 1986).41 The United to the fall in temperature and the loss States has 4,760, with yields ranging of sunlight and growing food would from 5 kilotons to 455 kilotons;42 and be virtually impossible for 4-5 years,

36 Global Catastrophic Risks 2016 37 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

5! FIGURE 2.2.1. NUCLEAR ARSENALS OF THE US, RUSSIA, creating an unprecedented famine. between one out of three and even”. 59 According to some models, even a Although tensions have decreased AND THE WORLD FROM 1945 UNTIL TODAY smaller scale regional war between since the end of the Cold War, in India and Pakistan involving fifty 15 the wake of the crisis in Ukraine the kiloton weapons would cause global chance of confrontation has risen,52 temperatures to fall by around 1.25°C and the geopolitical situation could Number of nuclear warheads in the first year. Some have suggested become more unstable over the next Thousands that this could disrupt agriculture so few decades. significantly that one billion people According to many experts, the 80 would be at risk of starvation, though most likely intentional nuclear war this has been criticised as overly pes- is between India and Pakistan. In- simistic.48 dia and Pakistan have had numerous wars in the past and there have been 70 Russia LIKELIHOOD OF THE CATASTROPHE various terrorist attacks against India It is very difficult to estimate the by Pakistani groups.53 Pakistan has probability of a nuclear war in the pledged to meet any Indian attack on US next century. However, the chance its territory with a retaliatory nuclear 60 may be too high to ignore. Over the strike.54 course of the nuclear age, we have Global come close to inadvertent nuclear war MAIN ACTIONS AVAILABLE 50 on numerous occasions. For exam- TO LIMIT THE RISK ple, in 1995 Russian systems mistook There is some disagreement about a Norwegian weather rocket for a po- how to reduce the risk of nuclear war. tential nuclear attack. Russian Pres- One policy option favoured by many 40 ident Boris Yeltsin retrieved launch is continued reduction of global nu- codes and had the nuclear suitcase clear arsenals.55 There has been a open in front of him. Thankfully, Rus- great deal of progress on this front: sian leaders decided the incident was in the 1960s 23 countries had weap- 30 a false alarm.49 In a 2013 paper Barrett ons or were pursuing programmes, et al estimated that the 90% confi- whereas today only nine countries dence interval of the annual probabil- have weapons.56 However, reduction ity of accidental nuclear war between of nuclear arsenals does not guaran- 20 the US and Russia is from 0.001% to tee safety.57 7%.50 This covers a significant range, Weapons systems could also be al- but a substantial portion of the possi- tered to reduce the risk of acciden- bilities are uncomfortably high. tal and intentional war. For instance, 10 We have also come close to inten- decision times could be increased tional nuclear war on many occa- by each side locating their weapons sions. President Kennedy said that further from each other’s borders.58 0 the chance of nuclear war during the More broadly, improved internation- Cuban Missile Crisis was “somewhere al relations would help to decrease 1945 1950 1960 1970 1980 1990 2000 2010 2014 Year

38 Global Catastrophic Risks 2016 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

FIGURE 2.2.2. GLOBAL TEMPERATURE ANOMALY FROM NUCLEAR WINTER NUCLEAR NEAR MISSES Global average surface air temperature change from a release of 5 Teragram, 50 Teragram, and 150 Teragram of particular matter in the context of the climate change of the past 125 years.61 • In 1973, when Israel encircled the Egyptian Third Army, the Soviets threatened to intervene, leading to implied nuclear threats. • In September 1983, a Soviet early warning satellite showed that the United States had launched five land-based missiles at the Soviet Temperature anomaly Union. The Soviet officer on duty, Stanislav Petrov, had only minutes °C +5 Tg +50 Tg +150 Tg to decide whether or not the satellite data were a false alarm. Since the satellite was found to be operating properly, following procedures 1 would have led him to report an incoming attack. Going partly on gut instinct and believing the United States was unlikely to fire only five missiles, he told his commanders that it was a false alarm before he knew that to be true. Later investigations revealed that reflection of 0 the sun on the tops of clouds had fooled the satellite into thinking it was detecting missile launches. • The Able Archer incident of November 1983 was, in the words of -1 former US Secretary of Defense Robert Gates, “one of the potential- ly most dangerous episodes of the Cold War”. With talk of fighting and winning a nuclear war emanating from Washington, the Soviets -2 reasoned that the West would mask preparations for a nuclear attack as a military exercise. The Able Archer exercise simulated the coordi- nated release of all NATO nuclear weapons. In response, the Soviets readied their nuclear forces and placed air units in East Germany and -3 Poland on alert.

There have been numerous other examples of nuclear near misses. -4

-5

-6 the likelihood of international nu- could work to ensure that terrorists -7 clear conflict. Research into nucle- cannot acquire nuclear weapons by ar winter would improve our knowl- theft or on illegal markets. Glob- edge and allow nuclear states to al enforceable standards for nuclear change their strategy accordingly. weapon security would help achieve -8 Finally, state and non-state actors this goal.60 1880 1890 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 2010 Year

40 Global Catastrophic Risks 2016 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

2.3. Natural pandemics

ntil the discovery of nucle- H5N1 would have to be easily transmis- ar weapons, pandemics were sible between humans, which is not U the most important global currently the case, though occasionally catastrophic risks, and it is plausible humans become infected through ani- they still are. Plague, HIV, smallpox mal vectors. Influenza pandemics have and other diseases have killed mil- previously infected about 24-38% of the lions upon millions of people. There world population.65 The case fatality of remains a serious chance that a pan- a novel strain of H5N1 is unpredictable, demic could kill a huge portion of the but estimates for the H5N1 case fatality world population, with pandemic in- rate until today vary particularly wide- fluenza the most serious threat. A va- ly – from 1% to 60%.66 A very rough es- riety of measures, including improv- timate of the death toll of a H5N1 pan- ing international coordination and demic can be found by multiplying data sharing, and increasing the pro- the usual pandemic attack rate (24% to duction of drugs and vaccines, would 38%) by the global population (rough- help to reduce this risk. ly 7 billion) times the case-fatality ratio (1% to 60%). This produces an estimat- POTENTIAL IMPACT ed death toll of between 16.8 million OF THE CATASTROPHE and 1.7 billion fatalities. The develop- The Black Death plague in the 14th ing world would probably bear an over- century killed between 11% and 17% whelming part of this burden.67 of the global population over the course of a decade.62 In the most dev- LIKELIHOOD OF THE CATASTROPHE astating pandemic of the last 100 Influenza pandemics occur relatively years, the 1918 Spanish influenza frequently: there have been ten in the outbreak, 50-100 million people lost last 300 years, though none of these their lives – between 2.5 and 5% of have killed more than 5% of the world the global population.63 Both of these population.68 It is difficult to know pandemics occurred before the age where and when the next natural pan- of modern medicine, so it is unlike- demic will occur. Globalisation and ly these diseases would have simi- increased interaction between hu- lar impact today, although our more mans and animals increase the risk of connected society may increase the pandemics, but improved health and spread of pandemics. sanitation would lessen their impact. Currently, H5N1 avian influenza is According to the UK National Risk thought to be the greatest pandemic Register there is between a 1 in 20 and threat.64 To develop into a pandemic a 1 in 2 chance of a pandemic killing up

42 Global Catastrophic Risks 2016 43 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

76 to 1% of the UK population in the next bodies such as the WHO can stockpile FIGURE 2.3.1. HISTORIC PLAGUES AND PANDEMICS five years. Such a pandemic would go drugs and vaccines to counter disease on to have global effects. Global cata- outbreaks. However, stockpiles may strophic risk-level pandemics (killing only be effective for some but not all more than 10% of the global popula- pandemic pathogens.72 Thirdly, since tion) are, however, further down the developing countries will probably tail of the probability distribution. Re- face the highest burden from future sults from expert surveys have put a pandemics, continued improvements 15% chance of an H5N1 pandemic over in developing world health systems a three year period.69 in accordance with the WHO’s Inter- national Health Regulations, and im- MAIN ACTIONS AVAILABLE provements in the global distribution TO LIMIT THE RISK of drugs and vaccines, would limit the There are numerous ways to limit the risk from pandemics.73 Fourthly, dis- risk from pandemics.70 Firstly, pharma- ease surveillance and response sys- ceutical companies could focus atten- tems could be improved so that nov- tion on developing vaccines and drugs el threats in both animal and human for very damaging pandemics, and the populations are detected and respond- international community can remove ed to quickly.74 Rapid dissemination barriers to development with measures of relevant data between countries is such as the establishment of a glob- also essential for effective outbreak al vaccine development fund.71 Sec- response, because of the exponential ondly, governments and international spread of infection in an outbreak.75

PANDEMIC DATE(S) LETHAL IMPACT

Plague of Justinian AD 541-542 25-33m (13-17% of the world population)

Black Death 14th Century 50-75m (11-17% of world population)

Smallpox 1520-1527 200,000 deaths within the Aztec population (75% of population in some areas)

Spanish influenza 1918 - 1919 50 - 100m (2.5-5% of population)

Smallpox 20th century 300m over the course of the 20th Century

HIV/AIDS 1981 - present 34m

44 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 45 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

2.4. Exogenous risks

xogenous risks are those which est mass extinction in history and re- arise independent of human duced the human ancestor popula- E activity (whereas even natu- tion from around 100,000 to around ral epidemics are spread by humans). 4,000 people for approximately They include such possibilities as 20,000 years, though this is contro- super-volcanic eruptions and large versial.81 According to a report by the asteroid and comet impacts, which Geological Society of London: are believed by some to have caused “A layer of ash estimated at 15 cen- mass extinctions.77 The likelihood of timetres thick fell over the entire exogenous risks is better understood Indian sub-continent, with similar than that of many other global cata- amounts over much of SE Asia. Just strophic risks because the underlying one centimetre of ash is enough to dynamics have been unchanged for devastate agricultural activity, at least a very long time. The historical evi- when it falls in the growing season. dence suggests that exogenous global An eruption of this size would have catastrophic risks cannot be too fre- catastrophic consequences. Many quent, and may therefore be much millions of lives throughout most less likely than some of the anthropo- of Asia would be threatened if Toba genic risks.78 erupted today.”82

A. SUPER-VOLCANOES LIKELIHOOD OF THE CATASTROPHE Super-volcanoes are volcanoes ca- In order to assess the likelihood of su- pable of producing at least 1012 m3 per-volcano eruptions, we have to rely bulk volume of fragmental material.79 on a relatively limited set of obser- Some experts believe that the erup- vations of past super-volcanic erup- tion of the Toba super-volcano in tions, which makes any estimates Indonesia around 70,000 years ago very uncertain. Existing data sug- brought humanity to the brink of ex- gest that there will be a super-volca- tinction, though there is significant nic eruption roughly every 30,000- disagreement about this. 50,000 years on average.83

POTENTIAL IMPACT OF THE RISK MAIN ACTIONS AVAILABLE The Toba eruption ejected large TO LIMIT THE RISK amounts of dust and sulphates into At present, humanity lacks the tech- the Earth’s atmosphere, which caused nical capacity to prevent volcanic global cooling of 3-5°C for sever- eruptions. Consequently, improving al years and led to enormous loss of resilience to catastrophe is the main plant and animal life.80 Some have way to limit the risk from super-vol- argued that Toba caused the great- canoes. We discuss this in more detail

46 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 47 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

in Chapter 3. Increased investment being released into the atmosphere.86 FIGURE 2.4. RISK BEFORE AND AFTER THE SPACEGUARD SURVEY 92 and research might also improve our The damage that would be caused by ability to predict volcanic eruptions, a multi-kilometre asteroid or comet which would in turn improve pre- impact is only modestly understood paredness.84 at present.87 An asteroid winter would undermine agriculture at least for an B. ASTEROIDS entire growing season and so could AND COMETS cause the deaths of billions of peo- Around 66 million years ago, an as- ple.88 According to a comprehensive teroid of around 10km in diameter report by the US National Research struck Chicxulub in Mexico. This im- Council, “above the conservatively pact probably caused one of the three assumed global catastrophe thresh- largest mass extinctions in history old from a 1.5km-diameter impac- and may have abruptly ended the age tor, the number of fatalities ramps up of the dinosaurs.85 Today, an impact from 10% of the world’s population to by a Near Earth Object (NEO) – an as- the entire population for impactors Statistical fatalities per year teroid or comet – larger than 1.5km in above 10km in diameter”.89 diameter would kill millions largely 250 by causing global cooling and agricul- LIKELIHOOD OF THE CATASTROPHE tural disruption. The likelihood of an NEOs are a comparatively well-un- Intrinsic risk before survey NEO impact is reasonably well-un- derstood global catastrophic risk. NA- derstood and important steps have SA’s Spaceguard system has mapped 200 Residual risk after 10 years of surveying been taken to monitor the risk they more than 90% of asteroids with a di- pose. ameter of more than 1km.90 Reinhardt et al argue that the total probabili- POTENTIAL IMPACT ty in a 100-year period of an aster- 150 OF THE RISK oid encounter with the Earth that Contemporary studies of aster- might cause a globally signifi- oids and comets conservative- cant effect is approximately ly assume that all objects 1 in 1,250.91 100 greater than 1.5km in di- One can calculate the ex- ameter would be capa- 1 pected cost of NEOs by ble of causing damage in multiplying the like- on a global scale, via ly deaths per event 50 firestorms generated by the frequency of by impact debris 1,250 events of a cer- and a so-called tain size. As of approximate risk of an asteroid asteroid win- 2010, the annu- encounter with the Earth that might 0 ter caused by cause a globally significant effect al expected 0,01 0,1 1 10 dust and in a 100-year period (probabili- sulphates ty-weight- Diameter of impactor Kilometres

48 Global Catastrophic Risks 2016 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

“The likelihood of an NEO impact is reasonably well-under- stood and important steps have been taken to monitor the risk they pose.”

ed) cost of all NEOs is currently 91 fa- detection of asteroids would enable talities.93 humanity to understand the risks it Once ongoing asteroid surveys faces and to take appropriate count- are completed, long-period comets er-measures. NASA’s ongoing Space- – comets which take more than 200 guard Survey has so far reduced the years to orbit the Sun – may domi- expected cost a large asteroid impact nate the remaining unknown impact by more than an order of magnitude.95 threat from NEOs.94 Secondly, technological research could help us to find ways to deflect threat- MAIN ACTIONS AVAILABLE ening NEOs.96 Thirdly, improvements TO LIMIT THE RISK in resilience could be made to ensure There are three main ways to limit the human survival in the event of a large risk from asteroids. Firstly, improved asteroid or comet impact.

50 Global Catastrophic Risks 2016 51 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

2.5. Emerging risks

n the coming decades, emerging come increasingly streamlined, the technologies will provide major tools enabling such risks may become I benefits to society, but they may widely accessible.99 Previously benign also create significant and unprece- digital information, such as the wide- dented risks. Certain types of bio- ly available online genetic data for technology, if more widely accessi- smallpox, will become more hazard- ble, could give terrorist groups the ous. access to pathogens as dangerous as smallpox. Geoengineering technol- POTENTIAL IMPACT OF THE RISK ogies could give single countries the Evolutionary pressure generally con- power to unilaterally alter the earth’s strains the lethality of pathogens, as climate. Finally, artificial general in- most highly lethal pathogens fail to telligence could, if developed, leave spread far before killing their host. human control. There is thus some evidence for an in- Technological risks could emerge verse relationship between a patho- very quickly and give certain groups gen’s lethality and transmissibility, large and perhaps unprecedented thereby limiting the damage from a destructive power. Moreover, some naturally occurring pandemic. Bio- emerging technologies may be partic- technology has the potential to break ularly difficult to control because bar- this correlation, allowing organisms riers to access the technology may be with extraordinarily high lethality quite low. and transmissibility. In 2001 Austra- lian researchers accidentally created A. ENGINEERED a highly lethal and vaccine resistant PANDEMIC form of mousepox.100 Similar tech- The past decades have seen rapid ad- niques could potentially be applied vances in biotechnology, in part due to smallpox.101 Two recent controver- to the falling costs of gene sequenc- sial papers have shown how to create ing and synthesis.97 Improvements in a version of H5N1 which is potential- ease-of-use of certain specific kinds ly transmissible between humans.102 of biotechnology bring increased con- Basic calculations based on H5N1 pa- cerns about bioterrorism. Gene syn- rameters suggest that a single release thesisers have the capacity to turn of these modified viruses could cause digital sequence data into physical hundreds of millions of casualties.103 genetic sequences, enabling individ- Engineered pathogens with danger- uals to create viruses from digital files ous features could be released acci- (as was done with the 1918 Spanish dentally from a lab or intentionally Flu virus).98 Should gene synthesis be- by states or terrorist groups. The legal

52 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 53 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

theorist Richard Posner has described As biotechnology develops, the FIGURE 2.5.1. COST PER GENOME 116 a scenario in which terrorists synthe- level of expertise required to create sise a version of smallpox that is in- dangerous pathogens will fall. In the curable, immune to vaccine and kills longer term, if biotechnology ma- all its victims, which is then released tures sufficiently and gene synthesis in a large city via aerosol.104 Although is not well-regulated, states and small such scenarios are not currently feasi- groups will find it increasingly easy to ble, eminent figures including George synthesise and alter dangerous patho- Church,105 Nathan Myhrvold,106 and gens.112 This poses a serious risk of a Martin Rees107 have all argued that global catastrophe. biotechnology poses serious risks. MAIN ACTIONS AVAILABLE LIKELIHOOD OF THE CATASTROPHE TO LIMIT THE RISK There is a real possibility that a dan- Many of the measures which limit the gerous engineered pathogen could risk from natural pandemics would be released by accident. The H1N1 also limit the risk from synthesised or influenza strain, responsible for sig- modified pathogens. However, other Cost of sequencing a human genome nificant morbidity and mortality measures would help specifically with Dollars (logarithmic scale) around the world from 1977 to 2009, is biotechnology risks. Fostering a cul- thought to have originated from a lab- ture of safety in the relevant research 100M oratory accident.108 As of 2012, there and technical communities is proba- were at the very least 42 laboratories bly very valuable. Researchers could engaged in live research on potential be encouraged to take significant pre- pandemic pathogens, though this will cautions with this research and to 10M Moore’s law equivalent have been affected by the US mora- avoid disseminating research when torium on ‘gain of function’ research doing so brings major risks. With re- in 2014.109 Lipsitch and Inglesby esti- gard to regulation, a licensing regime 1M mate that “work with a novel, trans- for DNA synthesis could be one first missible form of influenza virus car- step,113 while mandatory liability in- ries a risk of between 0.01% and 0.1% surance for dual-use research - which per laboratory-year of creating a pan- helps medical progress but could be 100K demic... or between 0.05% and 0.6% used maliciously - would ensure that per full-time worker-year”.110 These researchers have incentives to main- estimates are illustrative of the prob- tain high levels of laboratory biosafe- ability that existing ‘gain of function’ ty.114 Governments could also require 10K influenza research would produce a journal editors to consider whether pandemic, though there is significant publication of research could lead to uncertainty about them.111 adverse outcomes.115 1K 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Year

54 Global Catastrophic Risks 2016 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

B. ARTIFICIAL influence over the future. It has been INTELLIGENCE argued that if these goals were not Prominent figures uch as Stuart Rus- aligned with human values, the conse- sell, Professor of Computer Science at quences could be truly catastrophic.121 Berkeley, Peter Norvig, Director of Re- It is difficult not only to specify human search at Google, and Nick Bostrom, values in a robust, machine-interpre- author of Superintelligence, recent- table way, but also to agree on human ly signed a letter warning of the risks values in the first place. Moreover, if posed by Artificially Intelligent (AI) even if the values of very powerful ar- systems.117 Past experience demon- tificially intelligent systems can be strates that AI systems can go from aligned with their creators’, such sys- significantly subhuman to superhu- tems might destabilise the geopolitical man relatively quickly in narrow do- balance in a destructive way. mains, such as recently in the game Go. In the coming decades, we may LIKELIHOOD OF THE CATASTROPHE create AI systems which surpass hu- It is widely accepted that we will be mans in all relevant domains. If this able to create AI systems that are ca- were to happen, the effects would be pable of performing most tasks as uniquely transformative. However, well as a human at some point (‘hu- AI also has huge potential benefits, man-level AI’). Experts disagree as well as risks.118 It could, for exam- about when this will occur. According ple, greatly reduce the cost of many to the median surveyed expert, there goods, and allow us to solve other is a roughly 50% chance of such AI global problems. by 2050.122 The median surveyed ex- pert believes that there is at least a 5% POTENTIAL IMPACT OF THE RISK chance of superintelligent AI within Some risks of AI likely fall short of two years after human-level AI, and posing a global catastrophic risk. a 50% chance within thirty years.123 Widespread automation could cause Assuming that human-level AI is de- significant economic and social dis- veloped, its long-term social impact ruption, which has only a relative- is unclear. According to the medi- ly small chance of leading deaths at an surveyed expert, there is around a the scale of other global catastrophic 7% chance that it would be ‘extreme- risks.119 ly bad’.124 The extreme uncertainty In the longer term, AI may enable surrounding these estimates should, important new capabilities, perhaps however, be strongly emphasised. extremely quickly if it turns out that we can automate AI development.120 If we MAIN ACTIONS AVAILABLE have powerful and generalisable auto- TO LIMIT THE RISK mated systems, the goals they are pro- Different challenges from AI systems grammed with may exert significant will require different responses, and

56 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

2.5.2. GLOBAL TEMPERATURE ANOMALY FROM GEO-ENGINEERING FOLLOWED BY TERMINATION in many cases it will be more appro- using iron or urea could pose major Global average surface air temperature change from business as usual emissions, injection of 3 mega- 130 priate to respond closer to the time risks to marine ecosystems.125 How- tons/annum (Mt/a) of SO2 in the Arctic, 5 Mt/a of SO2 in the tropics, and 10 Mt/a SO2 in the tropics. the technologies mature. However, ever, most forms of SRM are thought there is a strong case for early work to carry much greater risks than most 126 +Anthro Forcing +3 Mt/a Arctic preparing for possibly rapid changes forms of CDR, and worries about ci- Temperature anomaly brought about by AI systems because vilisation-threatening consequences °C +5 Mt/a Tropical +10 Mt/a Tropical a reactive response won’t succeed have generally focused on SRM (and 1.4 when the change is very rapid. In par- in particular on currently the leading Geoengineering Geoengineering ticular, important research could be form of SRM: the injection of sulphate 1.3 starts ends done on how to give AI systems de- particles into the stratosphere).127 The sirable goals. Foresight work could be remainder of this section will there- 1.2 done to better understand which oth- fore focus on SRM only. 1.1 er issues require advance preparation, SRM is the only known technique or where there may be an eventual for quickly stopping (or even revers- 1.0 role for other responses. Work could ing) the rise in global temperatures. also be done to encourage under- This means that it could be used as 0.9 standing of the risks among AI devel- a complement to GHG reduction, to opers, especially around automated manage temperatures while the world 0.8 AI system development, which might phases out fossil fuels. Some have 0.7 enable very fast transitions. However, proposed that SRM could provide in- the benefits of AI could be very great surance against a ‘climate emergen- 0.6 so it is important not to unnecessarily cy’, such as unexpected abrupt and impede AI development. extreme warming.128 Thus, on some 0.5 scenarios, failing to use SRM could 0.4 C. GEOENGINEERING constitute a global catastrophic risk. As mentioned previously in this chap- Moreover, SRM has the potential to 0.3 ter, geo-engineering – in the form of reduce the costs of warming at very Carbon Dioxide Removal (CDR) or low cost. Some estimate that the an- 0.2 Solar Radiation Management (SRM) nual cost of stratospheric aerosols 0.1 – could help to limit the risks of cata- could be less than $10 billion per year, strophic climate change. CDR tech- which is orders of magnitude less -0.0 niques, such as ocean fertilisation or than the costs of climate change miti- carbon sequestration, remove Car- gation strategies.129 -0.1 bon Dioxide from the atmosphere, -0.2 whereas SRM techniques, such as POTENTIAL IMPACT OF THE RISK cloud brightening or the injection Four main arguments have been giv- -0.3 of sulphates or other particles into en for the view that SRM brings global the stratosphere, reflect the sun’s catastrophic risks. Firstly, while ex- -0.4 light and heat back into space. Cer- isting models suggest that SRM could -0.5 tain forms of CDR could carry major reduce the catastrophic effects of cli- risks. For example, ocean fertilisation mate change and will not bring their 1880 1890 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 2010 2020 2030 2040 Year

58 Global Catastrophic Risks 2016 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

own catastrophic impacts, it may nev- it could discourage GHG reduction ertheless bring currently unknown efforts.136 All major reports studying risks, particularly through impacts SRM have concluded that it does not on global precipitation.131 The climate present an alternative to emissions re- system is imperfectly understood and ductions, as it only masks the effects the deployment of a novel technol- of GHGs and does nothing to count- ogy with global effects is inherent- er ocean acidification.137 Therefore, it ly risky. Whether the use of SRM is could be problematic if geo-engineer- more risky than allowing the planet to ing drew a disproportionate amount warm is one of the key questions for of policy attention. However, it is im- anyone considering using SRM. Sec- portant to bear in mind that there are ondly, because SRM is so cheap, in- countervailing reasons in favour of dividual states could feasibly deploy SRM research and advocacy. it and unilaterally bring about global climatic impacts. Moreover, individ- LIKELIHOOD OF THE CATASTROPHE ual states might also have the incen- The probability that there will be a tive to do this because they could be global catastrophe brought about by particularly badly affected by climate SRM depends on many factors, in- change.132 Individual states acting cluding the timing, speed and severi- alone may be less likely to properly ty of global warming; the state of SRM take into account the interests of oth- technology and our knowledge of the er states and may be concerned about climate system; the response of the catastrophic consequences in other climate system to SRM; the form of regions. SRM deployed and how it is used; and Thirdly, sudden termination of SRM how well the world does at organising would lead to rapid and severe global governance of SRM. There is obvious- warming.133 There are some reasons ly very large uncertainty about all of to think that an SRM system could be these factors. very resilient against external shocks We argued above that unless strong and termination. If the current de- action is taken soon, there is a size- ployer were to suddenly stop SRM able chance of catastrophic warming. for some reason, every other country Catastrophic warming would create would have strong incentives to re- very strong incentives to use SRM. sume SRM.134 Thus, sudden termina- Therefore, unless strong GHG reduc- tion might only be likely in the event tion action is taken soon, the chance of a severe global catastrophe which that SRM is used will increase. The undermines the capacity of all coun- greatest chance of catastrophe prob- tries to use SRM.135 ably comes from poorly planned and Finally, research into geo-engineer- governed the use of SRM, perhaps ing or advocacy for geo-engineering by an individual state or small group could present a ‘moral hazard’ in that of states. Since most of the risks of a

60 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 61 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

well-governed and well-planned form ant for policymakers to be aware of of SRM are unknown, it is very dif- these issues. In the coming decades, ficult to say how likely that form of SRM is to produce a global catastro- THE CHALLENGE OF phe. EMERGING RISKS emerging technologies Finally, it is unclear the impact re- The catastrophic risks from emerging search into, advocacy for, or prepara- technology are particularly challeng- tion for, geo-engineering would have ing. Firstly, we have no track record on global willingness to cut green- of dealing with these emerging tech- will provide major house gas emissions. Therefore, it is nological risks and, as we discuss in not clear to what extent geo-engineer- chapter 4, existing national and inter- ing is a moral hazard. national institutions are not designed to deal with them. It is therefore less benefits to society, MAIN ACTIONS AVAILABLE likely that our eventual responses will TO LIMIT THE RISK be effective. The main way to reduce the risks Secondly, these technologies might, but they may also create from SRM would be to take strong like nuclear weapons, reach maturi- action on GHGs to reduce the ex- ty more quickly than expected. If so, pected costs of climate change and the nature of the problem would only thereby reduce the incentives to become fully apparent over a short significant and unprece- geo-engineer. Further research into time-frame and it would be difficult the different kinds of SRM and into to make an appropriate response. the response of the climate to them Thirdly, some of these technologies would reduce the unknown risks of could be harder to control than nucle- dented risks. SRM. Finally, working to develop ar weapons. Nuclear weapons require geo-engineering governance through the rare and controllable resources climate treaties and through global of uranium-235 or plutonium-239. In institutions might limit the risks of contrast, if some of these technolo- unilateral SRM.138 However, both of gies reach full maturity, they could be these actions might also have moral accessible to small countries or even hazards. We cannot settle the moral terrorist groups. hazard debate here, but it is import-

62 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 63 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

2.6. Other risks and unknown risks

here is a broad spectrum of ening global catastrophic risks were possible global catastroph- unforeseeable a few decades before T ic risks, from very salient and they became apparent. Forty years well-understood threats to those before the discovery of the nuclear which are extremely low probabili- bomb, few could have predicted that ty and highly speculative. In compil- nuclear weapons would come to be ing this report, we had to make deci- one of the leading global catastroph- sions about which risks to investigate ic risks. Immediately after the Second carefully, and which to ignore. Our World War, few could have known guiding principle was to include risks that catastrophic climate change, bio- which could clearly cause a glob- technology, and artificial intelligence al catastrophe in our definition if would come to pose such a significant they were to happen, which could be threat. speculative but not too speculative, These risks emerged due to rapid and which were not too low in prob- economic and technological devel- ability. opment, which looks set to contin- There are several risks we didn’t ue apace in the coming century. That include for these reasons. For exam- might create a number of new risks. ple, gamma ray bursts are fairly well Therefore, it seems likely that some understood and would be catastroph- future global catastrophic risks are at ic, but have a very low annual likeli- present unknown. hood.139 A global totalitarian state is Detailed planning for unknown a possible scenario, but it is unclear risks is of course impossible, but what proportion of the scenarios steps can be taken to improve our would meet the threshold for a global preparedness. Bodies specifically catastrophe and how likely it would tasked with horizon scanning and the be to arise given the current geopoliti- discovery of new global catastroph- cal environment.140 Conventional and ic risks would give the international chemical warfare are real threats, but community more time to craft an ap- quite unlikely to reach the scale of a propriate response. Measures can also global catastrophe. be taken to improve general societal It is important to remember, howev- resilience to catastrophe. These are er, that nearly all of the most threat- discussed in chapter 5.

64 Global Catastrophic Risks 2016 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

2.7. Our assessment Bodies specifically of the risks tasked with horizon

e’ve given qualitative de- two dimensions are linked, but they scriptions of each of these can come apart, as in the case of cli- scanning and the W risks. It is useful to con- mate change whose main harms are sider comparisons between them as not likely to be felt soon, but which well. Unfortunately there is a great demands action today. We think that deal of uncertainty around the level this split should be explicit. discovery of new global of many of the risks, and around how To assess current risk, we consid- much can usefully be done. Nonethe- ered the likelihood over the next five less, some attempt must be made in years. Our upper band consists of catastrophic risks would order to prioritise the world’s scarce events which we consider to be dis- resources, and there are some areas tinct possibilities. The lower band where we can be more confident than consists of risks which appear low in others. absolute likelihood, either because give the international In this section, therefore, we make the base rate is low enough or because some comparative assessments of the five years is too short a timescale for risks. Because of the high level of un- the risk to develop. Because of the certainty we categorise the risks into large scale of the potential catastro- community more time broad bands rather than attempt pre- phes, even risks in the lower category cise assessment. The categorisation may still be significant over this time remains our subjective judgement, period. and it is possible that some risks It is even harder to assess how large to craft an appropriate should be in different bands. But we the response should be. We consid- believe that it is better to offer even ered how much risk was posed, both such an imperfect assessment than at present and in years to come. We response. nothing. also considered how good the oppor- tunities to reduce the risk appear, and A. OUR METHOD for emerging risks how much they We have assessed the risks on two dif- benefit from early responses. Our up- ferent dimensions. First, the current per band indicates risks where we likelihood: how likely is it to materi- think a significant global response is alise in the next few years? Second, likely appropriate, perhaps involv- how much work should be given to re- ing thousands of people, or billions of ducing the risk in the next few years dollars. There is a wide range with- (according to our judgement)? These in this band, and we do not think all

66 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 67 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

of the risks there should receive the have decreased since the Cold War, FIGURE 2.7. OUR ASSESSMENT OF GLOBAL CATASTROPHIC RISKS same amount of attention. For risks nuclear war appears to remain a real in the lower band, we still think re- possibility. We have corresponding- sponding is appropriate, but perhaps ly placed it in the upper category for at a smaller scale for the time being. likelihood. It is the only catastrophe Our assessment is summarised in that is definitely within the reach and this diagram: control of humanity today, and we think it is therefore appropriate that B. HOW WE MADE THESE the response be high. ASSESSMENTS ABOUT Asteroid impacts and supervolca- nic eruptions are caused exogenous- PARTICULAR RISKS Natural pandemic Catastrophic climate change would ly to human actions. This means we likely have effects decades out, and have better estimates of their rates, we consider the likelihood in the next through knowledge of historical in- five years to be small unless the sci- cidents. A supervolcanic eruption entific community has significant- (which may or may not cause a glob- Unknown risks Nuclear war ly mis-modelled climate dynamics. al catastrophe) is estimated to oc- Nonetheless, because of cumulative cur very approximately every 30,000 nature of greenhouse gas emissions, years. An asteroid impact large and the relatively well understood dy- enough to cause a global catastrophe Engineered pandemic namics, we believe a large response is is estimated every 120,000 years. We justified. are therefore reasonably confident Natural pandemics have been re- that they belong to the lower category sponsible for past global catastro- of current risk. Because of this rela- Higher likelihood over next 5 years phes, and are seen high on lists of tively low risk, we have also put them national risk assessments. We are in the low category for attention. not confident assigning a partic- We are confident that this is justi- Catastrophic ular probability to a pandem- fied for supervolcanic eruptions, Asteroid impact climate change ic, but we do think it is among where there are few clear ac- the most likely risks in the tions to reduce the risk. As- next five years. Internation- teroids are exceptional in Supervolcanic Catastrophic al work in reducing the 1 that we have relatively eruption disruption from AI risk from pandemics every well-understood ways seems both important to reduce the risk, and and effective, and this could mean that we think a large re- 30,000 the attention cat- Failure of geo-engineering sponse is appro- years egory should be priate. is the estimated occurrence of a higher. Although supervolcanic eruption (which may or Engineered the likeli- may not cause a global catastrophe). pandem- Lower likelihood over next 5 years hood may ics are an Low High APPROPRIATE LEVEL OF ATTENTION

68 Global Catastrophic Risks 2016 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

emerging risk in that future tech- the speed of technological progress. nology will make them more plausi- In the longer term the risks could be There is a great deal of ble. But even existing experiments extremely significant, and experts have been claimed to pose real risk of have identified useful work that can sparking a pandemic, so we put this be done today, so we have put it in the uncertainty around many in the upper band for likelihood. We upper category of attention. also think it is appropriate that it gets Finally, risks from geoengineering significant amounts of attention, par- are another emerging technological ticularly as the unmanaged risk may risk where the current risks seem low. of the risks, and around be increasing. But because geoengineering may be Risks from artificial intelligence ap- employed to tackle climate change, pear to be low in the short-term, since it seems important to build a more we most likely are some way off the comprehensive understanding of the how much can usefully be kind of capabilities that might cause a risks it poses in itself. For this reason global catastrophe. We have therefore we have put it in the higher category put them in the lower category of cur- for attention. done. Nonetheless, some rent risk, although it is hard to predict attempt to estimate must be made in order to prioritise the world’s scarce resources.

70 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 71 CHAPTER 3 – RISK FACTORS AND INTERACTIONS BETWEEN RISKS

Chapter 3 Risk factors and interactions between risks n this chapter we discuss some Global catastrophic risks are parts of the factors that affect the like- of extremely complicated causal net- lihood of global catastrophic works, and the drivers of each one risks. We first discuss some of the could quite sensibly be the subject of key factors influencing individ- multiple books. Here, we provide a Iual risks, before turning to factors that brief overview where we highlight a the risks have in common, and ways in few key drivers. This serves as a back- which catastrophic events could trigger ground to the more action-oriented further catastrophes. discussions of Chapters 4 and 5.

72 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 73 CHAPTER 3 – RISK FACTORS AND INTERACTIONS BETWEEN RISKS

3.1. Drivers of “If ‘positive feedback loops’ prove individual risks to be worse than anticipated, the risk of catastrophic climate NUCLEAR WAR ty which could result from significant Increasing the number of nucle- climate change may increase the geo- change will be higher.” ar-armed actors -- horizontal prolif- political tensions that drive nuclear eration -- probably increases the like- risk.147 lihood of nuclear conflict. Increasing the size and sophistication of their PANDEMICS arsenals -- vertical proliferation -- in- The factors driving pandemic risk di- creases the amount of damage that vide into those affecting the likeli- could occur in a given conflict.141 hood of potentially pandemic patho- These factors are, in turn, influenced gens arising, and those affecting how by political factors such as leading much society will be affected by such nations’ ability to control rogue ac- pathogens. Whether a given pathogen tors, and the level of conflict between could cause a pandemic depends on nuclear states.142 its natural parameters, such as trans- The possibility of false alarms or mission vector, resistances, and le- miscalculation between nuclear pow- thality, over which we generally have ers is another driver of risk, exacer- little control.148 However, concentrat- bated further when geopolitical ten- ed populations of poultry or other es outside of human control. Howev- CATASTROPHIC CLIMATE CHANGE sions are high.143 There is also the animals can increase the probabili- er, our ability to predict, and, in the & GEO-ENGINEERING unresolved question of exactly how ty of zoonotic spillover, raising the case of asteroids and comets, to pre- The risk of catastrophic climate sensitive the climate system is to dust risk.149 The transmission of pathogens vent these catastrophes could have a change and the risk of a geoengineer- and sulphates. The higher the sensi- through society is increased by global significant bearing on their eventu- ing catastrophe are strongly influ- tivity, the greater the risk of a nuclear travel and dense populations, but de- al impact. Because we have less con- enced by the level of GHG emissions. winter scenario.144 creased by factors such as improved trol over the likelihood of an adverse These depend on factors including Our response to climate change hygiene.150 The effectiveness of our event with these than with most glob- the amount of energy the world con- may also have significant effects on countermeasures constitutes anoth- al catastrophic risks, resilience is a sumes and the proportion of this the risk of nuclear war. Many author- er important factor, ranging from dis- more central part of the risk manage- which comes from fossil fuels. The ities on climate change argue that it ease surveillance to healthcare access ment strategy. A particularly import- total energy consumption is shaped will be hard to reduce GHG emissions in the developing world.151 ant factor is the level of food security, by, among other things, population sufficiently without nuclear power including stockpiles and the ability growth, economic development, and being part of the energy mix.145 Unfor- SUPER-VOLCANOES, to switch to less sunlight-dependent energy efficiency. The proportion of tunately, rising use of nuclear power ASTEROIDS, AND COMETS food sources.152 Better food security energy which comes from fossil fuels may increase the chance of nuclear The probability of super-volcanic means a lower chance that a particu- depends on other factors such as the weapon proliferation.146 Moreover, the eruptions and asteroid and comet im- late winter would have catastrophic rate of progress in clean energy tech- mass migration and resource scarci- pacts are driven by natural process- consequences.153 nology development and the preva-

74 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 75 CHAPTER 3 – RISK FACTORS AND INTERACTIONS BETWEEN RISKS

lence of economic incentives to adopt just their risk management mecha- clean energy technologies.154 nisms.159 The speed and sud- Another important set of factors Formal regulation of technology concern how the climate is likely has an unclear impact—in some cas- to react to increased levels of GHG es it could prevent risky activities, but denness of technological emissions. If the climate turns out to in other cases it could stifle critical be generally more sensitive to GHG innovation or, if implemented before emissions than expected, or if “pos- the issues are properly understood, breakthroughs could itive feedback loops” prove to be even increase risk.160 On the positive worse than anticipated, the risk of side, technology developed in a cul- catastrophic climate change will in- ture of responsible innovation will also be a risk factor, as crease.155 likely be safer. Finally, the development of new For some of the emerging techno- geoengineering techniques might ei- logical risks we may also hope to de- sudden breakthroughs ther decrease or increase global cat- velop solutions. It may reduce risk to astrophic risk. They could decrease make progress on developing safety global catastrophic risk if they prove aspects of a technology faster relative might leave inadequate to be an effective tool to mitigate to progress on the technology itself. catastrophic climate change.156 But For example having excellent diag- they could increase risk if they have nosis and treatment tools to contain time for social and polit- a high chance of causing a catastro- outbreaks before there is significant phe of their own, or if they prove to risk from artificial pandemics could be ineffective while at the same time reduce the total risk significantly. ical institutions to adjust leading countries to avoid emission Similarly, a solution to the problem abatement (as discussed in Chapter of aligning advanced AI with human 2).157 values before we are able to create ad- their risk management vanced AI would reduce risk.161 The EMERGING TECHNOLOGY proposal of aiming for safety-enhanc- The factors that impact the risks of ing technologies ahead of risk-in- emerging technology are surround- creasing technologies in order to re- mechanisms. ed by a high degree of uncertainty. duce global catastrophic risk is called However, one plausible risk factor is ‘differential technological develop- misaligned incentives, which might ment’.162 cause nation-states to engage in arms record, which might have brought risks in the future (although techno- races or under-regulate risky tech- UNKNOWN RISKS many exogenous risks to our atten- logical progress could also reduce nology for economic reasons.158 The It is of course impossible to provide tion already. The historical record global catastrophic risk; see section speed and suddenness of technolog- a detailed account of the drivers of offers us little or no guidance on an- 3.2.). Another important factor in- ical breakthroughs could also be a currently unknown risks. Unknown thropogenic risks. They will often fluencing these unknown risks is the risk factor, as sudden breakthroughs risks could in principle include both be driven by social or technological quality of our foresight work, since might leave inadequate time for so- exogenous and anthropogenic risks. change, so a higher rate of change the ability to plan for a risk could help cial and political institutions to ad- We have access to a long historical could mean more currently unknown both avoidance and mitigation.

76 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 77 CHAPTER 3 – RISK FACTORS AND INTERACTIONS BETWEEN RISKS

3.2. Shared risk factors and Food stockpiles and the interactions between risks ability to rapidly increase o far we have mainly discussed might also be reversed. A global ca- each risk in isolation. But the tastrophe could lead to a breakdown S global catastrophic risks share of social and political institutions, production of alternate numerous risk factors, and also inter- which in turn could cause an outbreak act with each other. In this section, of violence. In a 2008 paper, Nel and we discuss some salient examples of Righarts argued that even smaller nat- shared risk factors - variables associ- ural disasters such as earthquakes, sources of food would ated with an altered level of risk - and tsunamis, and heat waves significant- interactions between risks. ly increase the risk of violent civil con- One important shared risk factor is flict in the short and medium term, increase resilience to a governance. Good governance may especially in low- and middle-income help timely and appropriate prepa- countries.163 Similarly, there has been ration and response to risks. By the significant attention on the relation- same token, bad governance, nation- ship between climate change and po- broad range of risks. ally or internationally, could increase litical conflict. Many scholars and se- the likelihood and potential impact of curity experts have argued that even every risk. Lax oversight might lead moderate climate change could in- to the accidental release of a danger- crease the risk of political violence ous engineered pathogen from a lab- because of conflicts over dwindling oratory. Dysfunctional governments natural resources (such as food and would be less able to intervene early water), massive international migra- in an outbreak to prevent it becom- tion and a range of other factors.164 ing a pandemic. Poor international Greater disasters are likely to lead coordination might radically wors- to even more upheaval, and might en our prospects to avoid catastroph- significantly weaken the defenses ic climate change, and internation- against further catastrophic events. In al tensions could increase the risk of this way, one global catastrophe might nuclear war. Bad governance could in trigger another. principle also lead to over-prioritising Another set of factors which affect catastrophic risk reduction, but be- many global catastrophic risks is tech- cause of the political distortions dis- nological. Technological advances cussed in Chapter 4 we think this is change the system we live in by giv- less likely. ing actors new powers. These in turn While bad governance can cause could increase or decrease global cat- global catastrophes, the causality astrophic risk, even aside from cases

78 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 79 CHAPTER 3 – RISK FACTORS AND INTERACTIONS BETWEEN RISKS

where the technology directly poses for unscrupulous states to control risk. It is hard to fully predict the ef- their citizens, which makes it easi- fects of technological trends, but we er for states to engage in extreme be- FIGURE 3.2. A WORKING GUN MADE FROM 165 will highlight three trends which may haviour. It might also weaken trust PLASTIC ON A 3D PRINTER be relevant: economic productivity between states, which could cause gains, surveillance, and distributed political tension. However, surveil- manufacturing. lance may make it more difficult for Global productivity has increased malicious states or terrorist groups dramatically in the last few decades. to act in secret. This will become in- Partly, this is because of technological creasingly important as barriers to ac- progress and organisational efficien- cess destructive weapons fall. Surveil- cy growth. Partly, it is due to automa- lance between countries could even tion, a form of technological progress facilitate international cooperation which we expect to continue. by making the actions of states more This growth may help us reduce transparent. It is not clear what over- risk because it lets us spend more on all effect surveillance has on levels of prevention and resilience. Measures global catastrophic risk. like clean energy and food stockpiles Distributed manufacturing is a set of are costly, and more likely to receive technologies that allow products to be investment when people are wealthy. designed and built without centralised In the tail case, with very powerful factories. They offer many benefits, artificial intelligence, there could be and may increase resilience to ca- a radical improvement in our ability tastrophe by spreading out production to manage other global catastrophic capacity. However, they also bypass risks. However, economic develop- some government controls designed ment may exacerbate some risks, at to prevent the construction of destruc- least in the short term, for example tive weapons. 3D printing, an early by increasing greenhouse gas emis- form of such technology, has already sions. generated security risks by allowing The global surveillance disclosures people to create functional homemade by Edward Snowden and others re- firearms. More powerful forms of dis- vealed that state surveillance is now tributed manufacturing could increase extensive. In addition, the use of risk caused by malicious actors, by in- CCTV is expanding, and police and creasing access to powerful weaponry other law enforcement agencies are such as bioweapons. making increasing use of cameras in Nuclear war, geo-engineering, su- of smoke, dust, and/or sulphates into egies for resilience that address one their day-to-day work. per-volcanoes, asteroids, and com- the stratosphere they could cause risk address several. Food stockpiles Increased surveillance has the po- ets all pose global catastrophic risk in global cooling, sunlight loss, ozone and the ability to rapidly incerase tential both to exacerbate and re- significant part because of the ‘par- loss, and subsequent agricultural production of alternate sources of duce global catastrophic risk. Exten- ticulate winter’ scenarios they might disruption.166 Because so many risks food would increase resilience to a sive surveillance could make it easier produce. By ejecting large amounts share this mechanism, many strat- broad range of risks.167

80 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 81 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

Chapter 4 We should expect Do institutions collectively global catastrophic risks underinvest in global to systematically receive catastrophic risk? less attention than they

n Chapters 2 and 3 we gave an of attention to the risks, and suggests merit. overview of several global cat- some mechanisms for reducing risk astrophic risks, the main fac- by countering the distortions. tors that affect their likelihood In the second half of the chapter, we and impact, and some of the examine the kinds of actors or insti- Ilevers available to influence them. In tutions that may be well-placed to act the first half of this chapter, we argue on global catastrophic risks, or have that market and political distortions a responsibility to do so. We look at mean that these risks are likely to be how they can help to correct the mar- systematically neglected by many ac- ket and political failures we consider tors. This increases the importance in the first half.

82 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 83 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

4.1. Market and political failures t may seem surprising that rela- Supra-national institutions are gener- tively little effort has gone into ally weaker and international com- I global catastrophic risk mitiga- munities less cohesive, and therefore tion. However, we should expect glob- are less able to implement these solu- al catastrophic risks to systematically tions to collective action problems. receive less attention than they merit, Sometimes, such as with the World for structural reasons described be- Trade Organisation, nations do col- low. lectively give up some of their auton- omy in order to provide a public good, 4.1.1. GLOBAL but this process requires a great deal PUBLIC GOODS of negotiation and trust. Thus, insti- Global catastrophic risk reduction is tutions for aligning the incentives of a global public good – the benefits of nation-states to provide global public reduction spill over to other countries goods are typically less mature and and it is hard (and arguably unethi- less effective than those that provide cal) to only protect the countries that national public goods. contribute to risk reduction. Many global public goods tend to be under- 4.1.2. INTERGENERA- provided, primarily because coun- TIONAL PUBLIC GOODS tries try to free-ride.168 Every country Many of the benefits of global cata- might hope that other countries in- strophic risk reduction accrue to fu- vest in risk reduction, so they can get ture generations. However, the in- protected for free. If everyone shares terests of future generations tend to this hope, no one will invest. To over- be systematically neglected because come this, there is a need for good co- they cannot vote and have no direct ordination between actors. voice in the political process.171 For ex- In well-functioning states, na- ample, reducing the risk from cata- tional public goods, such as defence strophic climate change may provide and clean air,169 are provided despite only small benefits to many people the difficulties of collective action. alive today, but could be very valu- Sometimes, central regulation sup- able to people who will exist in sixty plies these goods, sometimes mar- years’ time If we do not have formal ket mechanisms are employed, and processes for taking the interests of sometimes cooperative institutions future generations into account, we for supplying the goods emerge.170 may under-invest in risk reduction.

84 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 85 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK? FIGURE 4.1. PREPAREDNESS IN THE AFRICA REGION Multi-hazard national public health emergency preparedness & response plan in the African Region. 174

4.1.3. SPECIAL INTERESTS 4.1.4. UNPRECEDENTED The costs of regulatory actions to re- RISKS duce global catastrophic risks will Global catastrophic events occur very typically be concentrated on partic- infrequently. In the last two millen- ular industries, whereas the bene- nia, there may have only been two fits are dispersed. The small affected such events – the Plague of Justinian groups have strong incentives to lob- and the Black Death. Anthropogen- by and campaign, while those gaining ic global catastrophes are complete- the benefits will not regard those ben- ly unprecedented. Because unprec- efits as a voting priority. This means edented events are typically less that industry lobbying could wield salient, it is less likely that govern- disproportionate power over the reg- ments and voters will pay appropri- ulatory process.172 Consequently, the ate attention to them, in spite of their trade-offs made between econom- very high costs in expectation.173 ic profit and risk reduction could be skewed towards underprovision of risk reduction.

Implementation status (%)

0-24

25-49

50-74

75-100

Data not available

Not part of WHO African region

Not applicable

86 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 87 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

4.2. Which actors can help reduce global catastrophic risk?

here is a broad range of ac- the risk of nuclear war. International tors that can help us over- coordination can aim to reduce the T coming the current neglect of risk of nuclear conflict between these global catastrophic risk. In this sec- major actors.177 tion, we give an overview of some There have been many agreements ways in which different groups, intended to reduce global catastroph- ranging from the international com- ic risk. Two of the more important munity to individuals, can contrib- examples178 are the Treaty on the ute to the mitigation of global cata- Non-Proliferation of Nuclear Weap- strophic risks. ons179 and the 1992 United Nations Framework Convention on Climate 4.2.1. THE INTERNA- Change, with its objective of pre- TIONAL COMMUNITY venting “dangerous” anthropogen- As discussed in section 4.1, reduc- ic climate change.180 Similarly, there tion of global catastrophic risk is a are several permanent internation- global public good. This means that al bodies, such as the International the international community will Atomic Energy Agency and the World probably need to play a major role in Health Organisation, which are (part- reducing global catastrophic risk. In ly) concerned with reducing global some cases, the global level of risk is catastrophic risks (nuclear war181 and significantly dependent on the ca- pandemics, respectively). The inter- pabilities of the countries that are national community often collabo- the weakest links.175 For example, to rates productively with relevant parts safeguard against the risk of a global of the academic community, as it did pandemic, the least prepared coun- when it set up the Intergovernmen- tries should be enabled to strengthen tal Panel on Climate Change.182 Many their health systems.176 In other cas- of the global catastrophic risks - not es, the risk of a catastrophe is mainly least those associated with emerg- dependent on a few major actors. For ing technologies - require a deep level example, the nuclear-weapon states of technical expertise to be properly have a disproportionate influence on managed.

88 Global Catastrophic Risks 2016 89 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

4.2.2. NATION-STATES how to respond. For instance, we Although the international commu- would benefit from more resources nity has a crucial role in the reduc- devoted to the study of global cata- tion of global catastrophic risk — strophic risks such as tail risk climate because this is a global public good change183 and particulate winter sce- — much power currently resides in narios.184 We would also benefit from the hands of nation-states. In prac- more research on technical solutions tice, international bodies have limit- that could reduce global catastroph- ed power unless they are backed up ic risks, such as clean energy sourc- by nation-states (especially the more es, or how to how to align the actions powerful ones). This means that na- of above-human-level AI with our tion-states are essential to galvanis- values.185 Since catastrophic risk re- ing action. They also need to lead the duction is a public good, it is unlike- process of implementation. For in- ly that this research will be carried stance, it is up to nation-states to see out by companies under competitive that emissions actually are cut to the pressure. Instead, it will most likely extent required by the Paris agree- require public or charitable funding. ment. Similarly, it is the responsibili- In fact, several academic institutions ty of nuclear-weapon states to guard that focus on global catastrophic risk against the possibility of accidental have already been set up through launch. public and philanthropic funds. Nation-states can also reform their Among them are Oxford’s Future of internal political processes in ways Humanity Institute and Cambridge’s that are conducive to reduction of Centre for the Study of Existential global catastrophic risks. Ensuring Risk. that decisions take account of the in- Beyond this, the research commu- terests of future generations is, for nity should contribute to global cat- instance, likely to lead to a more ap- astrophic risk reduction by promot- propriate degree of focus on glob- ing a culture of safety within areas of al catastrophic risk. Similarly, con- research that could have the poten- straining the excessive power of tial to cause a catastrophe through special interests could decrease dis- accident or misuse. This is especially tortions on decision-making and so in relevant for emerging technologies, turn reduce global catastrophic risk. where it is not always clear in advance whether there are any risks.186 4.2.3. THE RESEARCH COMMUNITY 4.2.4. INDUSTRY Many of the global catastrophic risks Competitive pressures mean that it are not well-understood, and more is often hard for companies to make research would allow more appropri- large moves on issues which do not ate decisions about when to act and improve their profits. They can, how-

90 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 91 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

ever, show leadership on reducing the 4.2.6. THE NON-PROFIT risk of catastrophic climate change SECTOR The research community by choosing low carbon options at The non-profit sector contributes sig- the margin, or by developing clean nificantly to global catastrophic risk energy solutions. Just like the wider reduction. Non-profits are often less should contribute to research community, they can and constrained than companies and na- should promote a culture of safety in tional governments, which means biotechnology and AI research as, for that they are free to work on intergen- example, Google DeepMind has done erational global public goods such as global catastrophic risk by setting up an AI ethics board.187 global catastrophic risk reduction -- if they can find donors that support 4.2.5. GENERAL PUBLIC that cause. There are many charities Some global catastrophic risks can be working on individual global cata- reduction by promoting a addressed by individual action. For strophic risks. In recent years, donors instance, individuals can decrease have been increasingly interested in the risk of catastrophic climate global catastrophic risk reduction as culture of safety within change (if by ever so little) by mak- a general category. This has led to the ing low-carbon consumption choic- creation of groups such as the Future es. Perhaps the more promising route of Life Institute in Boston. to reduce global catastrophic risk for Besides supporting research, chari- areas of research that could individuals is, however, by exerting ties can also support direct interven- political pressure on policymakers. tions, e.g. to increase pandemic pre- For instance, voters could try to in- paredness in developing countries.188 fluence politicians to agree to carbon They can also exert political pressure, have the potential to cause emissions cuts, and then to actually as International Physicians for the implement the agreements once they Prevention of Nuclear War did to re- are in place. duce the risk of a nuclear exchange during the cold war, winning the No- a catastrophe through bel Peace Prize in 1985.189 accident or misuse.

92 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 93 CHAPTER 5 – WHAT CAN THE WORLD DO TO REDUCE GLOBAL CATASTROPHIC RISK?

Chapter 5 Our aim here is to What can the world offer some starting do to reduce global points for considering catastrophic risk? action on risks of global catastrophe n Chapter 2, we looked at dif- in order to reduce global catastroph- ferent global catastrophic risks ic risk. Some of these steps pertain to and their mechanisms. There individual risks, whereas others are was also some preliminary dis- cross-cutting opportunities which and to demonstrate that cussion of actions available to may reduce the chance or impact of Ilimit the risks. In Chapter 3 we ex- several different risks at once. plored the different factors which af- Our aim here is to offer some start- fect these risks. In Chapter 4 we con- ing points for considering action on there are real avenues to sidered why global catastrophic risk is risks of global catastrophe and to probably neglected, and how differ- demonstrate that there are real ave- ent actors can help with it. Finally in nues to making progress. For more this chapter we draw these strands detailed discussion of the actions making progress. together and outline a few of the most available, there exists a rich literature promising steps that existing commu- on most of the specific risks. nities can take or are already taking

94 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 95 CHAPTER 5 – WHAT CAN THE WORLD DO TO REDUCE GLOBAL CATASTROPHIC RISK?

CATASTROPHIC CLIMATE CHANGE NUCLEAR WAR • Research communities should in- • The international community crease their focus on understand- should continue the policy of nu- ing the pathways to and likelihood clear non-proliferation, and nu- of catastrophic climate change, and clear states can continue to reduce possible ways to respond.190 stockpiles. • Nations should continue to imple- • Nuclear-weapon states should ment and improve mechanisms continue to work to reduce the for emissions abatement such as chance of accidental launch or carbon taxes or tradable emissions escalation.193 quotas, as for non-catastrophic cli- • Nuclear weapon states can adopt mate change.191 military strategies that reduce the • Developed nations could commit chance of nuclear winter.194 to the goal of building no new coal- fired power stations without carbon capture and sequestration.192

NATURAL PANDEMICS • The World Health Organisation, nation states, and other bodies should increase their planning for extremely bad pandemics.195 • International and research com- munities could improve disease ASTEROIDS AND COMETS surveillance, for example by reduc- • Research communities should con- ing the delay between scientific tinue working to detect and track breakthrough and the availability asteroids and commets with a di- of diagnostic tools. ameter of 1 km or more.197 • The global health community should improve developing world SUPERVOLCANIC ERUPTIONS capacity for response, for example • Research communities should con- by ensuring that vaccine produc- tinue to work on understanding tion facilities are well-distributed their causes, to increase predict- around the world.196 ability.198

96 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 97 CHAPTER 5 – WHAT CAN THE WORLD DO TO REDUCE GLOBAL CATASTROPHIC RISK?

Research communities should focus greater attention on strategies

BIOTECHNOLOGY • Research communities can further • Research communities should fur- investigate the potential impacts of ther investigate the possible risks solar radiation management.202 and technologies for from emerging capabilities in bio- technology, and develop counter- UNKNOWN RISKS measures.199 • Research communities can contin- resilience to and • Policymakers should continue to ue to develop methods and tools work with researchers to understand for horizon scanning and reduc- the biosafety and biosecurity issues tion of unknown unknowns. that are likely to arise, and build • Research communities should recovery from global planned adaptive risk regulation.200 identify and carefully investigate speculative threats. ARTIFICIAL INTELLIGENCE • Research communities should fur- CROSS-CUTTING OPPORTUNITIES catastrophe, for example ther investigate the possible risks • Research communities should fur- from artificial intelligence, and ther investigate ‘particulate winter’ work on developing possible solu- scenarios, both in forecasting and tions.201 mitigation strategies.203 by developing alternate • Policymakers can work with re- • Nations and local communities can searchers to understand the impli- continue to take steps to build their cations of advanced artificial intel- resilience to catastrophe.204 food sources. ligence. • Research communities should fo- cus greater attention on strategies GEO-ENGINEERING and technologies for resilience to • The international community and recovery from global catastro- should continue work to stabilise phe, for example by developing al- and eventually reduce concentra- ternate food sources.205 tions of greenhouse gases in the • Nations should work to incorporate atmosphere in order to reduce the the interests of future generations incentives to use solar radiation into their decision-making frame- management. works.206

98 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 99 ENDNOTES

Warming to Cumulative Carbon Emissions,” Nature 459, War Between the United States and Russia,” Science & no. 7248 (June 11, 2009): 829–32. Global Security 21, no. 2 (May 1, 2013): 106–33, doi:10.10 35. Malte Meinshausen et al., “Greenhouse-Gas Emission 80/08929882.2013.798984. Targets for Limiting Global Warming to 2 °C,” Nature 458, 51. Graham Allison, “At 50, the Cuban Missile Crisis as Endnotes no. 7242 (April 30, 2009): 1158–62. Guide,” The New York Times, June 15, 2012. 1. William Rosen, Justinian’s Flea : Plague, Empire, and 16. Ali Nouri and Christopher F. Chyba, “Biotechnology 36. Ibid. 52. Zachary Keck, “Russia Threatens Nuclear Strikes Over the Birth of Europe (New York: Viking, 2007), 3, 209. and Biosecurity,” in Global Catastrophic Risks, ed. Nick 37. J. G. Shepherd, Geoengineering the Climate: Science, Crimea,” July 11, 2014; National Institute for Public Policy, Percentages deduced from Demographic Internet Staff Bostrom and Milan M. Ćirković (Oxford University Press, Governance and Uncertainty (Royal Society, 2009), chap. 2. “Russia’s Nuclear Posture” (Fairfax, VA, 2015). US Census Bureau, “International Programs, World 2008). 38. Ibid., chap. 3. 53. Hellman, “How Risky Is Nuclear Optimism?”; Tom Population,” accessed February 12, 2016, https://www. 17. Future of Life Institute, “AI Open Letter,” accessed Hussain, “Are India and Pakistan Heading for a Nuclear 39. Gernot Wagner and Martin L. Weitzman, Climate Shock : census.gov/population/international/data/worldpop/ February 3, 2016, http://futureoflife.org/ai-open-letter/. Showdown?,” Al Jazeera, March 2016, http://www. table_history.php. Some earlier estimates put the death The Economic Consequences of a Hotter Planet (Princeton: aljazeera.com/indepth/opinion/2016/03/india-pakistan- 18. NASA, “The Relentless Rise of Carbon Dioxide,” Climate toll as high as 100 million. See B. Lee Ligon, “Plague: Princeton University Press, 2015), 54. heading-nuclear-showdown-160303053541342.html. A Review of Its History and Potential as a Biological Change: Vital Signs of the Planet, accessed February 2, 40. See Martin E. Hellman, “How Risky Is Nuclear Optimism?,” 54. Michael Krepon, “Pakistan’s Nuclear Strategy and Weapon,” Seminars in Pediatric Infectious Diseases, 2016, http://climate.nasa.gov/climate_resources/24/. Bulletin of the Atomic Scientists 67, no. 2 (March 1, 2011): Deterrence Stability,” Stimson Center 10 (2012), http:// Return of the Gram-Positives, 17, no. 3 (July 2006): 19. Gernot Wagner and Martin L. Weitzman, Climate Shock : 47–56. indianstrategicknowledgeonline.com/web/Krepon_-_ 161–70. The Economic Consequences of a Hotter Planet (Princeton: 41. Bulletin of the Atomic Scientists, “Nuclear Notebook,” Pakistan_Nuclear_Strategy_and_Deterrence_Stability.pdf. 2. Rosen, Justinian’s Flea, 3–5, 321–324. Princeton University Press, 2015). Bulletin of the Atomic Scientists, accessed February 55. See Cirincione, “The Continuing Threat of Nuclear War”; 20. Clearly, enormous literatures exist in each of these 3. Edward Wilson, “Thank You Vasili Arkhipov, the Man Who 3, 2016, http://thebulletin.org/nuclear-notebook- Hellman, “How Risky Is Nuclear Optimism?” Stopped Nuclear War,” The Guardian, October 27, 2012, areas which can inform policy. We suggest only a small multimedia. selection of the policy options which are available. 56. Cirincione, “The Continuing Threat of Nuclear War,” 396. http://www.theguardian.com/commentisfree/2012/ 42. Hans M. Kristensen and Robert S. Norris, “US Nuclear oct/27/vasili-arkhipov-stopped-nuclear-war. 21. NASA, “The Relentless Rise of Carbon Dioxide,”. Forces, 2015,” Bulletin of the Atomic Scientists 71, no. 2 57. Matthew R. Costlow, “Do More Nukes Really Mean More Nuclear Crises? Not Necessarily,” Bulletin of the Atomic 4. This is adapted from Nick Bostrom, “Existential Risk 22. Sergey A. Zimov, Edward AG Schuur, and F. Stuart Chapin (2015): 108. Scientists, December 8, 2015, http://thebulletin.org/ Prevention as Global Priority,” Global Policy 4, no. 1 III, “Permafrost and the Global Carbon Budget,” accessed 43. Hans M. Kristensen and Robert S. Norris, “Russian do-more-nukes-really-mean-more-nuclear-crises-not- (February 1, 2013): 16. January 18, 2016, http://www.imedea.uib-csic.es/master/ Nuclear Forces, 2015,” Bulletin of the Atomic Scientists necessarily8965. 5. Steven Pinker, The Better Angels of Our Nature : A History of cambioglobal/Modulo_V_cod101619/Permafrost%20 71, no. 3 (2015): 85. 58. For other suggestions for weapons systems see Seth Violence and Humanity (London: Penguin, 2012),195. Note response.pdf. 44. “A Nuke by Any Other Name,” Bulletin of the Atomic D. Baum, “Confronting the Threat of Nuclear Winter,” that although the fatalities are too low, the non-fatality 23. Christopher B. Field et al., Climate Change 2014: Scientists, May 17, 2012, http://thebulletin.org/nuke-any- Futures, Confronting Future Catastrophic Threats To impacts mean it could be argued that the Second World Impacts, Adaptation, and Vulnerability: Summary for other-name. Humanity, 72 (September 2015): 69–79, doi:10.1016/j. War was just about large enough to count as a global Policymakers (Contribution of Working Group II to the 45. US Office of Technology Assessment, “The Effects of futures.2015.03.004. catastrophe in our sense. Fifth Assessment Report of the Intergovernmental Panel Nuclear War,” 1979, 8–9, http://ota.fas.org/reports/7906. on Climate Change, 2014), 14, http://epic.awi.de/37531/. 59. http://thebulletin.org/nuclear-notebook-multimedia 6. Niall Johnson and Juergen Mueller, “Updating the pdf; Joseph Cirincione, “The Continuing Threat of Nuclear Accounts: Global Mortality of the 1918-1920 ‘Spanish’ 24. David King et al., “Climate Change–a Risk Assessment” War,” in Global Catastrophic Risks, ed. Nick Bostrom 60. See the work of the Nuclear Threat Initiative at http:// Influenza Pandemic,” Bulletin of the History of Medicine 76, (Centre for Science Policy, University of Cambridge, and Milan M. Ćirković (Oxford University Press, 2008), www.nti.org/. no. 1 (2002): 105–15. 2015), 45–46, www.csap.cam.ac.uk/projects/climate- 386–390. 61. Alan Robock, Luke Oman, and Georgiy L. Stenchikov, change-risk-assessment/. 7. For further discussion see the sections on natural 46. This area has received less attention since the end of “Nuclear Winter Revisited with a Modern Climate pandemics, nuclear war and super-volcanic eruptions in 25. Gernot Wagner and Martin L. Weitzman, Climate the Cold War. Other estimates give death tolls within the Model and Current Nuclear Arsenals: Still Catastrophic Chapter 2. Shock : The Economic Consequences of a Hotter Planet same order of magnitude. Ira Helfand et al., “Projected Consequences,” Journal of Geophysical Research: Atmospheres 112, no. D13 (July 16, 2007): 10, 8. Adapted from Nick Bostrom, “Existential Risk Prevention (Princeton: Princeton University Press, 2015), 53–56. US Casualties and Destruction of US Medical Services doi:10.1029/2006JD008235. as Global Priority,” Global Policy 4, no. 1 (February 1, 26. King et al., “Climate Change–a Risk Assessment,” chap. from Attacks by Russian Nuclear Forces,” Medicine & 2013): 15–31, doi:10.1111/1758-5899.12002. 10. Global Survival 7, no. 2 (2002): 68–76; WHO, “Health 62. Ole J. Benedictow, “The Black Death: The Greatest Effects of the Chernobyl Accident and Special Health Care Catastrophe Ever,” History Today, 2005, http://www. 9. Anthony M. Barrett, Seth D. Baum, and Kelly Hostetler, 27. King et al., “Climate Change–a Risk Assessment,” pt. 1. Programmes” (WHO, 2006), http://www.who.int/ionizing_ historytoday.com/ole-j-benedictow/black-death- “Analyzing and Reducing the Risks of Inadvertent Nuclear 28. “Green Light,” The Economist, December 19, 2015, http:// radiation/chernobyl/who_chernobyl_report_2006.pdf; greatest-catastrophe-ever; Edwin Dennis Kilbourne, War Between the United States and Russia,” Science & www.economist.com/news/international/21684144-what- Owen B. Toon et al., “Atmospheric Effects and Societal “Plagues and Pandemics: Past, Present, and Future,” Global Security 21, no. 2 (May 1, 2013): 106–33, doi:10.108 expect-after-deal-exceeded-expectations-green-light. Consequences of Regional Scale Nuclear Conflicts and in Global Catastrophic Risks, ed. Nick Bostrom and 0/08929882.2013.798984. 29. Rajendra K. Pachauri et al., Climate Change 2014: Synthesis Acts of Individual Nuclear Terrorism,” Atmospheric Milan M. Ćirković (Oxford: Oxford University Press, 10. Figures inferred from Table A2 of WHO, “Global Status Report. Contribution of Working Groups I, II and III to the Chemistry and Physics 7, no. 8 (2007): 1973–2002. 2008), 295. Percentages deduced using population data Report on Road Safety 2015,” 264–271. Fifth Assessment Report of the Intergovernmental Panel on 47. For a good summary see Seth D. Baum, “Winter-Safe from Demographic Internet Staff US Census Bureau, 11. Ibid. Climate Change, 2014, 10-13 Deterrence: The Risk of Nuclear Winter and Its Challenge “International Programs, World Population,” accessed February 12, 2016, https://www.census.gov/population/ 12. N. H. Stern, The Economics of Climate Change : The Stern 30. Wagner and Weitzman, Climate Shock, 53–56. to Deterrence,” Contemporary Security Policy 36, no. 1 international/data/worldpop/table_history.php. Review (Cambridge: Cambridge University Press, 2007), 47. (January 2, 2015): 126, doi:10.1080/13523260.2015.101 31. Field et al., Climate Change 2014: Impacts, Adaptation, 2346. 63. Niall Johnson and Juergen Mueller, “Updating the 13. Derek Parfit, Reasons and Persons (Oxford: Oxford and Vulnerability: Summary for Policymakers, chap. 16. Accounts: Global Mortality of the 1918-1920 ‘Spanish’ University Press, 1984); Nick Bostrom, “Astronomical 48. Alan Robock and Owen Brian Toon, “Local Nuclear War, 32. Nicholas Stern, “The Economics of Climate Change,” The Influenza Pandemic,” Bulletin of the History of Medicine Waste: The Opportunity Cost of Delayed Technological Global Suffering,” Scientific American 302, no. 1 (2010): American Economic Review 98, no. 2 (May 1, 2008): 1–37; 76, no. 1 (2002): 105–15. Development,” Utilitas 15, no. 03 (November 2003): 74–81. See the discussion in Givewell, “Nuclear Weapons William D. Nordhaus, “A Review of the ‘Stern Review on 308–14; Nick Bostrom, “Existential Risk Prevention as Policy,” September 2015, http://www.givewell.org/labs/ 64. WHO, “Current WHO Phase of Pandemic Alert for Avian the Economics of Climate Change,’” Journal of Economic Global Priority”. causes/nuclear-weapons-policy. Influenza H5N1,” WHO, accessed February 15, 2016, Literature 45, no. 3 (September 1, 2007): 686–702; http://apps.who.int/csr/disease/avian_influenza/phase/ 14. We discuss this in more detail in chapter 4. Wagner and Weitzman, Climate Shock. 49. Martin E. Hellman, “How Risky Is Nuclear Optimism?” en/index.html; Declan Butler, “Death-Rate Row Blurs 15. http://www.atomicarchive.com/Docs/Begin/Einstein. 33. Wagner and Weitzman, Climate Shock, chap. 3. 50. Anthony M. Barrett, Seth D. Baum, and Kelly Hostetler, Mutant Flu Debate,” Nature 482, no. 7385 (February 13, shtml “Analyzing and Reducing the Risks of Inadvertent Nuclear 34. H. Damon Matthews et al., “The Proportionality of Global 2012): 289–289, doi:10.1038/482289a.

100 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 101 ENDNOTES

65. Marc Lipsitch and Thomas V. Inglesby, “Moratorium on Ever,” History Today, 2005, http://www.historytoday.com/ Letters 222, no. 1 (May 15, 2004): 11, doi:10.1016/j. SSRN Scholarly Paper (Rochester, NY: Social Science Research Intended To Create Novel Potential Pandemic ole-j-benedictow/black-death-greatest-catastrophe-ever; epsl.2004.03.004. Research Network, July 3, 2013), http://papers.ssrn.com/ Pathogens,” mBio 5, no. 6 (December 31, 2014): WHO and others, Bugs, Drugs and Smoke: Stories from 89. National Research Council (U. S.). Committee to Review abstract=2290382. e02366–14, doi:10.1128/mBio.02366-14. Public Health, 2011, chap. 1, http://apps.who.int/iris/ Near-Earth-Object Surveys and Hazard Mitigation 107. Martin J. Rees, Our Final Century : Will Civilisation Survive 66. Butler, “Death-Rate Row Blurs Mutant Flu Debate.” handle/10665/44700; WHO, “HIV/AIDS Fact Sheet,” WHO, Strategies, Defending Planet Earth, 23. the Twenty-First Century? (London: Arrow, 2004). November 2015, http://www.who.int/mediacentre/ 67. Christopher JL Murray et al., “Estimation of Potential factsheets/fs360/en/. 90. Alan Harris, “What Spaceguard Did,” Nature 453, no. 7199 108. Marc Lipsitch and Alison P. Galvani, “Ethical Alternatives Global Pandemic Influenza Mortality on the Basis of Vital (June 26, 2008): 1178–79, doi:10.1038/4531178a; NASA, to Experiments with Novel Potential Pandemic Registry Data from the 1918–20 Pandemic: A Quantitative 77. Arnon Dar, “Influence of Supernovae, Gamma-Ray “NASA Space Telescope Finds Fewer Asteroids near Pathogens,” PLoS Med 11, no. 5 (May 20, 2014): Analysis,” The Lancet 368, no. 9554 (December 23, 2006): Bursts, Solar Flares, and Cosmic Rays on the Terrestrial Earth,” September 2011, http://www.nasa.gov/mission_ e1001646, doi:10.1371/journal.pmed.1001646. 2211–18, doi:10.1016/S0140-6736(06)69895-4. Environment,” in Global Catastrophic Risks, ed. Nick pages/WISE/news/wise20110929.html. Bostrom and Milan M. Ćirković (Oxford: Oxford 109. Lynn C. Klotz and Edward J. Sylvester, “The Unacceptable 68. Michael T. Osterholm, “Preparing for the Next Pandemic,” University Press, 2008). 91. Jason C. Reinhardt et al., “Asteroid Risk Assessment: A Risks of a Man-Made Pandemic,” Bulletin of the Atomic New England Journal of Medicine 352, no. 18 (May 5, Probabilistic Approach,” Risk Analysis, July 1, 2015, 1-18, Scientists, August 7, 2012, http://thebulletin.org/ 2005): 1839–42, doi:10.1056/NEJMp058068; Neil M. 78. See the draft manuscript Toby Ord, “Will We Cause Our doi:10.1111/risa.12453. unacceptable-risks-man-made-pandemic. http://www. Ferguson et al., “Strategies for Mitigating an Influenza Own Extinction? Natural versus Anthropogenic Extinction the-scientist.com/?articles.view/articleNo/41263/title/ Risks” 2014. Ord’s argument is about extinction risks and 92. Source: Alan Harris, “What Spaceguard Did,” Nature 453, Pandemic,” Nature 442, no. 7101 (July 27, 2006): 448–52, no. 7199 (June 26, 2008): 1178–79, doi:10.1038/4531178a. Moratorium-on-Gain-of-Function-Research/ doi:10.1038/nature04795. not the broader category of global catastrophic risks, but it is very plausible that similar arguments would also show 93. National Research Council (U. S.). Committee to Review 110. Lipsitch and Inglesby, “Moratorium on Research Intended 69. Bruine de Bruin, Wändi, et al. “Expert judgments of that exogenous non-extinction global catastrophic risks Near-Earth-Object Surveys and Hazard Mitigation To Create Novel Potential Pandemic Pathogens,” 2. pandemic influenza risks.” Global Public Health 1.2 are less likely than anthropogenic non-extinction global Strategies, Defending Planet Earth, 23. 111. Ibid. (2006): 179-194. catastrophic risks. 94. Ibid., 22. 112. Gautam Mukunda, Kenneth A. Oye, and Scott C. Mohr, 70. For two recent overviews see Peter Sands, Carmen 79. Michael Rampino, “Super-Volcanism and Other 95. Harris, “What Spaceguard Did.” “What Rough Beast?,” Politics and the Life Sciences 28, Mundaca-Shah, and Victor J. Dzau, “The Neglected Geophysical Processes of Catastrophic Import,” in Global no. 2 (September 1, 2009): 8, doi:10.2990/28_2_2. Dimension of Global Security — A Framework for Catastrophic Risks, ed. Nick Bostrom and Milan M. 96. A. Harris et al., “Asteroid Impacts and Modern Civilization: 113. Church, “A Synthetic Biohazard Non-Proliferation Countering Infectious-Disease Crises,” New England Ćirković (Oxford: Oxford University Press, 2008). Can We Prevent a Catastrophe,” Asteroids IV. University Journal of Medicine 0, no. 0 (January 13, 2016): null, of Arizona Press, Tucson, AZ, 2015, http://elib.dlr. Proposal.” 80. G. A. Zielinski et al., “Potential Atmospheric Impact of doi:10.1056/NEJMsr1600236; WHO and Report of the de/100120/1/3004_Harris_reprint.pdf. 114. Owen Cotton-Barratt, Sebastian Farquhar, and Andrew the Toba Mega-Eruption 71,000 Years Ago,” Geophysical High-level Panel on the Global Response to Health Crises, Snyder-Beattie, “Beyond Risk-Benefit Analysis: Pricing Research Letters 23, no. 8 (April 15, 1996): 837–40, 97. Robert Carlson, “The Changing Economics of DNA “Protecting Humanity from Future Health Crises” (WHO, Externalities for Gain-of-Function Research of Concern” doi:10.1029/96GL00706; Rampino, “Super-Volcanism and Synthesis,” Nature Biotechnology 27, no. 12 (December January 25, 2016), http://www.un.org/News/dh/infocus/ (Future of Humanity Institute, Global Piorities Project, Other Geophysical Processes of Catastrophic Import,” 2009): 1091–94, doi:10.1038/nbt1209-1091. HLP/2016-02-05_Final_Report_Global_Response_to_ March 2016). Health_Crises.pdf. 209–210. 98. Terrence M. Tumpey et al., “Characterization of the Reconstructed 1918 Spanish Influenza Pandemic Virus,” 115. Chyba and Greninger, “Biotechnology and Bioterrorism,” 71. Stanley A. Plotkin, Adel A.F. Mahmoud, and Jeremy 81. Rampino, “Super-Volcanism and Other Geophysical Science 310, no. 5745 (2005): 77–80. 151–153. Farrar, “Establishing a Global Vaccine-Development Processes of Catastrophic Import,” 211–212; Christine 116. “DNA Sequencing Costs,” accessed March 31, 2016, Fund,” New England Journal of Medicine 373, no. 4 (July 23, S. Lane, Ben T. Chorn, and Thomas C. Johnson, “Ash 99. Markus Schmidt, “Diffusion of Synthetic Biology: A http://www.genome.gov/sequencingcosts/. 2015): 297–300, doi:10.1056/NEJMp1506820. from the Toba Supereruption in Lake Malawi Shows No Challenge to Biosafety,” Systems and Synthetic Biology 2, Volcanic Winter in East Africa at 75 Ka,” Proceedings of no. 1–2 (June 2008): 1–6, doi:10.1007/s11693-008-9018-z. 117. Future of Life Institute, “AI Open Letter,” accessed 72. Catherine Yen et al., “The Development of Global the National Academy of Sciences 110, no. 20 (May 14, 100. Ronald J. Jackson et al., “Expression of Mouse February 3, 2016, http://futureoflife.org/ai-open-letter/. Vaccine Stockpiles,” The Lancet Infectious Diseases 2013): 8025–29, doi:10.1073/pnas.1301474110. 15, no. 3 (March 2015): 340–47, doi:10.1016/S1473- Interleukin-4 by a Recombinant Ectromelia Virus 118. Eliezer Yudkowsky, “Artificial Intelligence as a Positive 3099(14)70999-5. 82. S. Sparks et al., Super-Eruptions: Global Effects and Suppresses Cytolytic Lymphocyte Responses and and Negative Factor in Global Risk,” in Global Catastrophic Future Threats. London, Report of a Geological Society of Overcomes Genetic Resistance to Mousepox,” Journal of Risks, ed. Nick Bostrom and Milan M. Ćirković, vol. 1 73. WHO, “International Health Regulations: Support to London Working Group, 2005, 6. Virology 75, no. 3 (2001): 1205–10. (Oxford University Press, 2008), 303. Global Outbreak Alert and Response, and Building and Maintaining National Capacities,” 2015, http:// 83. W. Aspinall et al., “Volcano Hazard and Exposure in 101. Christopher F. Chyba and Alex L. Greninger, 119. Carl Benedikt Frey and Michael A. Osborne, “The Future apps.who.int/iris/bitstream/10665/199747/1/WHO_ GFDRR Priority Countries and Risk Mitigation Measures,” “Biotechnology and Bioterrorism: An Unprecedented of Employment” 2013; Nick Bostrom, Superintelligence : HSE_GCR_2015.7_eng.pdf; WHO, “Pandemic Influenza Volcano Risk Study 0100806-00-1-R, 2011, 15; Rampino, World,” Survival 46, no. 2 (2004): 148–149. Paths, Dangers, Strategies (Oxford: Oxford University “Super-Volcanism and Other Geophysical Processes of Preparedness Framework,” 2011, http://www.who.int/ 102. Sander Herfst et al., “Airborne Transmission of Influenza Press, 2014), chap. 11. Catastrophic Import,” 212–213; Susan Loughlin et al., influenza/resources/pip_framework/en/. A/H5N1 Virus between Ferrets,” Science 336, no. 6088 120. Bostrom, Superintelligence, chap. 4. Global Volcanic Hazards and Risk (Cambridge University (2012): 1534–41; Masaki Imai et al., “Experimental 74. Emily H. Chan et al., “Global Capacity for Emerging Press, 2015), 97. 121. For extensive discussion of the ‘AI control problem’ see Infectious Disease Detection,” Proceedings of the Adaptation of an Influenza H5 HA Confers Respiratory Bostrom, Superintelligence. National Academy of Sciences 107, no. 50 (December 14, 84. Sparks et al., Super-Eruptions, 20. Droplet Transmission to a Reassortant H5 HA/H1N1 Virus 122. Bostrom, Superintelligence, 18–21. For more detail about 2010): 21701–6, doi:10.1073/pnas.1006219107. 85. Peter Schulte et al., “The Chicxulub Asteroid Impact and in Ferrets,” Nature 486, no. 7403 (June 21, 2012): 420–28, the surveys see Vincent C. Müller and Nick Bostrom, Mass Extinction at the Cretaceous-Paleogene Boundary,” doi:10.1038/nature10831. 75. Sands, Mundaca-Shah, and Dzau, “The Neglected “Future Progress in Artificial Intelligence: A Survey of Science 327, no. 5970 (March 5, 2010): 1214–18, Dimension of Global Security — A Framework for 103. Nicholas Greig Evans, Marc Lipsitch, and Meira Expert Opinion,” in Fundamental Issues of Artificial doi:10.1126/science.1177265. Countering Infectious-Disease Crises.” Levinson, “The Ethics of Biosafety Considerations in Intelligence, ed. Vincent C. Müller (Berlin: Synthese Gain-of-Function Research Resulting in the Creation 76. Edwin Dennis Kilbourne, “Plagues and Pandemics: Past, 86. National Research Council (U. S.). Committee to Review Library; Springer, 2014). of Potential Pandemic Pathogens,” Journal of Medical Present, and Future,” in Global Catastrophic Risks, ed. Near-Earth-Object Surveys and Hazard Mitigation Ethics 41, no. 11 (November 1, 2015): 901–8, doi:10.1136/ 123. Bostrom, Superintelligence, 18–21 Nick Bostrom and Milan M. Ćirković (Oxford: Oxford Strategies, Defending Planet Earth: Near-Earth Object medethics-2014-102619. 124. Bostrom, Superintelligence, 20–21. University Press, 2008); William Rosen, Justinian’s Surveys and Hazard Mitigation Strategies (Washington, Flea : Plague, Empire, and the Birth of Europe (New DC: National Academies Press, 2010), 21. 104. Richard A. Posner, Catastrophe : Risk and Response 125. Shepherd, Geoengineering the Climate, chap. 2. (Oxford: Oxford University Press, 2004), 78–79. York: Viking, 2007); Niall Johnson and Juergen Mueller, 87. National Research Council (U. S.). Committee to Review 126. Shepherd, Geoengineering the Climate. “Updating the Accounts: Global Mortality of the 1918- Near-Earth-Object Surveys and Hazard Mitigation 105. George Church, “A Synthetic Biohazard Non-Proliferation 127. Alan Robock, Kirsten Jerch, and Martin Bunzl, “20 Reasons 1920 ‘Spanish’ Influenza Pandemic,” Bulletin of the Strategies, Defending Planet Earth, 21. Proposal,” 2004, http://arep.med.harvard.edu/SBP/ Why Geoengineering May Be a Bad Idea,” Bulletin of the History of Medicine 76, no. 1 (2002): 105–15; Ole J. Church_Biohazard04c.htm. 88. Clark R. Chapman, “The Hazard of near-Earth Asteroid Atomic Scientists 64, no. 2 (2008): 14–59. Benedictow, “The Black Death: The Greatest Catastrophe Impacts on Earth,” Earth and Planetary Science 106. Nathan Myhrvold, “Strategic Terrorism: A Call to Action,”

102 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 103 ENDNOTES

128. Shepherd, Geoengineering the Climate, chap. 3. For “Climate Change–a Risk Assessment,” 38; International 158. For discussions of worries about arms races and emerging 169. Clean air can of course also be influenced by pollution in criticism of the ‘climate emergency’ argument for Energy Agency and Nuclear Energy Agency, “Technology technologies see Christopher F. Chyba, “Biotechnology and neighbouring countries. SRM see Oliver Morton, The Planet Remade : How Roadmap: Nuclear Energy,” 2015; Ottmar Edenhofer and the Challenge to Arms Control,” Arms Control Today 36, 170. Elinor Ostrom, Governing the Commons: the evolution Geoengineering Could Change the World (London: Intergovernmental Panel on Climate Change. Working no. 8 (2006), http://search.proquest.com/openview/3251 of institutions for collective action. 1990. Cambridge Granta, 2015), chap. 6; Jana Sillmann et al., “Climate Group III, Climate Change 2014 : Mitigation of Climate 1e461cdf35678bea5679269ac45b/1?pq-origsite=gscholar; University Press. Emergencies Do Not Justify Engineering the Climate,” Change : Working Group III Contribution to the Fifth Future of Life Institute, “Open Letter on Autonomous Nature Climate Change 5, no. 4 (2015): 290–92. Assessment Report of the Intergovernmental Panel on Weapons,” accessed March 18, 2016, http://futureoflife. 171. It is hotly disputed how we should go about taking the interests of future generations into account, but it 129. Scott Barrett, “The Incredible Economics of Climate Change (New York, NY: Cambridge University org/open-letter-autonomous-weapons/; Gernot Wagner Press, 2015), 516–518. and Martin L. Weitzman, Climate Shock : The Economic seems plausible that they should be given at least some Geoengineering,” Environmental and Resource consideration in public policy. Indeed, this is a standard Economics 39, no. 1 (2008): 45–54. 146. Edenhofer and Intergovernmental Panel on Climate Consequences of a Hotter Planet (Princeton: Princeton University Press, 2015), chap. 3. assumption in many governmental cost-benefit analyses. 130. Alan Robock, Luke Oman, and Georgiy L. Stenchikov, Change. Working Group III, Climate Change 2014, 531. 159. Nick Bostrom, Superintelligence : Paths, Dangers, 172. Mancur Olson, The Logic of Collective Action : Public “Regional Climate Responses to Geoengineering 147. According to Sailor et al “development of nuclear weapons Goods and the Theory of Groups, Harvard Economic with Tropical and Arctic SO2 Injections,” Journal of has been aided in at least three countries (India, Iraq, and Strategies (Oxford: Oxford University Press, 2014); Ali Nouri and Christopher F. Chyba, “Biotechnology and Studies ; v. 124 (Cambridge, Mass: Harvard University Geophysical Research: Atmospheres 113, no. D16 Israel) by use of research reactors obtained under the Press, 1965). (August 27, 2008): D16101, doi:10.1029/2008JD010050. cover of peaceful research programs”.William C. Sailor et Biosecurity,” in Global Catastrophic Risks, ed. Nick Bostrom and Milan M. Ćirković (Oxford University Press, 173. This is an example of availability bias, which is 131. Morton, The Planet Remade, chap. 4.. al., “A Nuclear Solution to Climate Change?,” Science 288, no. 5469 (2000): 1177–78. For a very good overview of the 2008). discussed along with other cognitive biases that can 132. Shepherd, Geoengineering the Climate, chap. 4. technical and political aspects of the impact of nuclear 160. For a discussion of this issue with respect to skew perceptions of global risks in Eliezer Yudkowsky, 133. Ibid., chap. 3. power on nuclear proliferation see A. Adamantiades and biotechnology and geo-engineering risks respectively see “Cognitive Biases Potentially Affecting Judgment of Global I. Kessides, “Nuclear Power for Sustainable Development: W. Paul Duprex et al., “Gain-of-Function Experiments: Risks,” in Global Catastrophic Risks, ed. Nick Bostrom 134. Morton, The Planet Remade, chap. 4. Current Status and Future Prospects,” Energy Policy Time for a Real Debate,” Nature Reviews Microbiology 13, and Milan M. Ćirković, vol. 1 (Oxford: Oxford University 135. Seth D. Baum, Timothy M. Maher Jr, and Jacob Haqq- 37, no. 12 (December 2009): 5149–66, doi:10.1016/j. no. 1 (January 2015): 58–64, doi:10.1038/nrmicro3405; Press, 2008). Misra, “Double Catastrophe: Intermittent Stratospheric enpol.2009.07.052. Oliver Morton, The Planet Remade : How Geoengineering 174. Barrett, Why Cooperate?, chap. 2. Geoengineering Induced by Societal Collapse,” 148. Edwin Dennis Kilbourne, “Plagues and Pandemics: Past, Could Change the World (London: Granta, 2015). Environment Systems & Decisions 33, no. 1 (January 8, 175. WHO, “Summary of States Parties 2013 Report on IHR Present, and Future,” in Global Catastrophic Risks, ed. 161. Bostrom, Superintelligence. 2013): 168–80, doi:10.1007/s10669-012-9429-y. Core Capacity Implementation,” 2014, 6, http://www.who. Nick Bostrom and Milan M. Ćirković (Oxford: Oxford 162. Bostrom, N. “Analyzing Human Extinction Scenarios and int/ihr/publications/WHO_HSE_GCR_2014.10/en/. 136. Shepherd, Geoengineering the Climate, chap. 4. University Press, 2008). Related Hazards”. Journal of Evolution and Technology. 176. Lawrence O. Gostin and Devi Sridhar, “Global Health and 137. See for example Shepherd, Geoengineering the Climate; 149. James M. Hughes et al., “The Origin and Prevention of Vol. 9. No. 1. (2002) the Law,” New England Journal of Medicine 370, no. 18 National Academy of Sciences, Climate Intervention: Pandemics,” Clinical Infectious Diseases 50, no. 12 (June 163. Philip Nel and Marjolein Righarts, “Natural Disasters and (May 1, 2014): 1732–40, doi:10.1056/NEJMra1314094. Reflecting Sunlight to Cool Earth (Washington, D.C.: 15, 2010): 1636–40, doi:10.1086/652860. the Risk of Violent Civil Conflict,” International Studies National Academies Press, 2015), http://www.nap.edu/ 177. Barrett, Why Cooperate? 150. Kate E. Jones et al., “Global Trends in Emerging Infectious Quarterly 52, no. 1 (March 1, 2008): 159–85, doi:10.1111/ catalog/18988; Solar Radiation Management Governance 178. Treaty on the Non-Proliferation of Nuclear Weapons, Diseases,” Nature 451, no. 7181 (February 21, 2008): j.1468-2478.2007.00495.x. Initiative et al., Solar Radiation Management: The Washington/Moscow/London, 1 July 1968, in force 5 990–93, doi:10.1038/nature06536. Governance of Research (SRMGI, 2012). 164. For an overview see Halvard Buhaug, Nils Petter March 1970, 729 UNTS 161 151. Peter Sands, Carmen Mundaca-Shah, and Victor J. Gleditsch, and Ole Magnus Theisen, “Implications of 138. Shepherd, Geoengineering the Climate; Barrett, “The 179. United Nations Framework Convention on Climate Dzau, “The Neglected Dimension of Global Security — A Climate Change for Armed Conflict” (World Bank, 2009); Incredible Economics of Geoengineering.” Change, Rio de Janeiro, 9 May 1992, in force 21 March Framework for Countering Infectious-Disease Crises,” Ragnhild Nordås and Nils Petter Gleditsch, “Climate 1994, 1771 UNTS 107. This convention provided the 139. Arnon Dar, “Influence of Supernovae, Gamma-Ray New England Journal of Medicine 0, no. 0 (January 13, Change and Conflict,” Political Geography, Climate platform for the Kyoto and Paris climate change Bursts, Solar Flares, and Cosmic Rays on the Terrestrial 2016): null, doi:10.1056/NEJMsr1600236. Change and Conflict, 26, no. 6 (August 2007): 627–38, Environment,” in Global Catastrophic Risks, ed. Nick agreements. 152. David Charles Denkenberger and Joshua Pearce, Feeding doi:10.1016/j.polgeo.2007.06.003. Bostrom and Milan M. Ćirković (Oxford: Oxford 180. The IAEA’s mandate is limited to civil nuclear applications, Everyone No Matter What : Managing Food Security after 165. BBC, “Working Gun Made with 3D Printer,” BBC News, University Press, 2008). but it has probably reduced the likelihood of nuclear Global Catastrophe (Amsterdam: Academic Press, 2015). accessed February 22, 2016, http://www.bbc.co.uk/news/ 140. Bryan Caplan, “The Totalitarian Threat,” in Global terrorism by imposing detailed nuclear fissile accounting. 153. S. Sparks et al., Super-Eruptions: Global Effects and science-environment-22421185. Catastrophic Risks, ed. Nick Bostrom and Milan M. This verification process as well as on-site inspection also Future Threats. London, Report of a Geological Society of 166. See Chapter 2 and Alan Robock, Luke Oman, and Ćirković (Oxford: Oxford University Press, 2008). make it very difficult for non-nuclear-weapon states to London Working Group, 2005; National Research Council Georgiy L. Stenchikov, “Nuclear Winter Revisited with a develop nuclear weapons without detection. 141. Seth D. Baum, “Winter-Safe Deterrence: The Risk of (U. S.). Committee to Review Near-Earth-Object Surveys Modern Climate Model and Current Nuclear Arsenals: 181. United Nations General Assembly, “Protection of Global Nuclear Winter and Its Challenge to Deterrence,” and Hazard Mitigation Strategies, Defending Planet Still Catastrophic Consequences,” Journal of Geophysical Climate for Present and Future Generations of Mankind,” Contemporary Security Policy 36, no. 1 (January 2, 2015): Earth: Near-Earth Object Surveys and Hazard Mitigation Research: Atmospheres 112, no. D13 (July 16, 2007): December 1988, http://www.ipcc.ch/docs/UNGA43-53.pdf. 123–48, doi:10.1080/13523260.2015.1012346. Strategies (Washington, DC: National Academies Press, D13107, doi:10.1029/2006JD008235; National Research 142. Joseph Cirincione, “The Continuing Threat of Nuclear 2010), chap. 2 and 5. Council (U. S.). Committee to Review Near-Earth-Object 182. Gernot Wagner and Martin L. Weitzman, Climate Shock : The Economic Consequences of a Hotter Planet War,” in Global Catastrophic Risks, ed. Nick Bostrom and 154. David King et al., “Climate Change–a Risk Assessment” Surveys and Hazard Mitigation Strategies, Defending (Princeton: Princeton University Press, 2015). Milan M. Ćirković (Oxford University Press, 2008). (Centre for Science Policy, University of Cambridge, Planet Earth: Near-Earth Object Surveys and Hazard Mitigation Strategies (Washington, DC: National 143. Pavel Podvig, “Reducing the Risk of an Accidental 2015), pt. 1, www.csap.cam.ac.uk/projects/climate- 183. See for example Alan Robock, “Nuclear Winter,” Wiley Academies Press, 2010), chap. 2; S. Sparks et al., Super- Launch,” Science & Global Security 14, no. 2–3 (December change-risk-assessment/. Interdisciplinary Reviews: Climate Change 1, no. 3 (May 1, Eruptions: Global Effects and Future Threats. London, 1, 2006): 75–115, doi:10.1080/08929880600992990. 2010): 418–27, doi:10.1002/wcc.45. 155. Sybren Drijfhout et al., “Catalogue of Abrupt Shifts in Report of a Geological Society of London Working Group, 184. Nick Bostrom, Superintelligence : Paths, Dangers, 144. For an overview of the literature on ‘nuclear winter’ Intergovernmental Panel on Climate Change Climate 2005. see Seth D. Baum, “Winter-Safe Deterrence: The Risk Models,” Proceedings of the National Academy of Strategies (Oxford: Oxford University Press, 2014). 167. Denkenberger and Pearce, Feeding Everyone No Matter of Nuclear Winter and Its Challenge to Deterrence,” Sciences 112, no. 43 (2015): E5777–86 185. For example, see the discussion of the responsibility What. Contemporary Security Policy 36, no. 1 (January 2, 2015): 156. For a qualified defence of this position see Oliver Morton, of research laboratories conducting gain of function 123–48, doi:10.1080/13523260.2015.1012346. The The Planet Remade : How Geoengineering Could Change 168. For an excellent discussion of global public goods see research in W. Paul Duprex et al., “Gain-of-Function resilience of the food supply is also an important risk the World (London: Granta, 2015). Scott Barrett, Why Cooperate? : The Incentive to Supply Experiments: Time for a Real Debate,” Nature factor, and is discussed in section 3.2. Global Public Goods (Oxford: Oxford University Press, Reviews Microbiology 13, no. 1 (January 2015): 58–64, 157. J. G. Shepherd, Geoengineering the Climate: Science, 2007). doi:10.1038/nrmicro3405. 145. Centre for Science Policy, University of Cambridge et al., Governance and Uncertainty (Royal Society, 2009).

104 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 105 ENDNOTES ACKNOWLEDGEMENTS

186. Charles Arthur, “DeepMind: ‘Artificial Intelligence Is a Tool 195. The WHO’s International Health Regulations were an That Humans Can Control and Direct,’” The Guardian, important development in this area, but the rules and June 9, 2015, sec. Technology, http://www.theguardian. their implementation could be improved. See Rebecca com/technology/2015/jun/09/deepmind-artificial- Katz and Scott F Dowell, “Revising the International intelligence-tool-humans-control. Health Regulations: Call for a 2017 Review Conference,” Acknowledgements 187. Anja Wolz, “Face to Face with Ebola — An Emergency The Lancet Global Health 3, no. 7 (July 2015): e352–53, Care Center in Sierra Leone,” New England Journal of doi:10.1016/S2214-109X(15)00025-X. his report has greatly For very helpful advice on indi- Medicine 371, no. 12 (September 18, 2014): 1081–83, 196. Ibid. benefited from the advice vidual risks, we would like to thank doi:10.1056/NEJMp1410179. 197. See National Research Council (U. S.). Committee to 188. T. Frängsmyr, and I. Abrams, I., 1997. Peace, 1981-1990 Review Near-Earth-Object Surveys and Hazard Mitigation and feedback of many Dan Bernie, Sean Brocklebank, Clark (Vol. 5). World Scientific. Strategies, Defending Planet Earth, chap. 2. people. For detailed com- Chapman, Daniel Dewey, Simon 189. National Research Council (U. S.). Committee to Review 198. Sparks et al., Super-Eruptions.. ments on a large portion Driscoll, Marius Gilbert, David Graff, Near-Earth-Object Surveys and Hazard Mitigation 199. See the discussion in Duprex et al., “Gain-of-Function Strategies, Defending Planet Earth: Near-Earth Object Experiments.”. Tof the report, we are very grateful Marc Lipsitch, Mark Lynas, Piers Surveys and Hazard Mitigation Strategies (Washington, DC: National Academies Press, 2010), chap. 2. 200. Nouri and Chyba, “Biotechnology and Biosecurity.” to Niel Bowerman, Martina Kunz, Millet, Oliver Morton, Andrew Parker, 190. For a defence of the importance of tail risk climate 201. Bostrom, Superintelligence. Catherine Rhodes. For their sugges- Alan Robock, Jonathan Skaff, Ste- change see Martin L. Weitzman, “A Review of ‘The Stern 202. Shepherd, Geoengineering the Climate. tions and comments we are grateful phen Sparks, and Naomi Standen. Review on the Economics of Climate Change”, Journal of 203. For discussion of different particulate winter scenarios Economic Literature 45, no. 3 (2007): 703–24. see Timothy M. Maher and Seth D. Baum, “Adaptation to to Seán Ó’heigeartaigh, Shahar Avin, 191. Wagner and Weitzman, Climate Shock, chap. 3. and Recovery from Global Catastrophe,” Sustainability 5, and Toby Ord. 192. See Meinshausen et al., “Greenhouse-Gas Emission no. 4 (March 28, 2013): 1461–79, doi:10.3390/su5041461. Targets for Limiting Global Warming to 2 °C.”. 204. Ibid. 193. On this see Pavel Podvig, “Reducing the Risk 205. See for example David Charles Denkenberger and Joshua of an Accidental Launch,” Science & Global Pearce, Feeding Everyone No Matter What : Managing Security 14, no. 2–3 (December 1, 2006): 75–115, Food Security after Global Catastrophe (Amsterdam: doi:10.1080/08929880600992990. Academic Press, 2015). 194. On this see Seth D. Baum, “Confronting the Threat 206. Nick Bostrom, “Existential Risk Prevention as Global of Nuclear Winter,” Futures, Confronting Future Priority,” Global Policy 4, no. 1 (February 1, 2013): 15–31, Catastrophic Threats To Humanity, 72 (September 2015): doi:10.1111/1758-5899.12002. 69–79, doi:10.1016/j.futures.2015.03.004.

CONTACT INFO Authors: Global Priorities Project 16-17 St. Ebbes Street Oxford, OX1 1PT United Kingdom

[email protected]

The Global Challenges Foundation: Strandvägen 7A 114 56 Stockholm Sweden

info@ globalchallenges.org +46 (0) 709 98 97 97

106 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 107 108 Global Catastrophic Risks 2016