sign in sign up
<<
Home , Biosecurity, Bioterrorism, Chemical warfare, Climate change, Eliezer Yudkowsky, Extinction event

Global Challenges Foundation

Global Catastrophic 2016

© Global Challenges Foundation/Global Priorities Project 2016

GLOBAL CATASTROPHIC RISKS 2016 The views expressed in this report are those of the authors. Their statements are not necessarily endorsed by the affiliated organisations.

Authors: Owen Cotton-Barratt*† Sebastian Farquhar* John Halstead* Stefan Schubert* Andrew Snyder-Beattie†

* = The Global Priorities Project † = The Future of Humanity Institute,

Graphic design: Accomplice/Elinor Hägg

Global Challenges Foundation in association with

4 Global Catastrophic Risks 2016 THE GLOBAL CHALLENGES FOUNDATION works to raise awareness of the Global Catastrophic Risks. Primarily focused on change, other en- vironmental degradation and politically motivated as well as how these threats are linked to and rapid growth. Against this background, the Foundation also works to both identify and stimulate the development of good proposals for a management model – a global gover- nance – able to decrease – and at best eliminate – these risks.

THE GLOBAL PRIORITIES PROJECT helps decision-makers effectively prior- itise ways to do good. We achieve his both by advising decision-makers on programme evaluation methodology and by encouraging specific . We are a collaboration between the Centre for Effective and the Future of Humanity Institute, part of the University of Oxford.

Global Catastrophic Risks 2016 5 Contents

Foreword 8 Introduction 10 Executive summary 12 1. An introduction to global catastrophic risks 20 2. What are the most important global catastrophic risks? 28 Catastrophic 30 Nuclear 36 Natural 42 Exogenous risks 46 Emerging risks 52 Other risks and unknown risks 64 Our assessment of the risks 66 3. factors and interactions between risks 72 Drivers of individual risks 74 Shared risk factors and interactions between risks 78 4. Do institutions collectively underinvest in ? 82 Market and political failures 84 Which actors can help reduce global catastrophic risk? 88 5. What can the do to reduce global catastrophic risk? 94 Endnotes 100 Acknowledgements 107 Contact info 107

6 Global Catastrophic Risks 2016 Definition: Global Catastrophic Risk – risk of events or processes that would lead to the deaths of approximately a tenth of the world’s population, or have a comparable impact.

Global Catastrophic Risks 2016 7 FOREWORD

Dear Reader!

early four ago when the Global Challenges Foundation was established, we decided on a direction Nwith two parallel strategies. The first is increasing the knowledge about Global Catastrophic Risks (GCRs), which with our terminology means threats that can eliminate at least 10% of the global population. This knowl- edge is an important prerequisite for the Foundation’s second strategy: to encourage debates and proposals as to how we can effectively and fairly reduce – and preferably eliminate – these catastrophic risks. This publication, the Foundation’s Annual Report for 2016, is the re- this. In addition to the risks involved sult of a collaboration between the in the Annual Report for 2016, the Foundation and the Future of Hu- Foundation actively works with envi- manity Institute (FHI) and the Global ronmental degradation, of Priorities Project at Oxford University mass destruction, in the U.K., which has now lasted for (that exacerbates several risks), and over two years. A big group of re- political violence which is behind searchers at the FHI, commissioned many of the world’s current problems. by the Foundation, summarized Political violence comes in many where research, focused on charting forms. Various kinds of weapons of some of the greatest global risks, cur- mass destruction represent poten- rently stands. tially devastating weaponry. Further, In addition to describing the risks, political violence creates uncon- their effects and their likelihood of trolled migration and we receive occurring, this ’s Annual Report repeated reminders that there is takes one step further and try’s to also “digital violence” in the form of show how different risks relate to one cyber-attacks. Together, this takes up another, what can be done to combat a significant amount of space on the the risks and who can and should do political agenda, thus stealing atten-

8 Global Catastrophic Risks 2016 tion from other important risks. And in favor of one or more organizations above all, the defense against various that have a mandate to decide on how forms of political violence requires to mitigate GCRs. a grotesquely large share of public Would this be ? My counter resources. Each day, the world spends question is whether there are any over SEK 40 billion on defence alternatives? To continue relying on expenditure – money that would be multilateral negotiations increases needed to fight poverty and prevent the that decisions and catastrophic risks. actions are insufficient and executed My personal opinion is that in order too late. This means that the likeli- to drastically minimize GCRs we hood of GCRs continues to escalate. must develop a model where a major- I hope that this publication can ity of the world’s nations, with strong deepen the understanding of GCRs support from leading nations, can and that these insights provide a make binding decisions which can fertile ground for both debates and be enforced in an effective and fair proposals on how we can develop a way. This would imply that individ- better way of managing and address- ual nations waive their sovereignty ing these risks.

Stockholm, April 2016

Laszlo Szombatfalvy Founder of Global Challenges Foundation

Global Catastrophic Risks 2016 9 INTRODUCTION

Global catastrophic risks pose a pressing challenge

his report addresses one of the most important issues of our age – global catastrophic risk. Over the last decades, behav- Tioural psychology has taught us that, as a , we are bad at assessing scope. Issues that affect ten people do not intuitively seem ten times more important than those that affect one person. Global catastrophic risks are one area where our scope insensitivi- ty might prove the most dangerous. These risks can’t just be treated as problem for the future, even though we might well expect them not to ma- terialise this year or the next. At the Future of Institute, my team and of which has inspired my own deep I have been calling for global leaders curiosity in . to address critical global risk issues This technological power is an including nuclear weapons, biotech- enormous force for good, but carries nology and . This its own risks. Although consuming builds on existing risk reduction work was critical in creating the led by institutions such as the United thriving and wonderful Nations. we live in today, we’ come to learn Over the last centuries, humanity that there are potentially catastroph- has achieved incredible things. New ic long-term consequences from medical technologies save millions of climate change. Other technologies, every year. Agricultural more powerful than combustion en- allows billions to be fed who might gines, might also offer huge benefits otherwise not exist. And we have and carry unforeseen risks. If we fail begun to explore the very foundations to manage this risk well, we might be of our itself – the beauty caught out by consequences that fol-

10 Global Catastrophic Risks 2016 low from the technology more rapidly means being realistic in weighing than climate change has. these factors against each other. This As a global community, we need report offers an excellent background to win the race between the growing to the underlying issues of global power of our technology and the catastrophic risks, and is an outstand- wisdom with which we manage it. ing starting point for -makers This requires a nuanced approach developing an interest in the area or towards technological developments, researchers considering how their acknowledging both that technol- own work might be brought into the ogy carries huge potential to make study of global catastrophic risks. lives better and also that it carries some risks. Smart risk management

Max Tegmark Co-founder of the Professor of at MIT

Global Catastrophic Risks 2016 11 EXECUTIVE SUMMARY

Executive Summary

ost generations generations. Reducing these risks is never experience a therefore both a global and an inter- global catastrophe. generational public good. However, the idea of The ever-evolving landscape of such catastrophes technology and compounds Mis not fanciful: plagues have killed these challenges. Technological over 10% of world’s population and and economic forces can create new we came close to nuclear war several global catastrophic risks, such as times in the . anthropogenic climate change and Despite their scale, the risks of the 20th century’s nuclear . global catastrophes receive limited But technology can also reduce risk, attention. One reason is that many for example through better vaccines of these risks are unlikely in any or clean energy. given decade. But even when the We believe the global community probability is low, the sheer magni- should work together to harness new tude of an adverse outcome warrants tools to address global catastrophic taking these risks seriously. A global risks. It is possible that, collectively, catastrophic risk not only threatens we significantly under-invest in glob- everyone alive today, but also future al catastrophic risk reduction.

12 Global Catastrophic Risks 2016 We believe the global community should work together to harness new tools to address global catastrophic risks.

Global Catastrophic Risks 2016 13 EXECUTIVE SUMMARY

What are the biggest threats?

he global catastrophic risks in this report can be divided into T two categories. Some are on- going and could potentially occur in any given year. Others are emerging and may be very unlikely today but will become significantly more likely in the coming decades. The most significant ongoing risks are natural pandemics and nuclear war, whereas the most significant emerging risks are catastrophic climate change and risks stemming from emerging tech- nologies. Even where risks remain in the future, there are things we can do today to address them. The Spanish of 1918 may have killed as much as 5% of the . Some outbreaks since then infected over a third of the world’s population (e.g., pandemic influenza), whereas others killed over half of people infected (e.g., or SARS). If a were to emerge that was as transmis- sible as the flu and as lethal as Ebola, the results could be catastrophic. Fortunately, this rarely transpires, but it is possible that it could, for ex- ample with the H5N1 influenza . The invention of nuclear weap- ons ushered in a new era of risks created by action. A large nuclear war between major powers would likely kill tens or hundreds of millions in the initial conflict, and perhaps many more if a nuclear win- ter were to follow. During the Cuban

14 Global Catastrophic Risks 2016

EXECUTIVE SUMMARY

Missile Crisis, President Kennedy from what is expected or if positive estimated the chance of nuclear environmental loops occur. conflict as “between one in three and Catastrophic risks from emerging even”. Tensions have eased some- technology are less well understood. what since the , but could promise sig- recur. Moreover, accidents or mis- nificant benefits, but a handful could calculation with nuclear weapons also create unprecedented risks to continue to pose a risk. civilisation and the alike. Climate change is a well-known could enable the crea- anthropogenic risk. Even if we suc- tion of far more damaging ceed in limiting emissions, scientists than those found in , while in expect significant climate change the longer run, artificial intelligence to occur. This could bring a host of could cause massive disruption. global challenges including environ- The relative likelihood and ur- mental degradation, migration, and gency of the different risks matters the possibility of resource conflict. when deciding how to respond. Even But this is not the worst-case sce- though the level of is nario. Although it receives far less extreme, rational action requires attention, scientists also acknowl- explicit assessments of how much edge the possibility of catastrophic attention the different risks deserve, climate change. There is a small and how likely they are. The views of likelihood that warming could even the authors on these vexed ques- exceed 6 °C, leaving large swathes of tions, based on our reading of the the planet dramatically less habita- scientific evidence, are summarised ble. This could occur if emissions are in the following table. More informa- not cut sufficiently, if the sensitivity tion can be found in Chapter 2. of the is different

16 Global Catastrophic Risks 2016 FIGURE 1. OUR ASSESSMENT OF GLOBAL CATASTROPHIC RISKS

Natural pandemic

Unknown risks Nuclear war

Engineered pandemic Higher likelihood over next 5 years

Catastrophic Asteroid impact climate change

Supervolcanic Catastrophic eruption disruption from AI

Failure of geo-engineering Lower likelihood over next 5 years Low High APPROPRIATE LEVEL OF ATTENTION EXECUTIVE SUMMARY

Humanity can respond to these risks

or each of the risks in this To reduce the risk of global ca- report, we consider actions tastrophe caused by nuclear war: F available to avoid or mitigate • The international community the risk, and which actors are best- should continue the policy of placed or responsible for taking that nuclear non-proliferation, and nu- action. For the most significant risks, clear states can continue to reduce some of the most promising opportu- stockpiles. nities are listed here. • Nuclear- states should con- tinue to work to reduce the chance To reduce the risk of global ca- of accidental launch or escalation. tastrophe caused by pandemic: • The World Organisation, To reduce the risk of global ca- nation states, and other bodies tastrophe caused by emerging should increase their planning for technologies: extremely bad pandemics. • Research communities should • The community further investigate the possible should improve developing world risks from emerging capabilities in capacity for response, for example biotechnology and artificial intel- by ensuring that vaccine produc- ligence, and possible solutions. tion facilities are well-distributed • Policymakers could work with around the world. researchers to understand the issues that may arise with these To reduce the risk of global ca- new technologies, and start to lay tastrophe caused by climate groundwork for planned adaptive change: risk . • Research communities should in- crease their focus on understand- To reduce global catastrophic risk ing the pathways to and likelihood in a cross-cutting way: of catastrophic climate change, • Research communities should fo- and possible ways to respond. cus greater attention on strategies • Nations should continue to imple- and technologies for resilience to ment and improve mechanisms and recovery from global catastro- for emissions abatement such as phe, for example by developing taxes or tradable emissions alternate sources. quotas, as for non-catastrophic • Nations should work to incorpo- climate change. rate the interests of future gener- ations into their decision-making frameworks.

18 Global Catastrophic Risks 2016 Research communities should further investigate the possible risks from emerging capabilities in biotechnology and artificial intelligence, and possible solutions. CHAPTER 1 – AN INTRODUCTION TO GLOBAL CATASTROPHIC RISKS

Chapter 1 An Introduction to Global Catastrophic Risks

ver the course of , times been uncomfortably high. the world has suffered Throughout the Cold War, the of such mag- threat of loomed nitude that human civ- large. The and the So- ilisation itself has been viet Union possessed tens of thou- Othreatened. Warfare and pandem- sands of high yield nuclear , ics have caused especially significant and their retaliatory strike systems damage. Originating in 541-542, the were programmed to respond to any initial outbreak of the ‘Great Plague of attack within minutes. The world has Justinian’ killed 25-33 million people come close to the nightmare scenar- – between 13% and 17% of the world io on a number of occasions. Perhaps population at the time.1 The plague the narrowest escape came on 27th had trans-generational consequenc- October 1962. Two Russian B-59 sub- es: many historians believe that it marine commanders off the weakened the at of gave the order to launch a a crucial time, undermining its at- nuclear strike against the United tempts to reconquer .2 States, on the mistaken assump- In recent times, humanity tion that war had already start- has not endured catastroph- ed. The launch of a nuclear ic events on the propor- As torpedo required the con- tionate scale of Plague of many sent of all three officers Justinian. However, the as on board; the second risk of global catastro- in command, Vasili phe, which is deter- Arkhipov, was alone mined by the po- in refusing permis- 17% 3 tential damage of sion. of the world’s population was the event and While the nu- killed in the initial outbreak of the its probabili- ‘Great ’. That equals clear threat ty of occur- 25-33 million people has receded ring, has at since the

20 Global Catastrophic Risks 2016 The most deadly event of the 20th century was probably the Spanish of 1918-1920 which killed between 2.5% and 5% of the world population… Our focus here is on even more extreme possibilities which receive less attention.

end of the Cold War, the risk remains. machine intelligence which could be- In addition, ongoing economic and have in a manner incompatible with technological developments bring, human values. alongside their benefits, a range of However, our and in- new unprecedented anthropogen- stitutions, whose primary focus is ic risks: for example, catastrophic understandably on more day-to-day climate change, pandemics of glob- concerns, may systematically be ne- al proportions, and the potential for glecting global catastrophic risks.

Global Catastrophic Risks 2016 21 CHAPTER 1 – AN INTRODUCTION TO GLOBAL CATASTROPHIC RISKS

1.1. Defining Global Catastrophic Risk

lobal catastrophes are events None of the various humanitarian or processes that would inflict disasters of the 20th century killed G serious damage to humanity more than 10% of the world popula- on a global scale, such as all-out nu- tion. Around 1% of the world popu- clear war or a pandemic killing hun- lation died in the First , dreds of millions. The severity of a while up to 3% died in the Second risk is a function of its scope (the size World War.5 The most deadly event of the population at risk), intensity of the 20th century was probably the (how badly this population would be Spanish influenza pandemic of 1918- affected), and probability (how likely 1920 which killed 50 - 100 million the is to occur).4 people – between 2.5% and 5% of the A fatal crash is a personal ca- world population.6 Although these tastrophe: a small number of unfor- were huge tragedies, our focus here tunate victims suffer a severe harm. is on even more extreme possibilities Most are examples of local which receive less attention. In Chap- catastrophes: thousands or millions ter 2 we give an overview of the ma- of people within a country or region jor risks of global catastrophe, and in lose their lives. The focus of this re- Chapter 3 we look at the factors which is catastrophic risks with global may increase or decrease these risks. scope. We define a global catastrophe Limited historical evidence makes as a possible event or process that, it very difficult to provide a definitive were it to occur, would end the lives list of past global catastrophes. There of approximately 10% or more of the have been at least two in the past two global population, or do comparable millennia – the Plague of Justinian damage. risks are a sub- and the . Some scholars set of global catastrophic risks, which have argued that more than 10% of would end the human race. the world population lost their lives It is important to put the scale of in pre-industrial , though this is global catastrophic risks in context. heavily disputed.7

22 Global Catastrophic Risks 2016 SCOPE FIGURE 1.1.QUALITATIVERISKCATEGORIES Personal Local Global in regionalairpollution Minute increase Loss ofonehair Global warming Imperceptible by 0.001C°

Recession inone Depletion ofthe Loss ofonearm layer Damaging country

8 Fatal carcrash Catastrophic Nuclear war National

SEVERITY CHAPTER 1 – AN INTRODUCTION TO GLOBAL CATASTROPHIC RISKS

1.2. Why Global Catastrophic Risks matter

lthough the chance of dy- Now let’s suppose that the chance ing in a car crash is small, we of extinction were 0.1% per year and A each take steps to mitigate consider the consequences. It may the risk such as wearing seat belts and seem at first glance that this would driving safely. National governments be an acceptable level of risk. How- take steps to mitigate the risk of rare ever, that would mean an individual natural disasters, such as would be more than five times as like- and hurricanes. Similarly, it is impor- ly to die in an than a tant that the global community works car crash.11 Moreover, these small an- to reduce the risk of catastroph- nual add up, so that the ic events which would have a global chance of extinction within the next scope. century under this scenario is 9.5%.12 The probabilities of these cat- A global catastrophe, which involves astrophic events are low but not the death of 10% of the global popu- negligible. Moreover, small annual lation, is more likely than an event probabilities compound significantly that involves . As a over the long term. result, even if 0.1% were on the high We do not know of a robust estimate side for extinction risk, it might be of of the annual probability of global the appropriate order of magnitude catastrophic risk. Nor do we believe for global catastrophic risk. that we are able to create a robust Reducing these risks has obvious estimate because the in humanitarian benefits for those alive key parameters are so large. However, today. But we should also consider for extinction risks some experts have the welfare of future generations. A suggested that a 0.1% annual chance global catastrophe could reduce the of extinction is within the range of standards of living for many genera- plausible orders of magnitude. A 2008 tions to come, while outright human Oxford survey of expert judgement on extinction denies existence to all fu- the topic implied an average annual ture generations. Many leading mor- extinction risk over the next centu- al philosophers have argued that the ry of around 0.2%.9 The UK’s Stern welfare of these future generations is Review on the of Climate of utmost importance.13 Change used 0.1% as an upper bound Global catastrophic risks are also modeling assumption for annual likely to be politically neglected.14 extinction risk. 10 Global catastrophic risk reduction is

24 Global Catastrophic Risks 2016 The probabilities of these catastrophic events are low but not negligible. Moreover, small annual probabilities compound significantly over the long term.

a , as even a large beneficiaries are future people who country would only capture a small have no voice in the political process. portion of the total benefit of risk mit- For these reasons and others, nation- igation. Moreover, it is an intergen- al and international actors are likely erational public good, as many of the to underinvest in risk reduction.

Global Catastrophic Risks 2016 25 CHAPTER 1 – AN INTRODUCTION TO GLOBAL CATASTROPHIC RISKS

1.3. Why Global Catastrophic Risks are especially relevant today

ow more than ever before, most important future risks may be at global catastrophic risks de- present unknown. Just as in the early N serve attention. Prior to the 20th century it would have been im- 20th century, the main global cata- possible to predict nuclear weapons, strophic risks that humankind faced catastrophic climate change, or bio- were natural pandemics and conven- technology risks, it may be that many tional warfare. However, economic of the future leading global cata- and technological development have strophic risks are not yet within sight. brought a range of new anthropogen- Moreover, to reduce these new an- ic risks. thropogenic risks we may need lev- The first of these new risks was nu- els of international coordination that clear weapons, which gave states un- existing institutions are not designed precedented destructive power and to produce, something we discuss in emerged very rapidly: the bombings Chapter 4. A good illustration of this of and came only is the threat of catastrophic climate six years after Einstein’s letter to Roo- change. Atmospheric concentrations sevelt warning of the dangers of nu- of greenhouses gases are now at their clear fission.15 Other anthropogenic highest level for hundreds of thou- risks might also mature quickly giv- sands of years,18 and if the interna- ing us little time to prepare. Advances tional community fails to take strong in certain kinds of biotechnology, for action soon there is a worryingly high example, might at some point in the chance of warming in excess of 6°C next few decades give states, or even (compared to pre-industrial levels) terrorist groups, the capacity to cre- by the end of the century.19 But be- ate devastating designer pathogens.16 cause unilateral action is costly for Likewise, experts warn of the longer- any state, and the benefits are felt by term risks associated with power- everyone regardless of their contribu- ful machine intelligence, which may tion, action on emis- prove hard to control safely.17 sions has been slow in coming. Indeed, experience over the last century suggests that many of the

26 Global Catastrophic Risks 2016 1.4. What can be done?

hen dealing with global policy-makers, industries, research catastrophic risks we can- communities, and citizens can take W not generally rely on his- preemptive steps to limit global cat- torical experience or trial and error. astrophic risks. This report outlines Given the severity of global catastro- the key features of the world’s most phes, learning from experience would significant global catastrophic risks be extremely costly or, in the event and identifies, in Chapter 5, some of human extinction, impossible. But strategies for limiting them.

1.5. How to read this report

hapter 2 gives a comprehen- Chapter 3 discusses the causes of sive overview over the main global catastrophic risks. These in- C global catastrophic risks, from clude both factors that increase the catastrophic climate change and nu- likelihood or impact of individu- clear war to risks associated with al risks, as well as factors that affect emerging technologies such as bio- multiple risks, such as poor govern- technology, artificial intelligence and ance. geo-engineering. We discuss the po- Chapter 4 discusses why we cur- tential impact and the likelihood of rently collectively underinvest in each risk, as well as the main actions global catastrophic risk. It also dis- to limit them. Finally, we attempt to cusses what actors are best placed to compare the risks in terms of how overcome this neglect. likely they are and in terms of how Finally, in Chapter 5 we briefly dis- much attention we ought to pay them cuss a number of concrete steps to re- at present. duce global catastrophic risk.20

Global Catastrophic Risks 2016 27 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

Chapter 2 What are the Most Important Global Catastrophic Risks?

or most of human histo- sion. The burning of fossil fuels has ry, humanity has had to brought huge improvements in hu- contend with a relatively man welfare, but unless strong action narrow range of global cat- is taken soon, there is an unaccept- astrophic risks. Super-vol- able chance that our children and Fcanic eruptions and large asteroid im- grandchildren will face catastrophic pacts were possible but very unlikely, global warming. Rapid developments so only natural pandemics and ex- in biotechnology could enable scien- treme seriously tists to develop new therapies to re- threatened the complete destruction duce the global burden of disease and or permanent stagnation of human feed a growing population, but might civilisation. also in the future give malicious Today, thanks to economic and groups the capacity to synthesise dev- technological , global living astating pathogens. standards have never been higher, This chapter surveys currently the but unfortunately, for the same rea- most important global catastrophic son, we face a number of new anthro- risks by examining expert scientific pogenic global catastrophic risks. opinion on the two determinants of Some of these appear to be at least as risk: potential impact and likelihood. threatening to human civilisation as There is also a brief discussion of ac- natural pandemics and convention- tions available to limit each of these al warfare. Splitting the atom brought risks, which is summarised in Chap- the promise of clean power, but also ter 5. In the final section, we give a led to the nuclear , which has comparative assessment of the differ- brought humanity to the brink of ca- ent catastrophic risks. tastrophe on more than one occa-

28 Global Catastrophic Risks 2016 Splitting the atom brought the promise of clean power, but also led to the nuclear bomb, which has brought humanity to the brink of catastrophe on more than one occasion.

Global Catastrophic Risks 2016 29 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

2.1. Catastrophic climate change s a result of human activi- ty or strong loops in ty since the Industrial Rev- the . For example, global A olution, atmospheric con- warming might cause the melting of centrations of Greenhouse Gases , which would re- (GHGs) are now at their highest level lease substantial amounts of meth- for hundreds of thousands of years,21 ane – a potent GHG – into the at- which has caused global surface and mosphere.22 This process could itself warming. Continued increas- trigger other positive feedback loops. es in GHG emissions are very likely Catastrophic warming could also oc- to cause future warming. The eventu- cur even without these feedback ef- al level of warming depends on total fects, if turns out GHG emissions and on the sensitiv- to be higher than estimates. ity of the climate to GHG emissions. Alternatively, it might occur simply The reports of the Intergovernmental because we are less able to coordinate Panel on Climate Change (IPCC) focus internationally to reduce emissions on the most likely levels of warming than we expect. given a particular emissions path- It is impossible to say with confi- way. It is widely agreed that the neg- dence exactly what level of warming ative consequences of the most likely would bring about global catastrophe, levels of warming will be substantial. in the sense we are interested in here. However, it is important to take ac- The IPCC states: count the ‘tail risk’ of lower probabili- “Global climate change risks are ty, but potentially catastrophic, levels high to very high with global mean of warming. Of course, what we refer temperature increase of 4°C or more to in this section as ‘non-catastroph- above preindustrial levels in all rea- ic climate change’ would still have se- sons for concern, and include severe vere consequences, but it would not and widespread impacts on unique constitute a global catastrophe, on and threatened systems, substan- our definition. tial species extinction, large risks to global and regional food , and POTENTIAL IMPACT the combination of high temperature OF THE CATASTROPHE and compromising normal Increasing GHG emissions could po- human activities, including growing tentially trigger catastrophic climate food or working outdoors in some ar- change due to high climate sensitivi- eas for parts of the year. The precise

30 Global Catastrophic Risks 2016 31 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

levels of climate change sufficient to most likely to follow a medium-high trigger tipping points (thresholds for emissions scenario, and there is some abrupt and irreversible change) re- chance we will follow a high emis- main uncertain, but the risk associ- sions scenario.27 However, the situ- ated with crossing multiple tipping ation may have changed as a result points in the system or in inter- of the December 2015 Paris Agree- linked human and natural systems ment.28 increases with rising temperature.”23 In estimating the impact of differ- The latest IPCC report focuses on ent emissions scenarios, the IPCC the impacts of warming of 1°C to 4°C does not set out the probability of all above pre-industrial levels, and sug- possible levels of warming, but in- gests that the impacts corresponding stead only sets out the likely range of to more extreme levels of warming warming, where ‘likely’ is defined as are relatively unstudied.24 However, it having a greater than 66% chance of is likely that damages increase signif- occurring.29 However, in order to pri- icantly at higher temperatures,25 and oritise resources effectively govern- possible that warming of 6°C or more ments need to take into account the above pre-industrial levels may be whole probability distribution, in- catastrophic. Warming of this magni- cluding lower probability but extreme tude is, for example, likely to render levels of warming. Indeed, these may most of the substantially less constitute the majority of the expect- habitable than at present.26 ed (probability-weighted) costs of cli- mate change. LIKELIHOOD OF Some scholars have provided esti- THE CATASTROPHE mates of the probability of extreme The probability of catastrophic cli- warming. The economists Ger- mate change depends on the lev- not Wagner and Martin Weitz- el of GHGs in the man have inferred estimates and on the sensitivity of the of the probability of warming climate to cumulative GHGs. of more than 6°C relative to According to a report by pre-industrial levels from King et al for the Cen- the IPCC figures. They tre for Science Policy argue that even on a at the University of low-medium emis- Cambridge, if ma- sions scenario, there jor countries and $40 is at least a 3% regions contin- per chance of even- ue with cur- Is the recommended global price tual 6°C warm- rent plans and of carbon by some leading economists. ing (with projects, the Current price is significantly less. significant world is uncertain-

32 Global Catastrophic Risks 2016 Governments need to take into account lower probability but extreme levels of warming. Indeed, these may constitute the majority of the probability-weighted costs of climate change.

Global Catastrophic Risks 2016 33 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

ty). On the medium-high emissions approximately linear relationship scenario, the chance could be around between the total amount of carbon 10%.30 These figures are of course emitted and the resulting tempera- speculative, but they do provide some ture increase.34 The majority of the reason to believe that the probabili- carbon still underground is in the ty of catastrophic climate change is form of .35 If we were to avoid non-negligible, unless strong action is burning the remaining global coal taken on GHG emissions. reserves we would likely avoid cata- strophic levels of climate change.36 MAIN ACTIONS AVAILABLE States could commit to building no TO LIMIT THE RISK new coal-fired power stations without There are three main ways to reduce carbon capture and sequestration to the risks from climate change: adap- limit the fraction of the global coal re- tation to climate change, abatement serves which are burned. of GHG emissions, and geo-engineer- Geo-engineering – the deliberate ing. It is likely to be very costly if not use of technology to alter the world’s impossible to to avoid many of the climate – in the form of Carbon Di- impacts of non-catastrophic climate oxide Removal (CDR) or Solar Radia- change by alone,31 and tion Management (SRM), could also adapting to warming of 6°C or more is help to reduce the risk of catastroph- likely to be even more costly and dif- ic climate change, as a complement ficult. to GHG abatement. CDR techniques, Turning to abatement, most econ- such as or iron omists agree that the best way to re- fertilisation of the , would re- duce GHG emissions is to impose a move CO2 from the atmosphere and or a cap and trade sys- thereby help us move towards net tem.32 At present, the global price of neutral or net negative emissions.37 carbon is approximately $4 per tonne, SRM techniques, such as the injection whereas according to Wagner and of sulphates into the , Weitzman, to fully price in the exter- cause by reflecting nalities from catastrophic climate . The benefits and risks of change, a price of at least $40 could geo-engineering are discussed in be required.33 more detail in section 2.5.38 Scientists have demonstrated an

34 Global Catastrophic Risks 2016 FIGURE 2.1. THE CHANCE OF EXTREME CLIMATE CHANGE The probability of warming of 6°C for different atmospheric concentrations of greenhouse gases.39

Probability of warming >6°C Percent

20

15

10

5

0 400 450 500 550 600 650 700 750 800 CO2e concentration Parts per million CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

2.2. Nuclear war

he invention of nuclear weap- has 4,300, with yields ranging ons gave humanity the tech- from 50 to 800 kilotons.42 For compar- T nical capacity to cause dev- ison, the bomb dropped on astation on a hitherto unseen scale. Hiroshima had a 15 kiloton yield.44 Although there have been no nucle- The damage from nuclear war can ar attacks since the Second World be divided into two main categories. War, we have come close to inadvert- Firstly, there is the damage from the ent and intentional nuclear war on a blast, fire and radiation. A 1979 re- number of occasions. A worrying pos- port by the U.S. Office of Technolo- sible consequence of nuclear war is gy Assessment estimated that, in an a nuclear with global climat- all-out war between the US and Rus- ic implications. While the chance of sia involving thousands of nuclear nuclear war may appear to have de- weapons, 35-77% of the U.S. popula- clined since the end of the Cold War, tion and 20-40% of the Soviet popula- tensions between nuclear states per- tion would die within the first 30 days sist. Reducing the likelihood of nu- of the attack, and millions would die clear war is a serious ongoing global globally in the following years due to challenge. the radioactive .45 The pro- Despite these risks, some have ar- portionate death toll today is likely to gued that the deterrence effects of be lower because nuclear arsenals at the nuclear bomb have been, and will that time were five times larger.46 be, very valuable in terms of ensur- The second category of damage is ing global .40 How to make the a possible , which is trade-off between lower likelihood caused by the burning of , in- of nuclear or conventional conflict is dustrial facilities and other flamma- unclear. ble materials, sending into the atmosphere. Scientists have applied POTENTIAL IMPACT modern climate models to predict the OF THE CATASTROPHE scale of nuclear winter, though these The scale of the damage done by nu- predictions are uncertain. According clear war obviously depends on the to one model, an all-out exchange of scale of the nuclear war itself. In 2014, 4,000 nuclear weapons would release there were 9,920 stockpiled nuclear 150 teragrams of smoke, leading to a warheads in the world (down from a 8°C fall in global temperature.47 Due peak of 65,000 in 1986).41 The United to the fall in temperature and the loss States has 4,760, with yields ranging of sunlight and growing food would from 5 kilotons to 455 kilotons;42 and be virtually impossible for 4-5 years,

36 Global Catastrophic Risks 2016 37 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

creating an unprecedented . between one out of three and even”.5! According to some models, even a Although tensions have decreased smaller scale regional war between since the end of the Cold War, in and involving fifty 15 the wake of the crisis in Ukraine the kiloton weapons would cause global chance of confrontation has risen,52 temperatures to fall by around 1.25°C and the geopolitical situation could in the first year. Some have suggested become more unstable over the next that this could disrupt so few decades. significantly that one billion people According to many experts, the would be at risk of , though most likely intentional nuclear war this has been criticised as overly pes- is between India and Pakistan. In- simistic.48 dia and Pakistan have had numerous wars in the past and there have been LIKELIHOOD OF THE CATASTROPHE various terrorist attacks against India It is very difficult to estimate the by Pakistani groups.53 Pakistan has probability of a nuclear war in the pledged to meet any Indian attack on next century. However, the chance its territory with a retaliatory nuclear may be too high to ignore. Over the strike.54 course of the nuclear age, we have come close to inadvertent nuclear war MAIN ACTIONS AVAILABLE on numerous occasions. For exam- TO LIMIT THE RISK ple, in 1995 Russian systems mistook There is some disagreement about a Norwegian rocket for a po- how to reduce the risk of nuclear war. tential nuclear attack. Russian Pres- One policy option favoured by many ident Boris Yeltsin retrieved launch is continued reduction of global nu- codes and had the nuclear suitcase clear arsenals.55 There has been a open in front of him. Thankfully, Rus- great deal of progress on this front: sian leaders decided the incident was in the 1960s 23 countries had weap- a false alarm.49 In a 2013 paper Barrett ons or were pursuing programmes, et al estimated that the 90% confi- whereas today only nine countries dence interval of the annual probabil- have weapons.56 However, reduction ity of accidental nuclear war between of nuclear arsenals does not guaran- the US and Russia is from 0.001% to tee safety.57 7%.50 This covers a significant range, Weapons systems could also be al- but a substantial portion of the possi- tered to reduce the risk of acciden- bilities are uncomfortably high. tal and intentional war. For instance, We have also come close to inten- decision times could be increased tional nuclear war on many occa- by each side locating their weapons sions. President Kennedy said that further from each other’s borders.58 the chance of nuclear war during the More broadly, improved internation- was “somewhere al relations would help to decrease

38 Global Catastrophic Risks 2016 FIGURE 2.2.1. NUCLEAR ARSENALS OF THE US, RUSSIA, AND THE WORLD FROM 1945 UNTIL TODAY 59

Number of nuclear warheads Thousands

80

70 Russia

US 60

Global

50

40

30

20

10

0 1945 1950 1960 1970 1980 1990 2000 2010 2014 Year CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

NUCLEAR NEAR MISSES • In 1973, when encircled the Egyptian Third Army, the Soviets threatened to intervene, leading to implied nuclear threats. • In September 1983, a Soviet early warning showed that the United States had launched five land-based missiles at the Soviet Union. The Soviet officer on duty, Stanislav Petrov, had only minutes to decide whether or not the satellite data were a false alarm. Since the satellite was found to be operating properly, following procedures would have led him to report an incoming attack. Going partly on gut instinct and believing the United States was unlikely to fire only five missiles, he told his commanders that it was a false alarm before he knew that to be true. Later investigations revealed that reflection of the on the tops of had fooled the satellite into thinking it was detecting missile launches. • The Able Archer incident of November 1983 was, in the words of former US Secretary of Defense Robert Gates, “one of the potential- ly most dangerous episodes of the Cold War”. With talk of fighting and winning a nuclear war emanating from Washington, the Soviets reasoned that the West would mask preparations for a nuclear attack as a exercise. The Able Archer exercise simulated the coordi- nated release of all NATO nuclear weapons. In response, the Soviets readied their nuclear forces and placed air units in East and Poland on alert.

There have been numerous other examples of nuclear near misses.

the likelihood of international nu- could work to ensure that terrorists clear conflict. Research into nucle- cannot acquire nuclear weapons by ar winter would improve our knowl- theft or on illegal markets. Glob- edge and allow nuclear states to al enforceable standards for nuclear change their strategy accordingly. weapon security would help achieve Finally, state and non-state actors this goal.60

40 Global Catastrophic Risks 2016 FIGURE 2.2.2. GLOBAL TEMPERATURE ANOMALY FROM NUCLEAR WINTER Global average surface air temperature change from a release of 5 Teragram, 50 Teragram, and 150 Teragram of particular matter in the context of the climate change of the past 125 years.61

Temperature anomaly °C +5 Tg +50 Tg +150 Tg

1

0

-1

-2

-3

-4

-5

-6

-7

-8 1880 1890 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 2010 Year CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

2.3. Natural pandemics

ntil the discovery of nucle- H5N1 would have to be easily transmis- ar weapons, pandemics were sible between , which is not U the most important global currently the case, though occasionally catastrophic risks, and it is plausible humans become infected through ani- they still are. Plague, HIV, mal vectors. Influenza pandemics have and other have killed mil- previously infected about 24-38% of the upon millions of people. There world population.65 The case fatality of remains a serious chance that a pan- a novel strain of H5N1 is unpredicta- demic could kill a huge portion of the ble, but estimates for the H5N1 case fa- world population, with pandemic in- tality rate until today vary particularly fluenza the most serious threat. A va- widely – from 1% to 60%.66 A very rough riety of measures, including improv- estimate of the death toll of a H5N1 ing international coordination and pandemic can be found by multiplying data sharing, and increasing the pro- the usual pandemic attack rate (24% to duction of drugs and vaccines, would 38%) by the global population (rough- help to reduce this risk. ly 7 billion) times the case-fatality ratio (1% to 60%). This produces an estimat- POTENTIAL IMPACT ed death toll of between 16.8 million OF THE CATASTROPHE and 1.7 billion fatalities. The develop- The Black Death plague in the 14th ing world would probably bear an over- century killed between 11% and 17% whelming part of this burden.67 of the global population over the course of a decade.62 In the most dev- LIKELIHOOD OF THE CATASTROPHE astating pandemic of the last 100 Influenza pandemics occur relatively years, the 1918 Spanish influenza frequently: there have been ten in the outbreak, 50-100 million people lost last 300 years, though none of these their lives – between 2.5 and 5% of have killed more than 5% of the world the global population.63 Both of these population.68 It is difficult to know pandemics occurred before the age where and when the next natural pan- of modern medicine, so it is unlike- demic will occur. Globalisation and ly these diseases would have simi- increased between hu- lar impact today, although our more mans and increase the risk of connected society may increase the pandemics, but improved health and spread of pandemics. would lessen their impact. Currently, H5N1 is According to the UK National Risk thought to be the greatest pandemic Register there is between a 1 in 20 and threat.64 To develop into a pandemic a 1 in 2 chance of a pandemic killing up

42 Global Catastrophic Risks 2016 43 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

to 1% of the UK population in the next bodies such as the WHO can stockpile five years. Such a pandemic would go drugs and vaccines to counter disease on to have global effects. Global cata- outbreaks. However, stockpiles may strophic risk-level pandemics (killing only be effective for some but not all more than 10% of the global popula- pandemic pathogens.72 Thirdly, since tion) are, however, further down the developing countries will probably tail of the probability distribution. Re- face the highest burden from future sults from expert surveys have put a pandemics, continued improvements 15% chance of an H5N1 pandemic over in developing world health systems a three year period.69 in accordance with the WHO’s Inter- national Health , and im- MAIN ACTIONS AVAILABLE provements in the global distribution TO LIMIT THE RISK of drugs and vaccines, would limit the There are numerous ways to limit the risk from pandemics.73 Fourthly, dis- risk from pandemics.70 Firstly, pharma- ease surveillance and response sys- ceutical companies could focus atten- tems could be improved so that nov- tion on developing vaccines and drugs el threats in both and human for very damaging pandemics, and the are detected and respond- international community can remove ed to quickly.74 Rapid dissemination barriers to development with measures of relevant data between countries is such as the establishment of a glob- also essential for effective outbreak al vaccine development fund.71 Sec- response, because of the exponential ondly, governments and international spread of in an outbreak.75

44 Global Catastrophic Risks 2016 FIGURE 2.3.1. HISTORIC PLAGUES AND PANDEMICS 76

PANDEMIC DATE(S) LETHAL IMPACT

Plague of Justinian AD 541-542 25-33m (13-17% of the world population)

Black Death 14th Century 50-75m (11-17% of world population)

Smallpox 1520-1527 200,000 deaths within the Aztec population (75% of population in some areas)

Spanish influenza 1918 - 1919 50 - 100m (2.5-5% of population)

Smallpox 20th century 300m over the course of the 20th Century

HIV/AIDS 1981 - present 34m

Global Catastrophic Risks 2016 45 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

2.4. Exogenous risks

xogenous risks are those which est mass extinction in history and re- arise independent of human duced the human ancestor popula- E activity (whereas even natu- tion from around 100,000 to around ral are spread by humans). 4,000 people for approximately They include such possibilities as 20,000 years, though this is contro- super-volcanic eruptions and large versial.81 According to a report by the asteroid and comet impacts, which Geological Society of London: are believed by some to have caused “A layer of ash estimated at 15 cen- mass .77 The likelihood of timetres thick fell over the entire exogenous risks is better understood Indian sub-, with similar than that of many other global cata- amounts over much of SE . Just strophic risks because the underlying one centimetre of ash is enough to dynamics have been unchanged for devastate agricultural activity, at least a very long time. The historical evi- when it falls in the growing . dence suggests that exogenous global An eruption of this size would have catastrophic risks cannot be too fre- catastrophic consequences. Many quent, and may therefore be much millions of lives throughout most less likely than some of the anthropo- of Asia would be threatened if Toba genic risks.78 erupted today.”82

A. SUPER-VOLCANOES LIKELIHOOD OF THE CATASTROPHE Super-volcanoes are volcanoes ca- In order to assess the likelihood of pable of producing at least 1012 m3 super- eruptions, we have bulk volume of fragmental material.79 to rely on a relatively limited set of Some experts believe that the erup- observations of past super-volcan- tion of the Toba super-volcano in ic eruptions, which makes any esti- around 70,000 years ago mates very uncertain. Existing data brought humanity to the brink of ex- suggest that there will be a super-vol- tinction, though there is significant canic eruption roughly every 30,000- disagreement about this. 50,000 years on average.83

POTENTIAL IMPACT OF THE RISK MAIN ACTIONS AVAILABLE The Toba eruption ejected large TO LIMIT THE RISK amounts of dust and sulphates into At present, humanity lacks the tech- the Earth’s atmosphere, which caused nical capacity to prevent volcanic global cooling of 3-5°C for sever- eruptions. Consequently, improving al years and led to enormous loss of resilience to catastrophe is the main and animal life.80 Some have way to limit the risk from super-vol- argued that Toba caused the great- canoes. We discuss this in more detail

46 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 47 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

in Chapter 3. Increased investment being released into the atmosphere.86 and research might also improve our The damage that would be caused by ability to predict volcanic eruptions, a multi-kilometre asteroid or comet which would in turn improve prepar- impact is only modestly understood edness.84 at present.87 An asteroid winter would undermine agriculture at least for an B. ASTEROIDS entire growing season and so could AND COMETS cause the deaths of billions of peo- Around 66 million years ago, an as- ple.88 According to a comprehensive teroid of around 10km in diameter report by the US National Research struck Chicxulub in Mexico. This im- Council, “above the conservatively pact probably caused one of the three assumed global catastrophe thresh- largest mass extinctions in history old from a 1.5km-diameter impac- and may have abruptly ended the age tor, the number of fatalities ramps up of the .85 Today, an impact from 10% of the world’s population to by a Near Earth Object (NEO) – an as- the entire population for impactors teroid or comet – larger than 1.5km in above 10km in diameter”.89 diameter would kill millions largely by causing global cooling and agricul- LIKELIHOOD OF THE CATASTROPHE tural disruption. The likelihood of an NEOs are a comparatively well-un- NEO impact is reasonably well-un- derstood global catastrophic risk. NA- derstood and important steps have SA’s system has mapped been taken to monitor the risk they more than 90% of asteroids with a di- pose. ameter of more than 1km.90 Reinhardt et al argue that the total probabili- POTENTIAL IMPACT ty in a 100-year period of an aster- OF THE RISK oid encounter with the Earth that Contemporary studies of aster- might cause a globally signifi- oids and comets conservative- cant effect is approximately ly assume that all objects 1 in 1,250.91 greater than 1.5km in di- One can calculate the ex- ameter would be capa- 1 pected cost of NEOs by ble of causing damage in multiplying the like- on a global scale, via ly deaths per event generated by the frequency of by impact debris 1,250 events of a cer- and a so-called tain size. As of approximate risk of an asteroid asteroid win- encounter with the Earth that might 2010, the annu- ter caused by cause a globally significant effect al expected dust and in a 100-year period (probabili- sulphates ty-weight-

48 Global Catastrophic Risks 2016 FIGURE 2.4. RISK BEFORE AND AFTER THE SPACEGUARD SURVEY 92

Statistical fatalities per year

250

Intrinsic risk before survey

200 Residual risk after 10 years of surveying

150

100

50

0 0,01 0,1 1 10 Diameter of impactor Kilometres CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

“The likelihood of an NEO impact is reasonably well- understood and important steps have been taken to monitor the risk they pose.”

ed) cost of all NEOs is currently 91 fa- detection of asteroids would enable talities.93 humanity to understand the risks it Once ongoing asteroid surveys faces and to take appropriate coun- are completed, long-period comets ter-measures. NASA’s ongoing Space- – comets which take more than 200 guard Survey has so far reduced the years to orbit the Sun – may domi- expected cost a large asteroid impact nate the remaining unknown impact by more than an order of magnitude.95 threat from NEOs.94 Secondly, technological research could help us to find ways to deflect threat- MAIN ACTIONS AVAILABLE ening NEOs.96 Thirdly, improvements TO LIMIT THE RISK in resilience could be made to ensure There are three main ways to limit the human survival in the event of a large risk from asteroids. Firstly, improved asteroid or comet impact.

50 Global Catastrophic Risks 2016 51 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

2.5. Emerging risks

n the coming decades, emerging come increasingly streamlined, the technologies will provide major tools enabling such risks may become I benefits to society, but they may widely accessible.99 Previously benign also create significant and unprece- digital information, such as the wide- dented risks. Certain types of bio- ly available online genetic data for technology, if more widely accessi- smallpox, will become more - ble, could give terrorist groups the ous. access to pathogens as dangerous as smallpox. Geoengineering technol- POTENTIAL IMPACT OF THE RISK ogies could give single countries the generally con- power to unilaterally alter the earth’s strains the lethality of pathogens, as climate. Finally, artificial general in- most highly lethal pathogens fail to telligence could, if developed, leave spread far before killing their host. human control. There is thus some evidence for an Technological risks could emerge inverse relationship between a path- very quickly and give certain groups ogen’s lethality and transmissibility, large and perhaps unprecedented thereby limiting the damage from a destructive power. Moreover, some naturally occurring pandemic. Bio- emerging technologies may be par- technology has the potential to break ticularly difficult to control because this correlation, allowing barriers to access the technology may with extraordinarily high lethality be quite low. and transmissibility. In 2001 Austral- ian researchers accidentally created A. ENGINEERED a highly lethal and vaccine resistant PANDEMIC form of mousepox.100 Similar tech- The past decades have seen rapid ad- niques could potentially be applied vances in biotechnology, in part due to smallpox.101 Two recent controver- to the falling costs of gene sequenc- sial papers have shown how to create ing and synthesis.97 Improvements in a version of H5N1 which is potential- ease-of-use of certain specific kinds ly transmissible between humans.102 of biotechnology bring increased con- Basic calculations based on H5N1 pa- about . Gene syn- rameters suggest that a single release thesisers have the capacity to turn of these modified could cause digital sequence data into physical hundreds of millions of casualties.103 genetic sequences, enabling individ- Engineered pathogens with danger- uals to create viruses from digital files ous features could be released acci- (as was done with the 1918 Spanish dentally from a lab or intentionally Flu virus).98 Should gene synthesis be- by states or terrorist groups. The legal

52 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 53 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

theorist has described As biotechnology develops, the a scenario in which terrorists synthe- level of expertise required to create sise a version of smallpox that is in- dangerous pathogens will fall. In the curable, immune to vaccine and kills longer term, if biotechnology ma- all its victims, which is then released tures sufficiently and gene synthesis in a large via .104 Although is not well-regulated, states and small such scenarios are not currently feasi- groups will find it increasingly easy to ble, eminent figures including George synthesise and alter dangerous path- Church,105 ,106 and ogens.112 This poses a serious risk of a Martin Rees107 have all argued that bi- global catastrophe. otechnology poses serious risks. MAIN ACTIONS AVAILABLE LIKELIHOOD OF THE CATASTROPHE TO LIMIT THE RISK There is a real possibility that a dan- Many of the measures which limit the gerous engineered could risk from natural pandemics would be released by accident. The H1N1 also limit the risk from synthesised or influenza strain, responsible for sig- modified pathogens. However, other nificant morbidity and mortality measures would help specifically with around the world from 1977 to 2009, is biotechnology risks. Fostering a cul- thought to have originated from a lab- ture of safety in the relevant research oratory accident.108 As of 2012, there and technical communities is proba- were at the very least 42 laboratories bly very valuable. Researchers could engaged in live research on potential be encouraged to take significant pre- pandemic pathogens, though this will cautions with this research and to have been affected by the US mora- avoid disseminating research when torium on ‘gain of function’ research doing so brings major risks. With re- in 2014.109 Lipsitch and Inglesby esti- gard to regulation, a licensing regime mate that “work with a novel, trans- for DNA synthesis could be one first missible form of influenza virus car- step,113 while mandatory liability in- ries a risk of between 0.01% and 0.1% surance for dual-use research - which per laboratory-year of creating a pan- helps medical progress but could be demic... or between 0.05% and 0.6% used maliciously - would ensure that per full-time worker-year”.110 These researchers have incentives to main- estimates are illustrative of the prob- tain high levels of laboratory biosafe- ability that existing ‘gain of function’ ty.114 Governments could also require influenza research would produce a journal editors to consider whether pandemic, though there is significant publication of research could lead to uncertainty about them.111 adverse outcomes.115

54 Global Catastrophic Risks 2016 FIGURE 2.5.1. COST PER GENOME 116

Cost of sequencing a human genome Dollars (logarithmic scale)

100M

10M Moore’s equivalent

1M

100K

10K

1K 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Year CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

B. ARTIFICIAL influence over the future. It has been INTELLIGENCE argued that if these goals were not Prominent figures uch as Stuart Rus- aligned with human values, the conse- sell, Professor of Computer Science at quences could be truly catastrophic.121 Berkeley, Peter Norvig, Director of Re- It is difficult not only to specify human search at Google, and , values in a robust, machine-interpret- author of , recent- able way, but also to agree on human ly signed a letter warning of the risks values in the first place. Moreover, if posed by Artificially Intelligent (AI) even if the values of very powerful ar- systems.117 Past experience demon- tificially intelligent systems can be strates that AI systems can go from aligned with their creators’, such sys- significantly subhuman to superhu- tems might destabilise the geopolitical man relatively quickly in narrow do- balance in a destructive way. mains, such as recently in the game Go. In the coming decades, we may LIKELIHOOD OF THE CATASTROPHE create AI systems which surpass hu- It is widely accepted that we will be mans in all relevant domains. If this able to create AI systems that are ca- were to happen, the effects would be pable of performing most tasks as uniquely transformative. However, well as a human at some point (‘hu- AI also has huge potential benefits, man-level AI’). Experts disagree as well as risks.118 It could, for exam- about when this will occur. According ple, greatly reduce the cost of many to the median surveyed expert, there goods, and allow us to solve other is a roughly 50% chance of such AI global problems. by 2050.122 The median surveyed ex- pert believes that there is at least a 5% POTENTIAL IMPACT OF THE RISK chance of superintelligent AI within Some risks of AI likely fall short of two years after human-level AI, and posing a global catastrophic risk. a 50% chance within thirty years.123 Widespread automation could cause Assuming that human-level AI is de- significant economic and social dis- veloped, its long-term social impact ruption, which has only a relative- is unclear. According to the medi- ly small chance of leading deaths at an surveyed expert, there is around a the scale of other global catastrophic 7% chance that it would be ‘extreme- risks.119 ly bad’.124 The extreme uncertainty In the longer term, AI may enable surrounding these estimates should, important new capabilities, perhaps however, be strongly emphasised. extremely quickly if it turns out that we can automate AI development.120 If we MAIN ACTIONS AVAILABLE have powerful and generalisable auto- TO LIMIT THE RISK mated systems, the goals they are pro- Different challenges from AI systems grammed with may exert significant will require different responses, and

56

CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

in many cases it will be more appro- iron or urea could pose major risks to priate to respond closer to the time marine .125 However, most the technologies mature. However, forms of SRM are thought to carry there is a strong case for early work much greater risks than most forms preparing for possibly rapid changes of CDR,126 and worries about civilisa- brought about by AI systems because tion-threatening consequences have a reactive response won’t succeed generally focused on SRM (and in when the change is very rapid. In par- particular on currently the leading ticular, important research could be form of SRM: the injection of sulphate done on how to give AI systems de- particles into the stratosphere).127 The sirable goals. Foresight work could be remainder of this section will there- done to better understand which oth- fore focus on SRM only. er issues require advance preparation, SRM is the only known technique or where there may be an eventual for quickly stopping (or even revers- role for other responses. Work could ing) the rise in global temperatures. also be done to encourage under- This means that it could be used as standing of the risks among AI devel- a complement to GHG reduction, to opers, especially around automated manage temperatures while the world AI system development, which might phases out fossil fuels. Some have enable very fast transitions. However, proposed that SRM could provide in- the benefits of AI could be very great surance against a ‘climate emergen- so it is important not to unnecessarily cy’, such as unexpected abrupt and impede AI development. extreme warming.128 Thus, on some scenarios, failing to use SRM could C. GEOENGINEERING constitute a global catastrophic risk. As mentioned previously in this chap- Moreover, SRM has the potential to ter, geo-engineering – in the form of reduce the costs of warming at very Removal (CDR) or low cost. Some estimate that the an- Solar Radiation Management (SRM) nual cost of stratospheric – could help to limit the risks of cata- could be less than $10 billion per year, strophic climate change. CDR tech- which is orders of magnitude less niques, such as ocean fertilisation or than the costs of climate change miti- carbon sequestration, remove Car- gation strategies.129 bon Dioxide from the atmosphere, whereas SRM techniques, such as POTENTIAL IMPACT OF THE RISK cloud brightening or the injection of Four main arguments have been giv- sulphates or other particles into the en for the view that SRM brings global stratosphere, reflect the sun’s light catastrophic risks. Firstly, while ex- and heat back into space. Certain isting models suggest that SRM could forms of CDR could carry major risks. reduce the catastrophic effects of cli- For example, ocean fertilisation using mate change and will not bring their

58 Global Catastrophic Risks 2016 2.5.2. GLOBAL TEMPERATURE ANOMALY FROM GEO-ENGINEERING FOLLOWED BY TERMINATION Global average surface air temperature change from business as usual emissions, injection of 3 mega- tons/annum (Mt/a) of SO2 in the Arctic, 5 Mt/a of SO2 in the tropics, and 10 Mt/a SO2 in the tropics.130

+Anthro Forcing +3 Mt/a Arctic Temperature anomaly °C +5 Mt/a Tropical +10 Mt/a Tropical 1.4 Geoengineering Geoengineering 1.3 starts ends 1.2

1.1

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

-0.0

-0.1

-0.2

-0.3

-0.4

-0.5 1880 1890 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 2010 2020 2030 2040 Year CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

own catastrophic impacts, it may nev- it could discourage GHG reduction ertheless bring currently unknown efforts.136 All major reports studying risks, particularly through impacts SRM have concluded that it does not on global .131 The climate present an alternative to emissions re- system is imperfectly understood and ductions, as it only masks the effects the deployment of a novel technol- of GHGs and does nothing to coun- ogy with global effects is inherent- ter .137 Therefore, it ly risky. Whether the use of SRM is could be problematic if geo-engineer- more risky than allowing the planet to ing drew a disproportionate amount warm is one of the key questions for of policy attention. However, it is im- anyone considering using SRM. Sec- portant to bear in mind that there are ondly, because SRM is so cheap, in- countervailing reasons in favour of dividual states could feasibly deploy SRM research and advocacy. it and unilaterally bring about global climatic impacts. Moreover, individ- LIKELIHOOD OF THE CATASTROPHE ual states might also have the incen- The probability that there will be a tive to do this because they could be global catastrophe brought about by particularly badly affected by climate SRM depends on many factors, in- change.132 Individual states acting cluding the timing, speed and severi- alone may be less likely to properly ty of global warming; the state of SRM take into account the interests of oth- technology and our knowledge of the er states and may be concerned about climate system; the response of the catastrophic consequences in other climate system to SRM; the form of regions. SRM deployed and how it is used; and Thirdly, sudden termination of SRM how well the world does at organising would lead to rapid and severe of SRM. There is obvious- warming.133 There are some reasons ly very large uncertainty about all of to think that an SRM system could be these factors. very resilient against external shocks We argued above that unless strong and termination. If the current de- action is taken soon, there is a sizea- ployer were to suddenly stop SRM ble chance of catastrophic warming. for some reason, every other country Catastrophic warming would create would have strong incentives to re- very strong incentives to use SRM. sume SRM.134 Thus, sudden termina- Therefore, unless strong GHG reduc- tion might only be likely in the event tion action is taken soon, the chance of a severe global catastrophe which that SRM is used will increase. The undermines the capacity of all coun- greatest chance of catastrophe prob- tries to use SRM.135 ably comes from poorly planned and Finally, research into geo-engineer- governed the use of SRM, perhaps ing or advocacy for geo-engineering by an individual state or small group could present a ‘moral hazard’ in that of states. Since most of the risks of a

60 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 61 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

well-governed and well-planned form tant for policymakers to be aware of of SRM are unknown, it is very dif- these issues. ficult to say how likely that form of SRM is to produce a global catastro- THE CHALLENGE OF phe. EMERGING RISKS Finally, it is unclear the impact re- The catastrophic risks from emerging search into, advocacy for, or prepara- technology are particularly challeng- tion for, geo-engineering would have ing. Firstly, we have no track record on global willingness to cut green- of dealing with these emerging tech- house gas emissions. Therefore, it is nological risks and, as we discuss in not clear to what extent geo-engineer- chapter 4, existing national and inter- ing is a moral hazard. national institutions are not designed to deal with them. It is therefore less MAIN ACTIONS AVAILABLE likely that our eventual responses will TO LIMIT THE RISK be effective. The main way to reduce the risks Secondly, these technologies might, from SRM would be to take strong like nuclear weapons, reach maturi- action on GHGs to reduce the ex- ty more quickly than expected. If so, pected costs of climate change and the nature of the problem would only thereby reduce the incentives to become fully apparent over a short geo-engineer. Further research into time-frame and it would be difficult the different kinds of SRM and into to make an appropriate response. the response of the climate to them Thirdly, some of these technologies would reduce the unknown risks of could be harder to control than nucle- SRM. Finally, working to develop ar weapons. Nuclear weapons require geo-engineering governance through the rare and controllable resources climate treaties and through global of uranium-235 or plutonium-239. In institutions might limit the risks of contrast, if some of these technolo- unilateral SRM.138 However, both of gies reach full maturity, they could be these actions might also have moral accessible to small countries or even . We cannot settle the moral terrorist groups. hazard debate here, but it is impor-

62 Global Catastrophic Risks 2016 In the coming decades, emerging technologies will provide major benefits to society, but they may also create significant and unprecedented risks.

Global Catastrophic Risks 2016 63 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

2.6. Other risks and unknown risks

here is a broad spectrum of unforeseeable a few decades before possible global catastroph- they became apparent. Forty years T ic risks, from very salient and before the discovery of the nuclear well-understood threats to those bomb, few could have predicted that which are extremely low probabili- nuclear weapons would come to be ty and highly speculative. In compil- one of the leading global catastroph- ing this report, we had to make deci- ic risks. Immediately after the Second sions about which risks to investigate World War, few could have known carefully, and which to ignore. Our that catastrophic climate change, bio- guiding principle was to include risks technology, and artificial intelligence which could clearly cause a global would come to pose such a significant catastrophe in our definition if they threat. were to happen, which could be spec- These risks emerged due to rapid ulative but not too speculative, and economic and technological devel- which were not too low in probability. opment, which looks set to contin- There are several risks we didn’t ue apace in the coming century. That include for these reasons. For exam- might create a number of new risks. ple, gamma ray bursts are fairly well Therefore, it seems likely that some understood and would be catastroph- future global catastrophic risks are at ic, but have a very low annual likeli- present unknown. hood.139 A global totalitarian state is Detailed planning for unknown a possible scenario, but it is unclear risks is of course impossible, but steps what proportion of the scenarios can be taken to improve our prepar- would meet the threshold for a global edness. Bodies specifically tasked catastrophe and how likely it would with and the dis- be to arise given the current geopoliti- covery of new global catastroph- cal environment.140 Conventional and ic risks would give the international are real threats, but community more time to craft an ap- quite unlikely to reach the scale of a propriate response. Measures can also global catastrophe. be taken to improve general societal It is important to remember, howev- resilience to catastrophe. These are er, that nearly all of the most threat- discussed in chapter 5. ening global catastrophic risks were

64 Global Catastrophic Risks 2016

CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

2.7. Our assessment of the risks

e’ve given qualitative de- two dimensions are linked, but they scriptions of each of these can come apart, as in the case of cli- W risks. It is useful to con- mate change whose main harms are sider comparisons between them as not likely to be felt soon, but which well. Unfortunately there is a great demands action today. We think that deal of uncertainty around the level this split should be explicit. of many of the risks, and around how To assess current risk, we consid- much can usefully be done. Nonethe- ered the likelihood over the next five less, some attempt must be made in years. Our upper band consists of order to prioritise the world’s scarce events which we consider to be dis- resources, and there are some areas tinct possibilities. The lower band where we can be more confident than consists of risks which appear low in others. absolute likelihood, either because In this section, therefore, we make the base rate is low enough or because some comparative assessments of the five years is too short a timescale for risks. Because of the high level of un- the risk to develop. Because of the certainty we categorise the risks into large scale of the potential catastro- broad bands rather than attempt pre- phes, even risks in the lower category cise assessment. The categorisation may still be significant over this time remains our subjective judgement, period. and it is possible that some risks It is even harder to assess how large should be in different bands. But we the response should be. We consid- believe that it is better to offer even ered how much risk was posed, both such an imperfect assessment than at present and in years to come. We nothing. also considered how good the oppor- tunities to reduce the risk appear, and A. OUR METHOD for emerging risks how much they We have assessed the risks on two dif- benefit from early responses. Our up- ferent dimensions. First, the current per band indicates risks where we likelihood: how likely is it to materi- think a significant global response is alise in the next few years? Second, likely appropriate, perhaps involv- how much work should be given to re- ing thousands of people, or billions of ducing the risk in the next few years dollars. There is a wide range with- (according to our judgement)? These in this band, and we do not think all

66 Global Catastrophic Risks 2016 Bodies specifically tasked with horizon scanning and the discovery of new global catastrophic risks would give the international community more time to craft an appropriate response.

Global Catastrophic Risks 2016 67 CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

of the risks there should receive the have decreased since the Cold War, same amount of attention. For risks nuclear war appears to remain a real in the lower band, we still think re- possibility. We have corresponding- sponding is appropriate, but perhaps ly placed it in the upper category for at a smaller scale for the time being. likelihood. It is the only catastrophe Our assessment is summarised in that is definitely within the reach and this diagram: control of humanity today, and we think it is therefore appropriate that B. HOW WE MADE THESE the response be high. ASSESSMENTS ABOUT Asteroid impacts and supervolcan- PARTICULAR RISKS ic eruptions are caused exogenous- Catastrophic climate change would ly to human actions. This means we likely have effects decades out, and have better estimates of their rates, we consider the likelihood in the next through knowledge of historical in- five years to be small unless the sci- cidents. A supervolcanic eruption entific community has significant- (which may or may not cause a glob- ly mis-modelled climate dynamics. al catastrophe) is estimated to oc- Nonetheless, because of cumulative cur very approximately every 30,000 nature of , years. An asteroid impact large and the relatively well understood dy- enough to cause a global catastrophe namics, we believe a large response is is estimated every 120,000 years. We justified. are therefore reasonably confident Natural pandemics have been re- that they belong to the lower category sponsible for past global catastro- of current risk. Because of this rela- phes, and are seen high on lists of tively low risk, we have also put them national risk assessments. We are in the low category for attention. not confident assigning a par- We are confident that this is justi- ticular probability to a pandem- fied for supervolcanic eruptions, ic, but we do think it is among where there are few clear ac- the most likely risks in the tions to reduce the risk. As- next five years. Internation- teroids are exceptional in al work in reducing the 1 that we have relatively risk from pandemics every well-understood ways seems both important to reduce the risk, and and effective, and this could mean that we think a large re- 30,000 the attention cat- sponse is appro- years egory should be priate. is the estimated occurrence of a higher. Although supervolcanic eruption (which may or Engineered the likeli- may not cause a global catastrophe). pandem- hood may ics are an

68 Global Catastrophic Risks 2016 FIGURE 2.7. OUR ASSESSMENT OF GLOBAL CATASTROPHIC RISKS

Natural pandemic

Unknown risks Nuclear war

Engineered pandemic Higher likelihood over next 5 years

Catastrophic Asteroid impact climate change

Supervolcanic Catastrophic eruption disruption from AI

Failure of geo-engineering Lower likelihood over next 5 years Low High APPROPRIATE LEVEL OF ATTENTION CHAPTER 2 – WHAT ARE THE MOST IMPORTANT GLOBAL CATASTROPHIC RISKS?

emerging risk in that future tech- the speed of technological progress. nology will make them more plausi- In the longer term the risks could be ble. But even existing experiments extremely significant, and experts have been claimed to pose real risk of have identified useful work that can sparking a pandemic, so we put this be done today, so we have put it in the in the upper band for likelihood. We upper category of attention. also think it is appropriate that it gets Finally, risks from geoengineering significant amounts of attention, par- are another emerging technological ticularly as the unmanaged risk may risk where the current risks seem low. be increasing. But because geoengineering may be Risks from artificial intelligence ap- employed to tackle climate change, pear to be low in the short-term, since it seems important to build a more we most likely are some way off the comprehensive understanding of the kind of capabilities that might cause a risks it poses in itself. For this reason global catastrophe. We have therefore we have put it in the higher category put them in the lower category of cur- for attention. rent risk, although it is hard to predict

70 Global Catastrophic Risks 2016 There is a great deal of uncertainty around many of the risks, and around how much can usefully be done. Nonetheless, some attempt to estimate must be made in order to prioritise the world’s scarce resources.

Global Catastrophic Risks 2016 71 CHAPTER 3 – RISK FACTORS AND INTERACTIONS BETWEEN RISKS

Chapter 3 Risk factors and interactions between risks n this chapter we discuss some Global catastrophic risks are parts of the factors that affect the like- of extremely complicated causal net- lihood of global catastrophic works, and the drivers of each one risks. We first discuss some of the could quite sensibly be the subject of key factors influencing individ- multiple books. Here, we provide a Iual risks, before turning to factors that brief overview where we highlight a the risks have in common, and ways in few key drivers. This serves as a back- which catastrophic events could trigger ground to the more action-oriented further catastrophes. discussions of Chapters 4 and 5.

72 Global Catastrophic Risks 2016 Global Catastrophic Risks 2016 73 CHAPTER 3 – RISK FACTORS AND INTERACTIONS BETWEEN RISKS

3.1. Drivers of individual risks

NUCLEAR WAR ty which could result from significant Increasing the number of nucle- climate change may increase the ge- ar-armed actors -- horizontal prolif- opolitical tensions that drive nuclear eration -- probably increases the like- risk.147 lihood of nuclear conflict. Increasing the size and sophistication of their PANDEMICS arsenals -- vertical proliferation -- in- The factors driving pandemic risk di- creases the amount of damage that vide into those affecting the likeli- could occur in a given conflict.141 hood of potentially pandemic patho- These factors are, in turn, influenced gens arising, and those affecting how by political factors such as leading much society will be affected by such nations’ ability to control rogue ac- pathogens. Whether a given pathogen tors, and the level of conflict between could cause a pandemic depends on nuclear states.142 its natural parameters, such as trans- The possibility of false alarms or mission vector, resistances, and le- miscalculation between nuclear pow- thality, over which we generally have ers is another driver of risk, exacer- little control.148 However, concentrat- bated further when geopolitical ten- ed populations of poultry or other sions are high.143 There is also the animals can increase the probabili- unresolved question of exactly how ty of zoonotic spillover, raising the sensitive the climate system is to dust risk.149 The transmission of pathogens and sulphates. The higher the sensi- through society is increased by global tivity, the greater the risk of a nuclear travel and dense populations, but de- winter scenario.144 creased by factors such as improved Our response to climate change hygiene.150 The effectiveness of our may also have significant effects on countermeasures constitutes anoth- the risk of nuclear war. Many author- er important factor, ranging from dis- ities on climate change argue that it ease surveillance to healthcare access will be hard to reduce GHG emissions in the developing world.151 sufficiently without being part of the energy mix.145 Unfor- SUPER-VOLCANOES, tunately, rising use of nuclear power ASTEROIDS, AND COMETS may increase the chance of nuclear The probability of super-volcanic weapon proliferation.146 Moreover, the eruptions and asteroid and comet im- mass migration and resource scarci- pacts are driven by natural process-

74 Global Catastrophic Risks 2016 “If ‘positive feedback loops’ prove to be worse than anticipated, the risk of catastrophic climate change will be higher.”

es outside of human control. Howev- CATASTROPHIC CLIMATE CHANGE er, our ability to predict, and, in the & GEO-ENGINEERING case of asteroids and comets, to pre- The risk of catastrophic climate vent these catastrophes could have a change and the risk of a geoengineer- significant bearing on their eventu- ing catastrophe are strongly influ- al impact. Because we have less con- enced by the level of GHG emissions. trol over the likelihood of an adverse These depend on factors including event with these than with most glob- the amount of energy the world con- al catastrophic risks, resilience is a sumes and the proportion of this more central part of the risk manage- which comes from fossil fuels. The ment strategy. A particularly impor- total is shaped tant factor is the level of food secu- by, among other things, population rity, including stockpiles and the growth, , and ability to switch to less sunlight-de- energy efficiency. The proportion of pendent food sources.152 Better food energy which comes from fossil fuels security means a lower chance that a depends on other factors such as the particulate winter would have cata- rate of progress in clean energy tech- strophic consequences.153 nology development and the preva-

Global Catastrophic Risks 2016 75 CHAPTER 3 – RISK FACTORS AND INTERACTIONS BETWEEN RISKS

lence of economic incentives to adopt just their risk management mecha- clean energy technologies.154 nisms.159 Another important set of factors Formal regulation of technology concern how the climate is likely has an unclear impact—in some cas- to react to increased levels of GHG es it could prevent risky activities, but emissions. If the climate turns out to in other cases it could stifle critical be generally more sensitive to GHG innovation or, if implemented before emissions than expected, or if “pos- the issues are properly understood, itive feedback loops” prove to be even increase risk.160 On the positive worse than anticipated, the risk of side, technology developed in a cul- catastrophic climate change will in- ture of responsible innovation will crease.155 likely be safer. Finally, the development of new For some of the emerging techno- geoengineering techniques might ei- logical risks we may also hope to de- ther decrease or increase global cat- velop solutions. It may reduce risk to astrophic risk. They could decrease make progress on developing safety global catastrophic risk if they prove aspects of a technology faster relative to be an effective tool to mitigate to progress on the technology itself. catastrophic climate change.156 But For example having excellent diag- they could increase risk if they have nosis and treatment tools to contain a high chance of causing a catastro- outbreaks before there is significant phe of their own, or if they prove to risk from artificial pandemics could be ineffective while at the same time reduce the total risk significantly. leading countries to avoid emission Similarly, a solution to the problem abatement (as discussed in Chapter of aligning advanced AI with human 2).157 values before we are able to create ad- vanced AI would reduce risk.161 The EMERGING TECHNOLOGY proposal of aiming for safety-enhanc- The factors that impact the risks of ing technologies ahead of risk-in- emerging technology are surround- creasing technologies in order to re- ed by a high degree of uncertainty. duce global catastrophic risk is called However, one plausible risk factor is ‘differential technological develop- misaligned incentives, which might ment’.162 cause nation-states to engage in arms races or under-regulate risky tech- UNKNOWN RISKS nology for economic reasons.158 The It is of course impossible to provide speed and suddenness of technolog- a detailed account of the drivers of ical breakthroughs could also be a currently unknown risks. Unknown risk factor, as sudden breakthroughs risks could in principle include both might leave inadequate time for so- exogenous and anthropogenic risks. cial and political institutions to ad- We have access to a long historical

76 Global Catastrophic Risks 2016 The speed and suddenness of technological breakthroughs could also be a risk factor, as sudden breakthroughs might leave inadequate time for social and political institutions to adjust their risk management mechanisms.

record, which might have brought risks in the future (although techno- many exogenous risks to our atten- logical progress could also reduce tion already. The historical record global catastrophic risk; see section offers us little or no guidance on an- 3.2.). Another important factor in- thropogenic risks. They will often fluencing these unknown risks is the be driven by social or technological quality of our foresight work, since change, so a higher rate of change the ability to plan for a risk could help could mean more currently unknown both avoidance and mitigation.

Global Catastrophic Risks 2016 77 CHAPTER 3 – RISK FACTORS AND INTERACTIONS BETWEEN RISKS

3.2. Shared risk factors and interactions between risks o far we have mainly discussed might also be reversed. A global ca- each risk in . But the tastrophe could lead to a breakdown S global catastrophic risks share of social and political institutions, numerous risk factors, and also inter- which in turn could cause an outbreak act with each other. In this section, of violence. In a 2008 paper, Nel and we discuss some salient examples of Righarts argued that even smaller nat- shared risk factors - variables associ- ural disasters such as earthquakes, ated with an altered level of risk - and tsunamis, and heat waves significant- interactions between risks. ly increase the risk of violent civil con- One important shared risk factor is flict in the short and medium term, governance. Good governance may especially in low- and middle-income help timely and appropriate prepa- countries.163 Similarly, there has been ration and response to risks. By the significant attention on the relation- same token, bad governance, nation- ship between climate change and po- ally or internationally, could increase litical conflict. Many scholars and se- the likelihood and potential impact of curity experts have argued that even every risk. Lax oversight might lead moderate climate change could in- to the accidental release of a danger- crease the risk of political violence ous engineered pathogen from a lab- because of conflicts over dwindling oratory. Dysfunctional governments natural resources (such as food and would be less able to intervene early water), massive international migra- in an outbreak to prevent it becom- tion and a range of other factors.164 ing a pandemic. Poor international Greater disasters are likely to lead to coordination might radically wors- even more upheaval, and might signif- en our prospects to avoid catastroph- icantly weaken the defenses against ic climate change, and internation- further catastrophic events. In this al tensions could increase the risk of way, one global catastrophe might nuclear war. Bad governance could in trigger another. principle also lead to over-prioritising Another set of factors which affect catastrophic risk reduction, but be- many global catastrophic risks is tech- cause of the political distortions dis- nological. Technological advances cussed in Chapter 4 we think this is change the system we live in by giv- less likely. ing actors new powers. These in turn While bad governance can cause could increase or decrease global cat- global catastrophes, the causality astrophic risk, even aside from cases

78 Global Catastrophic Risks 2016 Food stockpiles and the ability to rapidly increase production of alternate sources of food would increase resilience to a broad range of risks.

Global Catastrophic Risks 2016 79 CHAPTER 3 – RISK FACTORS AND INTERACTIONS BETWEEN RISKS

where the technology directly poses unscrupulous states to control their risk. It is hard to fully predict the ef- citizens, which makes it easier for fects of technological trends, but we states to engage in extreme behav- will highlight three trends which may iour. It might also weaken trust be- be relevant: economic productivity tween states, which could cause po- gains, surveillance, and distributed litical tension. However, surveillance . may make it more difficult for - Global productivity has increased cious states or terrorist groups to act dramatically in the last few decades. in secret. This will become increas- Partly, this is because of technological ingly important as barriers to access progress and organisational efficien- destructive weapons fall. Surveil- cy growth. Partly, it is due to automa- lance between countries could even tion, a form of technological progress facilitate international cooperation which we expect to continue. by making the actions of states more This growth may help us reduce transparent. It is not clear what over- risk because it lets us spend more on all effect surveillance has on levels of prevention and resilience. Measures global catastrophic risk. like clean energy and food stockpiles Distributed manufacturing is a set of are costly, and more likely to receive technologies that allow products to be investment when people are wealthy. designed and built without centralised In the tail case, with very powerful factories. They offer many benefits, artificial intelligence, there could be and may increase resilience to cata- a radical improvement in our ability strophe by spreading out production to manage other global catastrophic capacity. However, they also bypass risks. However, economic develop- some controls designed ment may exacerbate some risks, at to prevent the construction of destruc- least in the short term, for example tive weapons. 3D printing, an early by increasing greenhouse gas emis- form of such technology, has already sions. generated security risks by allowing The global surveillance disclosures people to create functional homemade by Edward Snowden and others re- firearms. More powerful forms of dis- vealed that state surveillance is now tributed manufacturing could increase extensive. In addition, the use of risk caused by malicious actors, by in- CCTV is expanding, and police and creasing access to powerful weaponry other law enforcement agencies are such as bioweapons. making increasing use of cameras in Nuclear war, geo-engineering, su- their day-to-day work. per-volcanoes, asteroids, and com- Increased surveillance has the po- ets all pose global catastrophic risk in tential both to exacerbate and reduce significant part because of the ‘par- global catastrophic risk. Extensive ticulate winter’ scenarios they might surveillance could make it easier for produce. By ejecting large amounts

80 Global Catastrophic Risks 2016 FIGURE 3.2. A WORKING GUN MADE FROM ON A 3D PRINTER165

of smoke, dust, and/or sulphates into tegies for resilience that address one the stratosphere they could cause risk address several. Food stockpiles global cooling, sunlight loss, ozone and the ability to rapidly incerase loss, and subsequent agricultural production of alternate sources of disruption.166 Because so many risks food would increase resilience to a share this mechanism, many stra- broad range of risks.167

Global Catastrophic Risks 2016 81 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

Chapter 4 Do institutions collectively underinvest in global catastrophic risk?

n Chapters 2 and 3 we gave an of attention to the risks, and suggests overview of several global cat- some mechanisms for reducing risk astrophic risks, the main fac- by countering the distortions. tors that affect their likelihood In the second half of the chapter, we and impact, and some of the examine the kinds of actors or insti- Ilevers available to influence them. In tutions that may be well-placed to act the first half of this chapter, we argue on global catastrophic risks, or have that market and political distortions a responsibility to do so. We look at mean that these risks are likely to be how they can help to correct the mar- systematically neglected by many ac- ket and political failures we consider tors. This increases the importance in the first half.

82 Global Catastrophic Risks 2016 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

We should expect global catastrophic risks to systematically receive less attention than they merit.

Global Catastrophic Risks 2016 83 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

4.1. Market and political failures t may seem surprising that rela- Supra-national institutions are gener- tively little effort has gone into ally weaker and international com- I global catastrophic risk mitiga- munities less cohesive, and therefore tion. However, we should expect glob- are less able to implement these solu- al catastrophic risks to systematically tions to collective action problems. receive less attention than they merit, Sometimes, such as with the World for structural reasons described be- Trade Organisation, nations do col- low. lectively give up some of their auton- omy in order to provide a public good, 4.1.1. GLOBAL but this process requires a great deal PUBLIC GOODS of negotiation and trust. Thus, insti- Global catastrophic risk reduction is tutions for aligning the incentives of a global public good – the benefits of nation-states to provide global public reduction spill over to other countries goods are typically less mature and and it is hard (and arguably unethi- less effective than those that provide cal) to only protect the countries that national public goods. contribute to risk reduction. Many global public goods tend to be under- 4.1.2. INTERGENERA- provided, primarily because coun- TIONAL PUBLIC GOODS tries try to free-ride.168 Every country Many of the benefits of global cata- might hope that other countries in- strophic risk reduction accrue to fu- vest in risk reduction, so they can get ture generations. However, the in- protected for free. If everyone shares terests of future generations tend to this hope, no one will invest. To over- be systematically neglected because come this, there is a need for good co- they cannot vote and have no direct ordination between actors. voice in the political process.171 For ex- In well-functioning states, na- ample, reducing the risk from cata- tional public goods, such as defence strophic climate change may provide and clean air,169 are provided despite only small benefits to many people the difficulties of collective action. alive today, but could be very valua- Sometimes, central regulation sup- ble to people who will exist in sixty plies these goods, sometimes mar- years’ time If we do not have formal ket mechanisms are employed, and processes for taking the interests of sometimes cooperative institutions future generations into account, we for supplying the goods emerge.170 may under-invest in risk reduction.

84 Global Catastrophic Risks 2016 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

Global Catastrophic Risks 2016 85 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

4.1.3. SPECIAL INTERESTS 4.1.4. UNPRECEDENTED The costs of regulatory actions to re- RISKS duce global catastrophic risks will Global catastrophic events occur very typically be concentrated on par- infrequently. In the last two millen- ticular industries, whereas the bene- nia, there may have only been two fits are dispersed. The small affected such events – the Plague of Justinian groups have strong incentives to lob- and the Black Death. Anthropogen- by and campaign, while those gaining ic global catastrophes are complete- the benefits will not regard those ben- ly unprecedented. Because unprec- efits as a voting priority. This means edented events are typically less that industry lobbying could wield salient, it is less likely that govern- disproportionate power over the reg- ments and voters will pay appropri- ulatory process.172 Consequently, the ate attention to them, in spite of their trade-offs made between econom- very high costs in expectation.173 ic profit and risk reduction could be skewed towards underprovision of risk reduction.

86 Global Catastrophic Risks 2016 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK? FIGURE 4.1. PREPAREDNESS IN THE REGION Multi-hazard national emergency preparedness & response plan in the African Region. 174

Implementation status (%)

0-24

25-49

50-74

75-100

Data not available

Not part of WHO African region

Not applicable

Global Catastrophic Risks 2016 87 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

4.2. Which actors can help reduce global catastrophic risk?

here is a broad range of ac- the risk of nuclear war. International tors that can help us over- coordination can aim to reduce the T coming the current neglect of risk of nuclear conflict between these global catastrophic risk. In this sec- major actors.177 tion, we give an overview of some There have been many agreements ways in which different groups, intended to reduce global catastroph- ranging from the international com- ic risk. Two of the more important munity to individuals, can contrib- examples178 are the Treaty on the ute to the mitigation of global cata- Non-Proliferation of Nuclear Weap- strophic risks. ons179 and the 1992 Framework Convention on Climate 4.2.1. THE INTERNA- Change, with its objective of pre- TIONAL COMMUNITY venting “dangerous” anthropogen- As discussed in section 4.1, reduc- ic climate change.180 Similarly, there tion of global catastrophic risk is a are several permanent internation- global public good. This means that al bodies, such as the International the international community will Atomic Energy Agency and the World probably need to play a major role in Health Organisation, which are (part- reducing global catastrophic risk. In ly) concerned with reducing global some cases, the global level of risk is catastrophic risks (nuclear war181 and significantly dependent on the ca- pandemics, respectively). The inter- pabilities of the countries that are national community often collabo- the weakest links.175 For example, to rates productively with relevant parts safeguard against the risk of a global of the academic community, as it did pandemic, the least prepared coun- when it set up the Intergovernmen- tries should be enabled to strengthen tal Panel on Climate Change.182 Many their health systems.176 In other cas- of the global catastrophic risks - not es, the risk of a catastrophe is mainly least those associated with emerg- dependent on a few major actors. For ing technologies - require a deep level example, the nuclear-weapon states of technical expertise to be properly have a disproportionate influence on managed.

88 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

Global Catastrophic Risks 2016 89 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

4.2.2. NATION-STATES how to respond. For instance, we Although the international commu- would benefit from more resources nity has a crucial role in the reduc- devoted to the study of global cata- tion of global catastrophic risk — strophic risks such as tail risk climate because this is a global public good change183 and particulate winter sce- — much power currently resides in narios.184 We would also benefit from the hands of nation-states. In prac- more research on technical solutions tice, international bodies have limit- that could reduce global catastroph- ed power unless they are backed up ic risks, such as clean energy sourc- by nation-states (especially the more es, or how to how to align the actions powerful ones). This means that na- of above-human-level AI with our tion-states are essential to galvanis- values.185 Since catastrophic risk re- ing action. They also need to lead the duction is a public good, it is unlike- process of implementation. For in- ly that this research will be carried stance, it is up to nation-states to see out by companies under competitive that emissions actually are cut to the pressure. Instead, it will most likely extent required by the Paris agree- require public or charitable funding. ment. Similarly, it is the responsibili- In fact, several academic institutions ty of nuclear-weapon states to guard that focus on global catastrophic risk against the possibility of accidental have already been set up through launch. public and philanthropic funds. Nation-states can also reform their Among them are Oxford’s Future of internal political processes in ways Humanity Institute and Cambridge’s that are conducive to reduction of Centre for the Study of Existential global catastrophic risks. Ensuring Risk. that decisions take account of the in- Beyond this, the research commu- terests of future generations is, for nity should contribute to global cat- instance, likely to lead to a more ap- astrophic risk reduction by promot- propriate degree of focus on glob- ing a of safety within areas of al catastrophic risk. Similarly, con- research that could have the poten- straining the excessive power of tial to cause a catastrophe through special interests could decrease dis- accident or misuse. This is especially tortions on decision-making and so in relevant for emerging technologies, turn reduce global catastrophic risk. where it is not always clear in advance whether there are any risks.186 4.2.3. THE RESEARCH COMMUNITY 4.2.4. INDUSTRY Many of the global catastrophic risks Competitive pressures mean that it are not well-understood, and more is often hard for companies to make research would allow more appropri- large moves on issues which do not ate decisions about when to act and improve their profits. They can, how-

90 Global Catastrophic Risks 2016 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

Global Catastrophic Risks 2016 91 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

ever, show leadership on reducing the 4.2.6. THE NON-PROFIT risk of catastrophic climate change SECTOR by choosing low carbon options at The non-profit sector contributes sig- the margin, or by developing clean nificantly to global catastrophic risk energy solutions. Just like the wider reduction. Non-profits are often less research community, they can and constrained than companies and na- should promote a culture of safety in tional governments, which means biotechnology and AI research as, for that they are free to work on intergen- example, Google DeepMind has done erational global public goods such as by setting up an AI ethics board.187 global catastrophic risk reduction -- if they can find donors that support 4.2.5. GENERAL PUBLIC that cause. There are many charities Some global catastrophic risks can be working on individual global cata- addressed by individual action. For strophic risks. In recent years, donors instance, individuals can decrease have been increasingly interested in the risk of catastrophic climate global catastrophic risk reduction as change (if by ever so little) by mak- a general category. This has led to the ing low-carbon consumption choic- creation of groups such as the Future es. Perhaps the more promising route of Life Institute in . to reduce global catastrophic risk for Besides supporting research, chari- individuals is, however, by exerting ties can also support direct interven- political pressure on policymakers. tions, e.g. to increase pandemic pre- For instance, voters could try to in- paredness in developing countries.188 fluence politicians to agree to carbon They can also exert political pressure, emissions cuts, and then to actually as International Physicians for the implement the agreements once they Prevention of Nuclear War did to re- are in place. duce the risk of a nuclear exchange during the cold war, winning the No- bel Peace Prize in 1985.189

92 Global Catastrophic Risks 2016 CHAPTER 4 – DO INSTITUTIONS COLLECTIVELY UNDERINVEST IN GLOBAL CATASTROPHIC RISK?

The research community should contribute to global catastrophic risk reduction by promoting a culture of safety within areas of research that could have the potential to cause a catastrophe through accident or misuse.

Global Catastrophic Risks 2016 93 CHAPTER 5 – WHAT CAN THE WORLD DO TO REDUCE GLOBAL CATASTROPHIC RISK?

Chapter 5 What can the world do to reduce global catastrophic risk?

n Chapter 2, we looked at dif- in order to reduce global catastroph- ferent global catastrophic risks ic risk. Some of these steps pertain to and their mechanisms. There individual risks, whereas others are was also some preliminary dis- cross-cutting opportunities which cussion of actions available to may reduce the chance or impact of Ilimit the risks. In Chapter 3 we ex- several different risks at once. plored the different factors which af- Our aim here is to offer some start- fect these risks. In Chapter 4 we con- ing points for considering action on sidered why global catastrophic risk is risks of global catastrophe and to probably neglected, and how differ- demonstrate that there are real ave- ent actors can help with it. Finally in nues to making progress. For more this chapter we draw these strands detailed discussion of the actions together and outline a few of the most available, there exists a rich literature promising steps that existing commu- on most of the specific risks. nities can take or are already taking

94 Global Catastrophic Risks 2016 CHAPTER 5 – WHAT CAN THE WORLD DO TO REDUCE GLOBAL CATASTROPHIC RISK?

Our aim here is to offer some starting points for considering action on risks of global catastrophe and to demonstrate that there are real avenues to making progress.

Global Catastrophic Risks 2016 95 CHAPTER 5 – WHAT CAN THE WORLD DO TO REDUCE GLOBAL CATASTROPHIC RISK?

CATASTROPHIC CLIMATE CHANGE NUCLEAR WAR • Research communities should in- • The international community crease their focus on understand- should continue the policy of nu- ing the pathways to and likelihood clear non-proliferation, and nu- of catastrophic climate change, and clear states can continue to reduce possible ways to respond.190 stockpiles. • Nations should continue to imple- • Nuclear-weapon states should ment and improve mechanisms continue to work to reduce the for emissions abatement such as chance of accidental launch or carbon taxes or tradable emissions escalation.193 quotas, as for non-catastrophic cli- • states can adopt mate change.191 military strategies that reduce the • Developed nations could commit chance of nuclear winter.194 to the goal of building no new coal- fired power stations without carbon capture and sequestration.192

96 Global Catastrophic Risks 2016 CHAPTER 5 – WHAT CAN THE WORLD DO TO REDUCE GLOBAL CATASTROPHIC RISK?

NATURAL PANDEMICS • The World Health Organisation, nation states, and other bodies should increase their planning for extremely bad pandemics.195 • International and research com- munities could improve disease ASTEROIDS AND COMETS surveillance, for example by reduc- • Research communities should con- ing the delay between scientific tinue working to detect and track breakthrough and the availability asteroids and commets with a di- of diagnostic tools. ameter of 1 km or more.197 • The global health community should improve developing world SUPERVOLCANIC ERUPTIONS capacity for response, for example • Research communities should con- by ensuring that vaccine produc- tinue to work on understanding tion facilities are well-distributed their causes, to increase predicta- around the world.196 bility.198

Global Catastrophic Risks 2016 97 CHAPTER 5 – WHAT CAN THE WORLD DO TO REDUCE GLOBAL CATASTROPHIC RISK?

BIOTECHNOLOGY • Research communities can further • Research communities should fur- investigate the potential impacts of ther investigate the possible risks solar radiation management.202 from emerging capabilities in bio- technology, and develop counter- UNKNOWN RISKS measures.199 • Research communities can contin- • Policymakers should continue to ue to develop methods and tools work with researchers to understand for horizon scanning and reduc- the and issues tion of unknown unknowns. that are likely to arise, and build • Research communities should planned adaptive risk regulation.200 identify and carefully investigate speculative threats. ARTIFICIAL INTELLIGENCE • Research communities should fur- CROSS-CUTTING OPPORTUNITIES ther investigate the possible risks • Research communities should fur- from artificial intelligence, and ther investigate ‘particulate winter’ work on developing possible solu- scenarios, both in forecasting and tions.201 mitigation strategies.203 • Policymakers can work with re- • Nations and local communities can searchers to understand the impli- continue to take steps to build their cations of advanced artificial intel- resilience to catastrophe.204 ligence. • Research communities should fo- cus greater attention on strategies GEO-ENGINEERING and technologies for resilience to • The international community and recovery from global catastro- should continue work to stabilise phe, for example by developing al- and eventually reduce concentra- ternate food sources.205 tions of greenhouse gases in the • Nations should work to incorporate atmosphere in order to reduce the the interests of future generations incentives to use solar radiation into their decision-making frame- management. works.206

98 Global Catastrophic Risks 2016 CHAPTER 5 – WHAT CAN THE WORLD DO TO REDUCE GLOBAL CATASTROPHIC RISK?

Research communities should focus greater attention on strategies and technologies for resilience to and recovery from global catastrophe, for example by developing alternate food sources.

Global Catastrophic Risks 2016 99 ENDNOTES

Endnotes 1. William Rosen, Justinian’s : Plague, Empire, and 15. http://www.atomicarchive.com/Docs/Begin/Einstein. the Birth of Europe (: Viking, 2007), 3, 209. shtml Percentages deduced from Demographic Internet 16. Ali Nouri and Christopher F. Chyba, “Biotechnology US Census Bureau, “International Programs, World and Biosecurity,” in Global Catastrophic Risks, ed. Nick Population,” accessed February 12, 2016, https://www. Bostrom and Milan M. Ćirković (, census.gov/population/international/data/worldpop/ 2008). table_history.php. Some earlier estimates put the death toll as high as 100 million. See B. Lee Ligon, “Plague: 17. Future of Life Institute, “AI Open Letter,” accessed A Review of Its History and Potential as a Biological February 3, 2016, http://futureoflife.org/ai-open-letter/. Weapon,” Seminars in Pediatric Infectious Diseases, 18. NASA, “The Relentless Rise of Carbon Dioxide,” Climate Return of the Gram-Positives, 17, no. 3 (July 2006): Change: Vital Signs of the Planet, accessed February 2, 161–70. 2016, http://climate.nasa.gov/climate_resources/24/. 2. Rosen, Justinian’s Flea, 3–5, 321–324. 19. Gernot Wagner and Martin L. Weitzman, Climate Shock : 3. Edward Wilson, “Thank You Vasili Arkhipov, the Man Who The Economic Consequences of a Hotter Planet (Princeton: Stopped Nuclear War,” , October 27, 2012, Press, 2015). http://www.theguardian.com/commentisfree/2012/ 20. Clearly, enormous literatures exist in each of these oct/27/vasili-arkhipov-stopped-nuclear-war. areas which can inform policy. We suggest only a small 4. This is adapted from Nick Bostrom, “Existential Risk selection of the policy options which are available. Prevention as Global Priority,” Global Policy 4, no. 1 21. NASA, “The Relentless Rise of Carbon Dioxide,”. (February 1, 2013): 16. 22. Sergey A. Zimov, Edward AG Schuur, and F. Stuart Chapin 5. , The Better Angels of Our Nature : A History of III, “Permafrost and the Global Carbon Budget,” accessed Violence and Humanity (London: Penguin, 2012),195. Note January 18, 2016, http://www.imedea.uib-csic.es/master/ that although the fatalities are too low, the non-fatality cambioglobal/Modulo_V_cod101619/Permafrost%20 impacts mean it could be argued that the Second World response.pdf. War was just about large enough to count as a global 23. Christopher B. Field et al., Climate Change 2014: catastrophe in our sense. Impacts, Adaptation, and : Summary for 6. Niall Johnson and Juergen Mueller, “Updating the Policymakers (Contribution of Working Group II to the Accounts: Global Mortality of the 1918-1920 ‘Spanish’ Fifth Assessment Report of the Intergovernmental Panel Influenza Pandemic,” Bulletin of the 76, on Climate Change, 2014), 14, http://epic.awi.de/37531/. no. 1 (2002): 105–15. 24. David King et al., “Climate Change–a ” 7. For further discussion see the sections on natural (Centre for Science Policy, , pandemics, nuclear war and super-volcanic eruptions in 2015), 45–46, www.csap.cam.ac.uk/projects/climate- Chapter 2. change-risk-assessment/. 8. Adapted from Nick Bostrom, “Existential Risk Prevention 25. Gernot Wagner and Martin L. Weitzman, Climate as Global Priority,” Global Policy 4, no. 1 (February 1, Shock : The Economic Consequences of a Hotter Planet 2013): 15–31, doi:10.1111/1758-5899.12002. (Princeton: Princeton University Press, 2015), 53–56. 9. Sandberg, A. & Bostrom, N. (2008): “Global Catastrophic 26. King et al., “Climate Change–a Risk Assessment,” chap. Risks Survey”, Technical. Report #2008-1, Future of 10. Humanity Institute, Oxford University: pp. 1-5. 27. King et al., “Climate Change–a Risk Assessment,” pt. 1. 10. N. H. Stern, The Economics of Climate Change : The Stern 28. “Green Light,” , December 19, 2015, http:// Review (Cambridge: Cambridge University Press, 2007), 47. www.economist.com/news/international/21684144-what- 11. “Institute for Health Metrics and Evaluation (IHME). GBD expect-after-deal-exceeded-expectations-green-light. Compare. Seattle, WA: IHME, University of Washington, 29. Rajendra K. Pachauri et al., Climate Change 2014: Synthesis 2015. Available from http://vizhub.healthdata.org/gbd- Report. Contribution of Working Groups I, II and III to the compare” from ‘ibid.’ Fifth Assessment Report of the Intergovernmental Panel on 12. Adapted from Nick Bostrom, “Existential Risk Prevention Climate Change, 2014, 10-13 as Global Priority,” Global Policy 4, no. 1 (February 1, 30. Wagner and Weitzman, Climate Shock, 53–56. 2013): 15–31, doi:10.1111/1758-5899.12002. 31. Field et al., Climate Change 2014: Impacts, Adaptation, 13. , Reasons and Persons (Oxford: Oxford and Vulnerability: Summary for Policymakers, chap. 16. University Press, 1984); Nick Bostrom, “Astronomical : The Opportunity Cost of Delayed Technological 32. Nicholas Stern, “The Economics of Climate Change,” The Development,” Utilitas 15, no. 03 (November 2003): American Economic Review 98, no. 2 (May 1, 2008): 1–37; 308–14; Nick Bostrom, “Existential Risk Prevention as William D. Nordhaus, “A Review of the ‘ on Global Priority”. the Economics of Climate Change,’” Journal of Economic Literature 45, no. 3 (September 1, 2007): 686–702; 14. We discuss this in more detail in chapter 4. Wagner and Weitzman, Climate Shock.

100 Global Catastrophic Risks 2016 ENDNOTES

33. Wagner and Weitzman, Climate Shock, chap. 3. 50. Anthony M. Barrett, Seth D. Baum, and Kelly Hostetler, 34. H. Damon Matthews et al., “The Proportionality of Global “Analyzing and Reducing the Risks of Inadvertent Nuclear Warming to Cumulative Carbon Emissions,” Nature 459, War Between the United States and Russia,” Science & no. 7248 (June 11, 2009): 829–32. Global Security 21, no. 2 (May 1, 2013): 106–33, doi:10.10 80/08929882.2013.798984. 35. Malte Meinshausen et al., “Greenhouse-Gas Emission Targets for Limiting Global Warming to 2 °C,” Nature 458, 51. Graham Allison, “At 50, the Cuban Missile Crisis as no. 7242 (April 30, 2009): 1158–62. Guide,” , June 15, 2012. 36. Ibid. 52. Zachary Keck, “Russia Threatens Nuclear Strikes Over Crimea,” July 11, 2014; National Institute for , 37. J. G. Shepherd, Geoengineering the Climate: Science, “Russia’s Nuclear Posture” (Fairfax, VA, 2015). Governance and Uncertainty (Royal Society, 2009), chap. 2. 53. Hellman, “How Risky Is Nuclear Optimism?”; Tom 38. Ibid., chap. 3. Hussain, “Are India and Pakistan Heading for a Nuclear 39. Gernot Wagner and Martin L. Weitzman, Climate Shock : Showdown?,” Al Jazeera, March 2016, http://www. The Economic Consequences of a Hotter Planet (Princeton: aljazeera.com/indepth/opinion/2016/03/india-pakistan- Princeton University Press, 2015), 54. heading-nuclear-showdown-160303053541342.html. 40. See Martin E. Hellman, “How Risky Is Nuclear Optimism?,” 54. Michael Krepon, “Pakistan’s and Bulletin of the Atomic Scientists 67, no. 2 (March 1, 2011): Deterrence Stability,” Stimson Center 10 (2012), http:// 47–56. indianstrategicknowledgeonline.com/web/Krepon_-_ Pakistan_Nuclear_Strategy_and_Deterrence_Stability.pdf. 41. Bulletin of the Atomic Scientists, “Nuclear Notebook,” Bulletin of the Atomic Scientists, accessed February 55. See Cirincione, “The Continuing Threat of Nuclear War”; 3, 2016, http://thebulletin.org/nuclear-notebook- Hellman, “How Risky Is Nuclear Optimism?” multimedia. 56. Cirincione, “The Continuing Threat of Nuclear War,” 396. 42. Hans M. Kristensen and Robert S. Norris, “US Nuclear 57. Matthew R. Costlow, “Do More Nukes Really Mean More Forces, 2015,” Bulletin of the Atomic Scientists 71, no. 2 Nuclear Crises? Not Necessarily,” Bulletin of the Atomic (2015): 108. Scientists, December 8, 2015, http://thebulletin.org/ 43. Hans M. Kristensen and Robert S. Norris, “Russian do-more-nukes-really-mean-more-nuclear-crises-not- Nuclear Forces, 2015,” Bulletin of the Atomic Scientists necessarily8965. 71, no. 3 (2015): 85. 58. For other suggestions for weapons systems see Seth 44. “A Nuke by Any Other Name,” Bulletin of the Atomic D. Baum, “Confronting the Threat of Nuclear Winter,” Scientists, May 17, 2012, http://thebulletin.org/nuke-any- Futures, Confronting Future Catastrophic Threats To other-name. Humanity, 72 (September 2015): 69–79, doi:10.1016/j. futures.2015.03.004. 45. US Office of Technology Assessment, “The Effects of Nuclear War,” 1979, 8–9, http://ota.fas.org/reports/7906. 59. http://thebulletin.org/nuclear-notebook-multimedia pdf; Joseph Cirincione, “The Continuing Threat of Nuclear 60. See the work of the Nuclear Threat Initiative at http:// War,” in Global Catastrophic Risks, ed. Nick Bostrom www.nti.org/. and Milan M. Ćirković (Oxford University Press, 2008), 386–390. 61. , Luke Oman, and Georgiy L. Stenchikov, “Nuclear Winter Revisited with a Modern Climate 46. This area has received less attention since the end of Model and Current Nuclear Arsenals: Still Catastrophic the Cold War. Other estimates give death tolls within the Consequences,” Journal of Geophysical Research: same order of magnitude. Ira Helfand et al., “Projected 112, no. D13 (July 16, 2007): 10, US Casualties and Destruction of US Medical Services doi:10.1029/2006JD008235. from Attacks by Russian Nuclear Forces,” Medicine & Global Survival 7, no. 2 (2002): 68–76; WHO, “Health 62. Ole J. Benedictow, “The Black Death: The Greatest Effects of the Chernobyl Accident and Special Health Care Catastrophe Ever,” History Today, 2005, http://www. Programmes” (WHO, 2006), http://www.who.int/ionizing_ historytoday.com/ole-j-benedictow/black-death- radiation/chernobyl/who_chernobyl_report_2006.pdf; greatest-catastrophe-ever; Edwin Dennis Kilbourne, Owen B. Toon et al., “Atmospheric Effects and Societal “Plagues and Pandemics: Past, Present, and Future,” Consequences of Regional Scale Nuclear Conflicts and in Global Catastrophic Risks, ed. Nick Bostrom and Acts of Individual Nuclear ,” Atmospheric Milan M. Ćirković (Oxford: Oxford University Press, and Physics 7, no. 8 (2007): 1973–2002. 2008), 295. Percentages deduced using population data from Demographic Internet Staff US Census Bureau, 47. For a good summary see Seth D. Baum, “Winter-Safe “International Programs, World Population,” accessed Deterrence: The Risk of Nuclear Winter and Its Challenge February 12, 2016, https://www.census.gov/population/ to Deterrence,” Contemporary Security Policy 36, no. 1 international/data/worldpop/table_history.php. (January 2, 2015): 126, doi:10.1080/13523260.2015.101 2346. 63. Niall Johnson and Juergen Mueller, “Updating the Accounts: Global Mortality of the 1918-1920 ‘Spanish’ 48. Alan Robock and Owen Brian Toon, “Local Nuclear War, Influenza Pandemic,” Bulletin of the History of Medicine Global Suffering,” Scientific American 302, no. 1 (2010): 76, no. 1 (2002): 105–15. 74–81. See the discussion in Givewell, “Nuclear Weapons Policy,” September 2015, http://www.givewell.org/labs/ 64. WHO, “Current WHO Phase of Pandemic Alert for Avian causes/nuclear-weapons-policy. Influenza H5N1,” WHO, accessed February 15, 2016, http://apps.who.int/csr/disease/avian_influenza/phase/ 49. Martin E. Hellman, “How Risky Is Nuclear Optimism?” en/index.html; Declan Butler, “Death-Rate Row Blurs

Global Catastrophic Risks 2016 101 ENDNOTES

Mutant Flu Debate,” Nature 482, no. 7385 (February 13, History of Medicine 76, no. 1 (2002): 105–15; Ole J. 2012): 289–289, doi:10.1038/482289a. Benedictow, “The Black Death: The Greatest Catastrophe 65. Marc Lipsitch and Thomas V. Inglesby, “Moratorium on Ever,” History Today, 2005, http://www.historytoday.com/ Research Intended To Create Novel Potential Pandemic ole-j-benedictow/black-death-greatest-catastrophe-ever; Pathogens,” mBio 5, no. 6 (December 31, 2014): WHO and others, Bugs, Drugs and Smoke: Stories from e02366–14, doi:10.1128/mBio.02366-14. Public Health, 2011, chap. 1, http://apps.who.int/iris/ handle/10665/44700; WHO, “HIV/AIDS Fact Sheet,” WHO, 66. Butler, “Death-Rate Row Blurs Mutant Flu Debate.” November 2015, http://www.who.int/mediacentre/ 67. Christopher JL Murray et al., “Estimation of Potential factsheets/fs360/en/. Global Pandemic Influenza Mortality on the Basis of Vital 77. Arnon Dar, “Influence of Supernovae, Gamma-Ray Registry Data from the 1918–20 Pandemic: A Quantitative Bursts, Solar Flares, and Cosmic Rays on the Terrestrial Analysis,” 368, no. 9554 (December 23, 2006): Environment,” in Global Catastrophic Risks, ed. Nick 2211–18, doi:10.1016/S0140-6736(06)69895-4. Bostrom and Milan M. Ćirković (Oxford: Oxford 68. Michael T. Osterholm, “Preparing for the Next Pandemic,” University Press, 2008). New England Journal of Medicine 352, no. 18 (May 5, 78. See the draft manuscript , “Will We Cause Our 2005): 1839–42, doi:10.1056/NEJMp058068; Neil M. Own Extinction? Natural versus Anthropogenic Extinction Ferguson et al., “Strategies for Mitigating an Influenza Risks” 2014. Ord’s argument is about extinction risks and Pandemic,” Nature 442, no. 7101 (July 27, 2006): 448–52, not the broader category of global catastrophic risks, but doi:10.1038/nature04795. it is very plausible that similar arguments would also show 69. Bruine de Bruin, Wändi, et al. “Expert judgments of that exogenous non-extinction global catastrophic risks pandemic influenza risks.” Global Public Health 1.2 are less likely than anthropogenic non-extinction global (2006): 179-194. catastrophic risks. 70. For two recent overviews see Peter Sands, Carmen 79. Michael Rampino, “Super-Volcanism and Other Mundaca-Shah, and Victor J. Dzau, “The Neglected Geophysical Processes of Catastrophic Import,” in Global Dimension of Global Security — A Framework for Catastrophic Risks, ed. Nick Bostrom and Milan M. Countering Infectious-Disease Crises,” New England Ćirković (Oxford: Oxford University Press, 2008). Journal of Medicine 0, no. 0 (January 13, 2016): null, 80. G. A. Zielinski et al., “Potential Atmospheric Impact of doi:10.1056/NEJMsr1600236; WHO and Report of the the Toba Mega-Eruption 71,000 Years Ago,” Geophysical High-level Panel on the Global Response to Health Crises, Research Letters 23, no. 8 (April 15, 1996): 837–40, “Protecting Humanity from Future Health Crises” (WHO, doi:10.1029/96GL00706; Rampino, “Super-Volcanism and January 25, 2016), http://www.un.org/News/dh/infocus/ Other Geophysical Processes of Catastrophic Import,” HLP/2016-02-05_Final_Report_Global_Response_to_ 209–210. Health_Crises.pdf. 81. Rampino, “Super-Volcanism and Other Geophysical 71. Stanley A. Plotkin, Adel A.F. Mahmoud, and Jeremy Processes of Catastrophic Import,” 211–212; Christine Farrar, “Establishing a Global Vaccine-Development S. Lane, Ben T. Chorn, and Thomas C. Johnson, “Ash Fund,” New England Journal of Medicine 373, no. 4 (July 23, from the Toba Supereruption in Shows No 2015): 297–300, doi:10.1056/NEJMp1506820. in East Africa at 75 Ka,” Proceedings of 72. Catherine Yen et al., “The Development of Global the National Academy of 110, no. 20 (May 14, Vaccine Stockpiles,” The Lancet Infectious Diseases 2013): 8025–29, doi:10.1073/pnas.1301474110. 15, no. 3 (March 2015): 340–47, doi:10.1016/S1473- 82. S. Sparks et al., Super-Eruptions: Global Effects and 3099(14)70999-5. Future Threats. London, Report of a Geological Society of 73. WHO, “International Health Regulations: Support to London Working Group, 2005, 6. Global Outbreak Alert and Response, and Building 83. W. Aspinall et al., “Volcano Hazard and Exposure in and Maintaining National Capacities,” 2015, http:// GFDRR Priority Countries and Risk Mitigation Measures,” apps.who.int/iris/bitstream/10665/199747/1/WHO_ Volcano Risk Study 0100806-00-1-R, 2011, 15; Rampino, HSE_GCR_2015.7_eng.pdf; WHO, “Pandemic Influenza “Super-Volcanism and Other Geophysical Processes of Preparedness Framework,” 2011, http://www.who.int/ Catastrophic Import,” 212–213; Susan Loughlin et al., influenza/resources/pip_framework/en/. Global and Risk (Cambridge University 74. Emily H. Chan et al., “Global Capacity for Emerging Press, 2015), 97. Infectious Disease Detection,” Proceedings of the 84. Sparks et al., Super-Eruptions, 20. National Academy of Sciences 107, no. 50 (December 14, 2010): 21701–6, doi:10.1073/pnas.1006219107. 85. Peter Schulte et al., “The Chicxulub Asteroid Impact and Mass Extinction at the - Boundary,” 75. Sands, Mundaca-Shah, and Dzau, “The Neglected Science 327, no. 5970 (March 5, 2010): 1214–18, Dimension of Global Security — A Framework for doi:10.1126/science.1177265. Countering Infectious-Disease Crises.” 86. National Research Council (U. S.). Committee to Review 76. Edwin Dennis Kilbourne, “Plagues and Pandemics: Past, Near-Earth-Object Surveys and Hazard Mitigation Present, and Future,” in Global Catastrophic Risks, ed. Strategies, Defending Planet Earth: Near-Earth Object Nick Bostrom and Milan M. Ćirković (Oxford: Oxford Surveys and Hazard Mitigation Strategies (Washington, University Press, 2008); William Rosen, Justinian’s DC: National Academies Press, 2010), 21. Flea : Plague, Empire, and the Birth of Europe (New York: Viking, 2007); Niall Johnson and Juergen Mueller, 87. National Research Council (U. S.). Committee to Review “Updating the Accounts: Global Mortality of the 1918- Near-Earth-Object Surveys and Hazard Mitigation 1920 ‘Spanish’ Influenza Pandemic,” Bulletin of the Strategies, Defending Planet Earth, 21.

102 Global Catastrophic Risks 2016 ENDNOTES

88. Clark R. Chapman, “The Hazard of near-Earth Asteroid Church_Biohazard04c.htm. Impacts on Earth,” Earth and 106. Nathan Myhrvold, “Strategic Terrorism: A Call to Action,” Letters 222, no. 1 (May 15, 2004): 11, doi:10.1016/j. SSRN Scholarly Paper (Rochester, NY: Social Science epsl.2004.03.004. Research Network, July 3, 2013), http://papers.ssrn.com/ 89. National Research Council (U. S.). Committee to Review abstract=2290382. Near-Earth-Object Surveys and Hazard Mitigation 107. Martin J. Rees, Our Final Century : Will Civilisation Survive Strategies, Defending Planet Earth, 23. the Twenty-First Century? (London: , 2004). 90. Alan Harris, “What Spaceguard Did,” Nature 453, no. 7199 108. Marc Lipsitch and Alison P. Galvani, “Ethical Alternatives (June 26, 2008): 1178–79, doi:10.1038/4531178a; NASA, to Experiments with Novel Potential Pandemic “NASA Space Telescope Finds Fewer Asteroids near Pathogens,” PLoS Med 11, no. 5 (May 20, 2014): Earth,” September 2011, http://www.nasa.gov/mission_ e1001646, doi:10.1371/journal.pmed.1001646. pages/WISE/news/wise20110929.html. 109. Lynn C. Klotz and Edward J. Sylvester, “The Unacceptable 91. Jason C. Reinhardt et al., “Asteroid Risk Assessment: A Risks of a Man-Made Pandemic,” Bulletin of the Atomic Probabilistic Approach,” Risk Analysis, July 1, 2015, 1-18, Scientists, August 7, 2012, http://thebulletin.org/ doi:10.1111/risa.12453. unacceptable-risks-man-made-pandemic. http://www. 92. Source: Alan Harris, “What Spaceguard Did,” Nature 453, the-scientist.com/?articles.view/articleNo/41263/title/ no. 7199 (June 26, 2008): 1178–79, doi:10.1038/4531178a. Moratorium-on-Gain-of-Function-Research/ 93. National Research Council (U. S.). Committee to Review 110. Lipsitch and Inglesby, “Moratorium on Research Intended Near-Earth-Object Surveys and Hazard Mitigation To Create Novel Potential Pandemic Pathogens,” 2. Strategies, Defending Planet Earth, 23. 111. Ibid. 94. Ibid., 22. 112. Gautam Mukunda, Kenneth A. Oye, and Scott C. Mohr, 95. Harris, “What Spaceguard Did.” “What Rough Beast?,” Politics and the Life Sciences 28, 96. A. Harris et al., “Asteroid Impacts and Modern Civilization: no. 2 (September 1, 2009): 8, doi:10.2990/28_2_2. Can We Prevent a Catastrophe,” Asteroids IV. University 113. Church, “A Synthetic Biohazard Non-Proliferation of Arizona Press, Tucson, AZ, 2015, http://elib.dlr. Proposal.” de/100120/1/3004_Harris_reprint.pdf. 114. Owen Cotton-Barratt, Sebastian Farquhar, and Andrew 97. Robert Carlson, “The Changing Economics of DNA Snyder-Beattie, “Beyond Risk-Benefit Analysis: Pricing Synthesis,” Nature Biotechnology 27, no. 12 (December for Gain-of-Function Research of Concern” 2009): 1091–94, doi:10.1038/nbt1209-1091. (Future of Humanity Institute, Global Piorities Project, 98. Terrence M. Tumpey et al., “Characterization of the March 2016). Reconstructed 1918 Spanish Influenza Pandemic Virus,” 115. Chyba and Greninger, “Biotechnology and Bioterrorism,” Science 310, no. 5745 (2005): 77–80. 151–153. 99. Markus Schmidt, “Diffusion of Synthetic : A 116. “DNA Sequencing Costs,” accessed March 31, 2016, Challenge to Biosafety,” Systems and 2, http://www.genome.gov/sequencingcosts/. no. 1–2 (June 2008): 1–6, doi:10.1007/s11693-008-9018-z. 117. Future of Life Institute, “AI Open Letter,” accessed 100. Ronald J. Jackson et al., “Expression of Mouse February 3, 2016, http://futureoflife.org/ai-open-letter/. Interleukin-4 by a Recombinant Ectromelia Virus 118. , “Artificial Intelligence as a Positive Suppresses Cytolytic Lymphocyte Responses and and Negative Factor in Global Risk,” in Global Catastrophic Overcomes Genetic Resistance to Mousepox,” Journal of Risks, ed. Nick Bostrom and Milan M. Ćirković, vol. 1 75, no. 3 (2001): 1205–10. (Oxford University Press, 2008), 303. 101. Christopher F. Chyba and Alex L. Greninger, 119. Carl Benedikt Frey and Michael A. Osborne, “The Future “Biotechnology and Bioterrorism: An Unprecedented of Employment” 2013; Nick Bostrom, Superintelligence : World,” Survival 46, no. 2 (2004): 148–149. Paths, Dangers, Strategies (Oxford: Oxford University 102. Sander Herfst et al., “ of Influenza Press, 2014), chap. 11. A/H5N1 Virus between Ferrets,” Science 336, no. 6088 120. Bostrom, Superintelligence, chap. 4. (2012): 1534–41; Masaki Imai et al., “Experimental Adaptation of an Influenza H5 HA Confers Respiratory 121. For extensive discussion of the ‘AI control problem’ see Droplet Transmission to a Reassortant H5 HA/H1N1 Virus Bostrom, Superintelligence. in Ferrets,” Nature 486, no. 7403 (June 21, 2012): 420–28, 122. Bostrom, Superintelligence, 18–21. For more detail about doi:10.1038/nature10831. the surveys see Vincent C. Müller and Nick Bostrom, 103. Nicholas Greig Evans, Marc Lipsitch, and Meira “Future Progress in Artificial Intelligence: A Survey of Levinson, “The Ethics of Biosafety Considerations in Expert Opinion,” in Fundamental Issues of Artificial Gain-of-Function Research Resulting in the Creation Intelligence, ed. Vincent C. Müller (Berlin: Synthese of Potential Pandemic Pathogens,” Journal of Medical Library; Springer, 2014). Ethics 41, no. 11 (November 1, 2015): 901–8, doi:10.1136/ 123. Bostrom, Superintelligence, 18–21 medethics-2014-102619. 124. Bostrom, Superintelligence, 20–21. 104. Richard A. Posner, Catastrophe : Risk and Response 125. Shepherd, Geoengineering the Climate, chap. 2. (Oxford: Oxford University Press, 2004), 78–79. 126. Shepherd, Geoengineering the Climate. 105. George Church, “A Synthetic Biohazard Non-Proliferation Proposal,” 2004, http://arep.med.harvard.edu/SBP/ 127. Alan Robock, Kirsten Jerch, and Martin Bunzl, “20 Reasons

Global Catastrophic Risks 2016 103 ENDNOTES

Why Geoengineering May Be a Bad Idea,” Bulletin of the factor, and is discussed in section 3.2. Atomic Scientists 64, no. 2 (2008): 14–59. 145. Centre for Science Policy, University of Cambridge et al., 128. Shepherd, Geoengineering the Climate, chap. 3. For “Climate Change–a Risk Assessment,” 38; International criticism of the ‘climate emergency’ argument for Energy Agency and Nuclear Energy Agency, “Technology SRM see Oliver Morton, The Planet Remade : How Roadmap: Nuclear Energy,” 2015; Ottmar Edenhofer and Geoengineering Could Change the World (London: Intergovernmental Panel on Climate Change. Working Granta, 2015), chap. 6; Jana Sillmann et al., “Climate Group III, Climate Change 2014 : Mitigation of Climate Emergencies Do Not Justify Engineering the Climate,” Change : Working Group III Contribution to the Fifth Nature Climate Change 5, no. 4 (2015): 290–92. Assessment Report of the Intergovernmental Panel on 129. Scott Barrett, “The Incredible Economics of Climate Change (New York, NY: Cambridge University Geoengineering,” Environmental and Resource Press, 2015), 516–518. Economics 39, no. 1 (2008): 45–54. 146. Edenhofer and Intergovernmental Panel on Climate 130. Alan Robock, Luke Oman, and Georgiy L. Stenchikov, Change. Working Group III, Climate Change 2014, 531. “Regional Climate Responses to Geoengineering 147. According to Sailor et al “development of nuclear weapons with Tropical and Arctic SO2 Injections,” Journal of has been aided in at least three countries (India, Iraq, and Geophysical Research: Atmospheres 113, no. D16 Israel) by use of research reactors obtained under the (August 27, 2008): D16101, doi:10.1029/2008JD010050. cover of peaceful research programs”.William C. Sailor et 131. Morton, The Planet Remade, chap. 4.. al., “A Nuclear Solution to Climate Change?,” Science 288, no. 5469 (2000): 1177–78. For a very good overview of the 132. Shepherd, Geoengineering the Climate, chap. 4. technical and political aspects of the impact of nuclear 133. Ibid., chap. 3. power on see A. Adamantiades and I. Kessides, “Nuclear Power for : 134. Morton, The Planet Remade, chap. 4. Current Status and Future Prospects,” 135. Seth D. Baum, Timothy M. Maher Jr, and Jacob Haqq- 37, no. 12 (December 2009): 5149–66, doi:10.1016/j. Misra, “Double Catastrophe: Intermittent Stratospheric enpol.2009.07.052. Geoengineering Induced by ,” 148. Edwin Dennis Kilbourne, “Plagues and Pandemics: Past, Environment Systems & Decisions 33, no. 1 (January 8, Present, and Future,” in Global Catastrophic Risks, ed. 2013): 168–80, doi:10.1007/s10669-012-9429-y. Nick Bostrom and Milan M. Ćirković (Oxford: Oxford 136. Shepherd, Geoengineering the Climate, chap. 4. University Press, 2008). 137. See for example Shepherd, Geoengineering the Climate; 149. James M. Hughes et al., “The Origin and Prevention of National Academy of Sciences, Climate Intervention: Pandemics,” Clinical Infectious Diseases 50, no. 12 (June Reflecting Sunlight to (Washington, D.C.: 15, 2010): 1636–40, doi:10.1086/652860. National Academies Press, 2015), http://www.nap.edu/ 150. Kate E. Jones et al., “Global Trends in Emerging Infectious catalog/18988; Solar Radiation Management Governance Diseases,” Nature 451, no. 7181 (February 21, 2008): Initiative et al., Solar Radiation Management: The 990–93, doi:10.1038/nature06536. Governance of Research (SRMGI, 2012). 151. Peter Sands, Carmen Mundaca-Shah, and Victor J. 138. Shepherd, Geoengineering the Climate; Barrett, “The Dzau, “The Neglected Dimension of Global Security — A Incredible Economics of Geoengineering.” Framework for Countering Infectious-Disease Crises,” 139. Arnon Dar, “Influence of Supernovae, Gamma-Ray New England Journal of Medicine 0, no. 0 (January 13, Bursts, Solar Flares, and Cosmic Rays on the Terrestrial 2016): null, doi:10.1056/NEJMsr1600236. Environment,” in Global Catastrophic Risks, ed. Nick 152. David Charles Denkenberger and Joshua Pearce, Feeding Bostrom and Milan M. Ćirković (Oxford: Oxford Everyone No Matter What : Managing after University Press, 2008). Global Catastrophe (Amsterdam: Academic Press, 2015). 140. Bryan Caplan, “The Totalitarian Threat,” in Global 153. S. Sparks et al., Super-Eruptions: Global Effects and Catastrophic Risks, ed. Nick Bostrom and Milan M. Future Threats. London, Report of a Geological Society of Ćirković (Oxford: Oxford University Press, 2008). London Working Group, 2005; National Research Council 141. Seth D. Baum, “Winter-Safe Deterrence: The Risk of (U. S.). Committee to Review Near-Earth-Object Surveys Nuclear Winter and Its Challenge to Deterrence,” and Hazard Mitigation Strategies, Defending Planet Contemporary Security Policy 36, no. 1 (January 2, 2015): Earth: Near-Earth Object Surveys and Hazard Mitigation 123–48, doi:10.1080/13523260.2015.1012346. Strategies (Washington, DC: National Academies Press, 142. Joseph Cirincione, “The Continuing Threat of Nuclear 2010), chap. 2 and 5. War,” in Global Catastrophic Risks, ed. Nick Bostrom and 154. David King et al., “Climate Change–a Risk Assessment” Milan M. Ćirković (Oxford University Press, 2008). (Centre for Science Policy, University of Cambridge, 143. Pavel Podvig, “Reducing the Risk of an Accidental 2015), pt. 1, www.csap.cam.ac.uk/projects/climate- Launch,” Science & Global Security 14, no. 2–3 (December change-risk-assessment/. 1, 2006): 75–115, doi:10.1080/08929880600992990. 155. Sybren Drijfhout et al., “Catalogue of Abrupt Shifts in 144. For an overview of the literature on ‘nuclear winter’ Intergovernmental Panel on Climate Change Climate see Seth D. Baum, “Winter-Safe Deterrence: The Risk Models,” Proceedings of the National Academy of of Nuclear Winter and Its Challenge to Deterrence,” Sciences 112, no. 43 (2015): E5777–86 Contemporary Security Policy 36, no. 1 (January 2, 2015): 156. For a qualified defence of this position see Oliver Morton, 123–48, doi:10.1080/13523260.2015.1012346. The The Planet Remade : How Geoengineering Could Change resilience of the food supply is also an important risk the World (London: Granta, 2015).

104 Global Catastrophic Risks 2016 ENDNOTES

157. J. G. Shepherd, Geoengineering the Climate: Science, Global Public Goods (Oxford: Oxford University Press, Governance and Uncertainty (Royal Society, 2009). 2007). 158. For discussions of worries about arms races and emerging 169. Clean air can of course also be influenced by in technologies see Christopher F. Chyba, “Biotechnology and neighbouring countries. the Challenge to Arms Control,” Arms Control Today 36, 170. Elinor Ostrom, Governing the Commons: the no. 8 (2006), http://search.proquest.com/openview/3251 of institutions for collective action. 1990. Cambridge 1e461cdf35678bea5679269ac45b/1?pq-origsite=gscholar; University Press. Future of Life Institute, “Open Letter on Autonomous Weapons,” accessed March 18, 2016, http://futureoflife. 171. It is hotly disputed how we should go about taking org/open-letter-autonomous-weapons/; Gernot Wagner the interests of future generations into account, but it and Martin L. Weitzman, Climate Shock : The Economic seems plausible that they should be given at least some Consequences of a Hotter Planet (Princeton: Princeton consideration in public policy. Indeed, this is a standard University Press, 2015), chap. 3. assumption in many governmental cost-benefit analyses. 159. Nick Bostrom, Superintelligence : Paths, Dangers, 172. Mancur Olson, The of Collective Action : Public Strategies (Oxford: Oxford University Press, 2014); Ali Goods and the of Groups, Harvard Economic Nouri and Christopher F. Chyba, “Biotechnology and Studies ; v. 124 (Cambridge, Mass: Biosecurity,” in Global Catastrophic Risks, ed. Nick Press, 1965). Bostrom and Milan M. Ćirković (Oxford University Press, 173. This is an example of availability bias, which is 2008). discussed along with other cognitive biases that can 160. For a discussion of this issue with respect to skew perceptions of global risks in Eliezer Yudkowsky, biotechnology and geo-engineering risks respectively see “Cognitive Biases Potentially Affecting Judgment of Global W. Paul Duprex et al., “Gain-of-Function Experiments: Risks,” in Global Catastrophic Risks, ed. Nick Bostrom Time for a Real Debate,” Nature Reviews Microbiology 13, and Milan M. Ćirković, vol. 1 (Oxford: Oxford University no. 1 (January 2015): 58–64, doi:10.1038/nrmicro3405; Press, 2008). Oliver Morton, The Planet Remade : How Geoengineering 174. Barrett, Why Cooperate?, chap. 2. Could Change the World (London: Granta, 2015). 175. WHO, “Summary of States Parties 2013 Report on IHR 161. Bostrom, Superintelligence. Core Capacity Implementation,” 2014, 6, http://www.who. 162. Bostrom, N. “Analyzing Human Extinction Scenarios and int/ihr/publications/WHO_HSE_GCR_2014.10/en/. Related Hazards”. Journal of Evolution and Technology. 176. Lawrence O. Gostin and Devi Sridhar, “Global Health and Vol. 9. No. 1. (2002) the Law,” New England Journal of Medicine 370, no. 18 163. Philip Nel and Marjolein Righarts, “Natural Disasters and (May 1, 2014): 1732–40, doi:10.1056/NEJMra1314094. the Risk of Violent Civil Conflict,” International Studies 177. Barrett, Why Cooperate? Quarterly 52, no. 1 (March 1, 2008): 159–85, doi:10.1111/ 178. Treaty on the Non-Proliferation of Nuclear Weapons, j.1468-2478.2007.00495.. Washington//London, 1 July 1968, in force 5 164. For an overview see Halvard Buhaug, Nils Petter March 1970, 729 UNTS 161 Gleditsch, and Ole Magnus Theisen, “Implications of 179. United Nations Framework Convention on Climate Climate Change for Armed Conflict” (, 2009); Change, Rio de Janeiro, 9 May 1992, in force 21 March Ragnhild Nordås and Nils Petter Gleditsch, “Climate 1994, 1771 UNTS 107. This convention provided the Change and Conflict,” Political Geography, Climate platform for the Kyoto and Paris climate change Change and Conflict, 26, no. 6 (August 2007): 627–38, agreements. doi:10.1016/j.polgeo.2007.06.003. 180. The IAEA’s mandate is limited to civil nuclear applications, 165. BBC, “Working Gun Made with 3D Printer,” BBC News, but it has probably reduced the likelihood of nuclear accessed February 22, 2016, http://www.bbc.co.uk/news/ terrorism by imposing detailed nuclear fissile accounting. science-environment-22421185. This verification process as well as on-site inspection also 166. See Chapter 2 and Alan Robock, Luke Oman, and make it very difficult for non-nuclear-weapon states to Georgiy L. Stenchikov, “Nuclear Winter Revisited with a develop nuclear weapons without detection. Modern and Current Nuclear Arsenals: 181. United Nations General Assembly, “Protection of Global Still Catastrophic Consequences,” Journal of Geophysical Climate for Present and Future Generations of Mankind,” Research: Atmospheres 112, no. D13 (July 16, 2007): December 1988, http://www.ipcc.ch/docs/UNGA43-53.pdf. D13107, doi:10.1029/2006JD008235; National Research Council (U. S.). Committee to Review Near-Earth-Object 182. Gernot Wagner and Martin L. Weitzman, Climate Surveys and Hazard Mitigation Strategies, Defending Shock : The Economic Consequences of a Hotter Planet Planet Earth: Near-Earth Object Surveys and Hazard (Princeton: Princeton University Press, 2015). Mitigation Strategies (Washington, DC: National 183. See for example Alan Robock, “Nuclear Winter,” Wiley Academies Press, 2010), chap. 2; S. Sparks et al., Super- Interdisciplinary Reviews: Climate Change 1, no. 3 (May 1, Eruptions: Global Effects and Future Threats. London, 2010): 418–27, doi:10.1002/wcc.45. Report of a Geological Society of London Working Group, 184. Nick Bostrom, Superintelligence : Paths, Dangers, 2005. Strategies (Oxford: Oxford University Press, 2014). 167. Denkenberger and Pearce, Feeding Everyone No Matter 185. For example, see the discussion of the responsibility What. of research laboratories conducting gain of function 168. For an excellent discussion of global public goods see research in W. Paul Duprex et al., “Gain-of-Function Scott Barrett, Why Cooperate? : The Incentive to Supply Experiments: Time for a Real Debate,” Nature

Global Catastrophic Risks 2016 105 ENDNOTES

Reviews Microbiology 13, no. 1 (January 2015): 58–64, 69–79, doi:10.1016/j.futures.2015.03.004. doi:10.1038/nrmicro3405. 195. The WHO’s International Health Regulations were an 186. Charles Arthur, “DeepMind: ‘Artificial Intelligence Is a Tool important development in this area, but the rules and That Humans Can Control and Direct,’” The Guardian, their implementation could be improved. See Rebecca June 9, 2015, sec. Technology, http://www.theguardian. Katz and Scott F Dowell, “Revising the International com/technology/2015/jun/09/-artificial- Health Regulations: Call for a 2017 Review Conference,” intelligence-tool-humans-control. The Lancet Global Health 3, no. 7 (July 2015): e352–53, 187. Anja Wolz, “Face to Face with Ebola — An Emergency doi:10.1016/S2214-109X(15)00025-X. Care Center in Sierra Leone,” New England Journal of 196. Ibid. Medicine 371, no. 12 (September 18, 2014): 1081–83, 197. See National Research Council (U. S.). Committee to doi:10.1056/NEJMp1410179. Review Near-Earth-Object Surveys and Hazard Mitigation 188. T. Frängsmyr, and I. Abrams, I., 1997. Peace, 1981-1990 Strategies, Defending Planet Earth, chap. 2. (Vol. 5). World Scientific. 198. Sparks et al., Super-Eruptions.. 189. National Research Council (U. S.). Committee to Review 199. See the discussion in Duprex et al., “Gain-of-Function Near-Earth-Object Surveys and Hazard Mitigation Experiments.”. Strategies, Defending Planet Earth: Near-Earth Object Surveys and Hazard Mitigation Strategies (Washington, 200. Nouri and Chyba, “Biotechnology and Biosecurity.” DC: National Academies Press, 2010), chap. 2. 201. Bostrom, Superintelligence. 190. For a defence of the importance of tail risk climate 202. Shepherd, Geoengineering the Climate. change see Martin L. Weitzman, “A Review of ‘The Stern 203. For discussion of different particulate winter scenarios Review on the Economics of Climate Change”, Journal of see Timothy M. Maher and Seth D. Baum, “Adaptation to Economic Literature 45, no. 3 (2007): 703–24. and Recovery from Global Catastrophe,” 5, 191. Wagner and Weitzman, Climate Shock, chap. 3. no. 4 (March 28, 2013): 1461–79, doi:10.3390/su5041461. 192. See Meinshausen et al., “Greenhouse-Gas Emission 204. Ibid. Targets for Limiting Global Warming to 2 °C.”. 205. See for example David Charles Denkenberger and Joshua 193. On this see Pavel Podvig, “Reducing the Risk Pearce, Feeding Everyone No Matter What : Managing of an Accidental Launch,” Science & Global Food Security after Global Catastrophe (Amsterdam: Security 14, no. 2–3 (December 1, 2006): 75–115, Academic Press, 2015). doi:10.1080/08929880600992990. 206. Nick Bostrom, “Existential Risk Prevention as Global 194. On this see Seth D. Baum, “Confronting the Threat Priority,” Global Policy 4, no. 1 (February 1, 2013): 15–31, of Nuclear Winter,” Futures, Confronting Future doi:10.1111/1758-5899.12002. Catastrophic Threats To Humanity, 72 (September 2015):

106 Global Catastrophic Risks 2016 ENDNOTES ACKNOWLEDGEMENTS

Acknowledgements

CONTACT INFO Authors: Global Priorities Project 16-17 St. Ebbes Street Oxford, OX1 1PT

[email protected]

The Global Challenges Foundation: Strandvägen 7A 114 56 Stockholm

info@ globalchallenges.org +46 (0) 709 98 97 97

Global Catastrophic Risks 2016 107 108 Global Catastrophic Risks 2016