TECHNOLOGY ROULETTE Managing Loss of Control As Many Militaries Pursue Technological Superiority

JUNE 2018 TECHNOLOGY ROULETTE Managing Loss of Control as Many Militaries Pursue Technological Superiority Richard Danzig About the Author RICHARD DANZIG is a Senior Fellow at the Johns Hopkins University Applied Physics Laboratory. He was Secretary of the Navy in the Clinton Administration. Acknowledgments This paper began as a small section added to a more In addition, Chris Beinecke, Steve Burton, Miles Brundage, general study I undertook for the Intelligence Advanced Paul Gewirtz, Michael Hopmeier, Lou Pringle, Al Martin, Projects Activity (IARPA). That study addressed the Alex Montgomery, and Jonathan Reiber all commented challenges for the United States of maintaining superiority on the manuscript and enriched my thinking. Peter Levin in the absorption and development of technologies offered his usual fine blend of trenchant criticism, editorial relevant to national security. Jason Matheny, the Director suggestions, and encouragement. Professor Jack Goldsmith of IARPA, supported that work and has been an invaluable helped me by assigning the paper to his Harvard Law School proponent and commentator on the work as it has class and eliciting useful comments from his students. I am developed. I am grateful to him for his exceptional intellect, grateful to all of them and to Chris Kirchhoff who helped character and concern about these issues. lead this discussion when a Boston snowstorm kept me from attending. Alex Downes provided valuable insight The Open Philanthropy Project supported broadening and on the political science literature relating to norms. Terry deepening this part of the IARPA paper and preparing it Leighton was particularly helpful in resolutely pursuing the for general publication. The Center for a New American challenging question of what caused the 1957 H1N1 global Security undertook that publication. I am particularly influenza pandemic. I am grateful as well to Gigi Kwik indebted to Helen Toner and Claire Zabel at Open Gronvall and Tom Inglesby for their correspondence with me Philanthropy and to Paul Scharre and Loren DeJonge on this subject. Schulman at CNAS for their contributions. I would like to thank Kara Frederick, Maura McCarthy, and Melody Cook Of course, none of those mentioned have responsibility for for their role in the review, production, and design of this the views expressed here or any errors that remain. I am report. grateful to all for their considerable help. Open Philanthropy’s support included sponsorship of a workshop earlier this year in which a draft of the paper was discussed by a number of participants with special insight. I am grateful to Jeff Alstott, Jim Baker, Jack Clark, Allan Dafoe, Craig Fields, Dan Geer, Larry Gershwin, Chris Inglis, Jason Matheny, Tim Maurer, Tyson Meadors, Jim Miller, Ernest Moniz, Anne Neuberger, David Relman, Peter Singer, Helen Toner, Renee Wegrzyn, Bob Work, and Philip Zelikow for their energetic participation. Craig Fields, Dan Geer and Phil Zelikow went beyond this by providing detailed follow up comments. Cover Photo Getty Images adapted by CNAS TECHNOLOGY ROULETTE Managing Loss of Control as Many Militaries Pursue Technological Superiority Table of Contents Executive Summary Introduction Differentiating Between Lay Fear and Expert Fear Why Focus on Military Development and Use of Technologies? 02 Executive Summary Causes of Loss of Control Inability to Reliably and Persuasively Measure These 04 Introduction Risks Humans in the Loop What Is to Be Done? 06 Differentiating Between Lay Fear and Expert Fear Conclusion Appendix 07 Why Focus on Military Development and Use End Notes of Technologies? 09 Causes of Loss of Control 12 Inability to Reliably and Persuasively Measure These Risks 13 Humans in the Loop 17 What Is to Be Done? 21 Conclusion 22 Appendix 23 End Notes 1 TECHNOLOGY & NATIONAL SECURITY | JUNE 2018 Technology Roulette: Managing Loss of Control as Many Militaries Pursue Technological Superiority Executive Summary We are not, however, impotent. With more attention the American military at least can dampen the likely his report recognizes the imperatives that inspire increase in accidents and moderate adverse conse- the U.S. military’s pursuit of technological supe- quences when they occur. Presuming that the United T riority over all potential adversaries. These pages States will be a victim of accidents, emergent effects, emphasize, however, that superiority is not synony- and sabotage, America should improve its planning for mous with security. Experience with nuclear weapons, coping with these consequences. This will involve real- aviation, and digital information systems should inform locating some of the immense energy our military invests discussion about current efforts to control artificial intel- in preparing for malevolence to planning for situations ligence (AI), synthetic biology, and autonomous systems. arising from inadvertent actions and interactions. In this light, the most reasonable expectation is that the The U.S. Department of Defense and intelligence introduction of complex, opaque, novel, and interactive agencies also must design technologies and systems not technologies will produce accidents, emergent effects, just for efficacy and efficiency, but also with more atten- and sabotage. In sum, on a number of occasions and in a tion to attributes that can mitigate the consequences of number of ways, the American national security estab- failure and facilitate resilient recovery. The pervasive lishment will lose control of what it creates. insecurity of digital information systems should be A strong justification for our pursuit of technolog- an object demonstration that it is exceedingly costly, ical superiority is that this superiority will enhance perhaps impossible, to attempt to counter loss of control our deterrent power. But deterrence is a strategy after we have become dependent on a new technology, for reducing attacks, not accidents; it discourages rather than at the time of design. malevolence, not inadvertence. In fact, technological Most demandingly, the United States also must proliferation almost invariably closely follows techno- work with its opponents to facilitate their control and logical innovation. Our risks from resulting growth in the minimize their risks of accidents. Twenty-first century number and complexity of interactions are amplified by technologies are global not just in their distribution, the fact that proliferation places great destructive power but also in their consequences. Pathogens, AI systems, in the hands of others whose safety priorities and stan- computer viruses, and radiation that others may acci- dards are likely to be less ambitious and less well funded dentally release could become as much our problem than ours. as theirs. Agreed reporting systems, shared controls, Accordingly, progress toward our primary goal, common contingency plans, norms, and treaties must be superiority, should be expected to increase rather pursued as means of moderating our numerous mutual than reduce collateral risks of loss of control. This risks. The difficulty of taking these important steps report contends that, unfortunately, we cannot reliably should remind us that our greatest challenges are not in estimate the resulting risks. Worse, there are no apparent constructing our relationships to technologies, it is in paths for eliminating them or even keeping them from constructing our relationships with each other. increasing. The benefit of an often referenced recourse, These arguments are made to the national security keeping “humans in the loop” of operations involving community. These reflections and recommendations, new technologies, appears on inspection to be of little however, should transcend their particulars and have and declining benefit. implications for all discussion about control of dangerous new technologies. 2 “Unlike a well-defined, precise game like Russian roulette, where the risks are visible to anyone capable of multiplying and dividing by six, one does not observe the barrel of reality.” —Nassim Nicholas Taleb 3 3 TECHNOLOGY & NATIONAL SECURITY | JUNE 2018 Technology Roulette: Managing Loss of Control as Many Militaries Pursue Technological Superiority Introduction assumes that our national security agencies will double down on our efforts to sustain supremacy; it even Innovations in technology and in warfare have long, presumes their success. perhaps always, been intertwined. The 20th century’s The focus in these pages is on a collateral, but con- world wars provided object lessons of the connection. sequential, set of risks that rise when America and, World War II especially shaped our present perspectives. following its lead, many other countries commit to using All military strategists recognized then, and remember new technologies to expand their military power. These now, that the war started with Blitzkrieg, a German risks include unintended consequences of complex strategy that exploited the combustion engine, message systems, errors in analysis or operation, interactive encryption, and the radio, and concluded with the code effects between separately developed systems, and dis- breakers in the U.K.’s Bletchley Park and the atomic tortions introduced by sabotage. bomb, a remarkable American orchestration of science The multinational reliance on ever-advancing tech- and engineering. nological capabilities is like loading increasing numbers At the end of that conflict, the United States had of bullets into increasing numbers of revolvers held immense

Load more