viewpoints

VDOI:10.1145/2770869 Thomas G. Dietterich and Eric J. Horvitz Viewpoint Rise of Concerns about AI: Reflections and Directions Research, leadership, and communication about AI futures.

ISCUSSIONS ABOUT ARTIFI- CIAL intelligence (AI) have jumped into the public eye over the past year, with sev- eral luminaries speaking Dabout the threat of AI to the future of humanity. Over the last several de- cades, AI—automated perception, learning, reasoning, and decision making—has become commonplace in our lives. We plan trips using GPS systems that rely on the A* algorithm to optimize the route. Our smartphones understand our speech, and Siri, Cor- tana, and Google Now are getting bet- ter at understanding our intentions. Machine vision detects faces as we take pictures with our phones and recogniz- es the faces of individual people when we post those pictures to Facebook. Internet search engines rely on a fabric of AI subsystems. On any day, AI pro- vides hundreds of millions of people with search results, traffic predictions, and recommendations about books AI has been in the headlines with such notable advances as self-driving vehicles, now under and movies. AI translates among lan- development at several companies; Google’s self-driving car is shown here. guages in real time and speeds up the operation of our laptops by guessing highest risk for complications, and AI lives, including those lost to accidents what we will do next. Several compa- algorithms are finding important nee- on our roadways and to errors made nies are working on cars that can drive dles in massive data haystacks, such as in medicine. Over the longer-term, themselves—either with partial hu- identifying rare but devastating side ef- advances in machine intelligence will man oversight or entirely autonomous- fects of medications. have deeply beneficial influences on ly. Beyond the influences in our daily The AI in our lives today provides a healthcare, education, transportation, lives, AI techniques are playing roles in small glimpse of more profound con- commerce, and the overall march of science and medicine. AI is already at tributions to come. For example, the science. Beyond the creation of new work in some hospitals helping physi- fielding of currently available technol- applications and services, the pursuit

cians understand which patients are at ogies could save many thousands of of insights about the computational OF GOOGLE.COM/SELFDRIVINGCAR/ COURTESY IMAGE

38 COMMUNICATIONS OF THE ACM | OCTOBER 2015 | VOL. 58 | NO. 10 viewpoints viewpoints

foundations of intelligence promises bustness may require self-monitoring to reveal new principles about cogni- architectures in which a meta-level pro- tion that can help provide answers to The AI in our lives cess continually observes the actions of longstanding questions in neurobiol- today provides a the system, checks that its behavior is ogy, psychology, and philosophy. consistent with the core intentions of On the research front, we have been small glimpse of the designer, and intervenes or alerts making slow, yet steady progress on more profound if problems are identified. Research “wedges” of intelligence, including contributions to come. on real-time verification and monitor- V work in , speech rec- ing of systems is already exploring such ognition, language understanding, layers of reflection, and these methods computer vision, search, optimization, could be employed to ensure the safe and planning. However, we have made operation of autonomous systems.3,6 surprisingly little progress to date on A second set of risks is cyberattacks: building the kinds of general intelli- criminals and adversaries are continu- gence that experts and the lay public We believe computer scientists ally attacking our computers with vi- envision when they think about “Arti- must continue to investigate and ad- ruses and other forms of malware. AI ficial Intelligence.” Nonetheless, ad- dress concerns about the possibili- algorithms are as vulnerable as any vances in AI—and the prospect of new ties of the loss of control of machine other software to cyberattack. As we roll AI-based autonomous systems—have intelligence via any pathway, even if out AI systems, we need to consider the stimulated thinking about the poten- we judge the risks to be very small and new attack surfaces that these expose. tial risks associated with AI. far in the future. More importantly, we For example, by manipulating train- A number of prominent people, urge the research ing data or preferences and trade-offs mostly from outside of computer sci- community to focus intensively on a encoded in utility models, adversaries ence, have shared their concerns that second class of near-term challenges could alter the behavior of these sys- AI systems could threaten the survival for AI. These risks are becoming sa- tems. We need to consider the implica- of humanity.1 Some have raised con- lient as our society comes to rely on au- tions of cyberattacks on AI systems, es- cerns that machines will become su- tonomous or semiautonomous com- pecially when AI methods are charged perintelligent and thus be difficult to puter systems to make high-stakes with making high-stakes decisions. control. Several of these speculations decisions. In particular, we call out five U.S. funding agencies and corporations envision an “intelligence chain reac- classes of risk: bugs, cybersecurity, the are supporting a wide range of cyberse- tion,” in which an AI system is charged “Sorcerer’s Apprentice,” shared auton- curity research projects, and artificial with the task of recursively designing omy, and socioeconomic impacts. intelligence techniques will themselves progressively more intelligent ver- The first set of risks stems from pro- provide novel methods for detecting sions of itself and this produces an gramming errors in AI software. We are and defending against cyberattacks. “intelligence explosion.”4 While for- all familiar with errors in ordinary soft- For example, machine learning can be mal work has not been undertaken to ware; bugs frequently arise in the de- employed to learn the fingerprints of deeply explore this possibility, such velopment and fielding of software ap- malware, and new layers of reflection a process runs counter to our current plications and services. Some software can be employed to detect abnormal understandings of the limitations that errors have been linked to extremely internal behaviors, which can reveal cy- computational complexity places on costly outcomes and deaths. The verifi- berattacks. Before we put AI algorithms algorithms for learning and reasoning. cation of software systems is challeng- in control of high-stakes decisions, we However, processes of self-design and ing and critical, and much progress must be confident these systems can optimization might still lead to signifi- has been made—some relying on AI survive large-scale cyberattacks. cant jumps in competencies. advances in theorem proving. Many A third set of risks echo the tale of the Other scenarios can be imagined in non-AI software systems have been de- Sorcerer’s Apprentice. Suppose we tell a which an autonomous computer sys- veloped and validated to achieve high self-driving car to “get us to the airport tem is given access to potentially dan- degrees of quality assurance. For exam- as quickly as possible!” Would the au- gerous resources (for example, devices ple, the software in autopilot and space- tonomous driving system put the pedal capable of synthesizing billons of bio- craft systems is carefully tested and to the metal and drive at 125 mph, put- logically active molecules, major por- validated. Similar practices must be ap- ting pedestrians and other drivers at tions of world financial markets, large plied to AI systems. One technical chal- risk? Troubling scenarios of this form weapons systems, or generalized task lenge is to guarantee that systems built have appeared recently in the press. markets9). The reliance on any comput- via machine learning methods behave Many of the dystopian scenarios of out- ing systems for control in these areas is properly. Another challenge is to en- of-control superintelligences are varia- fraught with risk, but an autonomous sure good behavior when an AI system tions on this theme. All of these exam- system operating without careful hu- encounters unforeseen situations. Our ples refer to cases where humans have man oversight and failsafe mechanisms automated vehicles, home robots, and failed to correctly instruct the AI system could be especially dangerous. Such a intelligent cloud services must perform on how it should behave. This is not a system would not need to be particu- well even when they receive surprising new problem. An important aspect of larly intelligent to pose risks. or confusing inputs. Achieving such ro- any AI system that interacts with people

OCTOBER 2015 | VOL. 58 | NO. 10 | COMMUNICATIONS OF THE ACM 39 viewpoints

is that it must reason about what people poor decisions. Here again, AI meth- Year Study on ,10,c intend rather than carrying out com- ods can help solve these problems by which is planning centuries of ongoing mands literally. An AI system must ana- anticipating when human control will studies about advances in AI and its in- lyze and understand whether the behav- be required and providing people with fluences on people and society. ior that a human is requesting is likely to the critical information that they need. The computer science community be judged as “normal” or “reasonable” A fifth set of risks concern the broad must take a leadership role in explor- by most people. In addition to relying on influences of increasingly competent ing and addressing concerns about internal mechanisms to ensure proper automation on socioeconomics and machine intelligence. We must work to behavior, AI systems need to have the ca- the distribution of wealth.2 Several ensure that AI systems responsible for pability—and responsibility—of work- lines of evidence suggest AI-based au- high-stakes decisions will behave safely ing with people to obtain feedback and tomation is at least partially respon- and properly, and we must also examine guidance. They must know when to stop sible for the growing gap between per and respond to concerns about poten- and “ask for directions”—and always be capita GDP and median wages. We tial transformational influences of AI. open for feedback. need to understand the influences Beyond scholarly studies, computer sci- Some of the most exciting opportu- of AI on the distribution of jobs and entists need to maintain an open, two- nities for deploying AI bring together on the economy more broadly. These way channel for communicating with the complementary talents of people questions move beyond computer sci- the public about opportunities, con- and computers.5 AI-enabled devices ence into the realm of economic poli- cerns, remedies, and realities of AI. are allowing the blind to see, the deaf cies and programs that might ensure to hear, and the disabled and elderly to that the benefits of AI-based productiv- b See http://www.aaai.org/Organization/pres- walk, run, and even dance. AI methods ity increases are broadly shared. idential-panel.php. c See https://ai100.stanford.edu. are also being developed to augment Achieving the potential tremendous

human cognition. As an example, pro- benefits of AI for people and society will References totypes have been aimed at predicting require ongoing and vigilant attention 1. Bostrum, N. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, 2014. what people will forget and helping to the near- and longer-term challenges 2. Brynjolfsson, E. and McAfee, A. The Second Machine them to remember and plan. Moving to to fielding robust and safe computing Age: Work Progress, and Prosperity in a Time of Brilliant Technologies. W.W. Norton & Company, New York, 2014. the realm of scientific discovery, people systems. Each of the first four challenges 3. Chen, F. and Rosu, G. Toward monitoring-oriented working together with the Foldit online listed in this Viewpoint (software qual- programming: A paradigm combining specification and 8 implementation. Electr. Notes Theor. Comput. Sci. 89, game were able to discover the struc- ity, cyberattacks, “Sorcerer’s Appren- 2 (2003), 108–127. ture of the virus that causes AIDS in only tice,” and shared autonomy) is being 4. Good, I.J. Speculations concerning the first ultraintelligent machine. In Advances in Computers, three weeks, a feat that neither people addressed by current research, but even Vol. 6. F.L. Alt and M. Rubinoff, Eds., Academic Press, nor computers working alone could greater efforts are needed. We urge our 1965, 31–88. 5. Horvitz, E. Principles of mixed-initiative user match. Other studies have shown how research colleagues and industry and interfaces. In Proceedings of CHI ’99, ACM SIGCHI the massive space of galaxies can be ex- government funding agencies to devote Conference on Human Factors in Computing Systems (Pittsburgh, PA, May 1999); http://bit.ly/1OEyLFW. plored hand-in-hand by people and ma- even more attention to software qual- 6. Huang, J. et al. ROSRV: Runtime verification for robots. chines, where the tireless AI astronomer ity, cybersecurity, and human-computer Runtime Verification, (2014), 247–254. 7. Kamar, E., Hacker, S., and Horvitz, E. Combining understands when it needs to reach out collaboration on tasks as we increasing- human and machine intelligence in large-scale crowdsourcing. AAMAS 2012 (Valencia, Spain, June and tap the expertise of human astrono- ly rely on AI in safety-critical functions. 2012); http://bit.ly/1h6gfbU. mers.7 There are many opportunities At the same time, we believe schol- 8. Khatib, F. et al. Crystal structure of a monomeric retroviral protease solved by protein folding game ahead for developing real-time systems arly work is needed on the longer-term players. Nature Structural and Molecular Biology 18 that involve a rich interleaving of prob- concerns about AI. Working with col- (2011), 1175–1177. 9. Shahaf, D. and Horvitz, E. Generalized task markets for lem solving by people and machines. leagues in economics, political science, human and machine computation. AAAI 2010, (Atlanta, However, building these collabora- and other disciplines, we must address GA, July 2010), 986–993; http://bit.ly/1gDIuho. 10. You, J. A 100-year study of artificial intelligence? tive systems raises a fourth set of risks the potential of automation to disrupt Science (Jan. 9, 2015); http://bit.ly/1w664U5. stemming from challenges with fluid- the economic sphere. Deeper study is ity of engagement and clarity about also needed to understand the poten- Thomas G. Dietterich ([email protected]) is a states and goals. Creating real-time tial of superintelligence or other path- Distinguished Professor in the School of Electrical Engineering and Computer at Oregon State University systems where control needs to shift ways to result in even temporary losses in Corvallis, OR, and president of the Association for the rapidly between people and AI sys- of control of AI systems. If we find there Advancement of Artificial Intelligence (AAAI). tems is difficult. For example, airline is significant risk, then we must work to Eric J. Horvitz (horvitz@.com) is Distinguished Scientist and Director of the lab in accidents have been linked to misun- develop and adopt safety practices that Redmond, Washington. He is the former president of derstandings arising when pilots took neutralize or minimize that risk. We AAAI and continues to serve on AAAI’s Strategic over from autopilots.a The problem is should study and address these con- Planning Board and Committee on Ethics in AI. that unless the human operator has cerns, and the broader constellation Copyright held by authors. been paying very close attention, he or of risks that might come to the fore in she will lack a detailed understanding the short- and long-term, via focused Watch the authors discuss of the current situation and can make their work in this exclusive research, meetings, and special efforts Communications video. such as the Presidential Panel on Long- http://cacm.acm.org/ b videos/rise-of-concerns- a See http://en.wikipedia.org/wiki/China_Air- Term AI Futures organized by the AAAI about-ai-reflections-and- lines_Flight_006. in 2008–2009 and the One Hundred directions

40 COMMUNICATIONS OF THE ACM | OCTOBER 2015 | VOL. 58 | NO. 10