Communications of the Acm

COMMUNICATIONS CACM.ACM.ORG OF THEACM 03/2018 VOL.61 NO.03

A Programmable Programming Language How Can We Trust a Robot? Policy Intervention Fosters Cybersecurity Intervention Here Comes Everyone … to Communications

Association for Q&A with Yann LeCun Computing Machinery INSPIRING MINDS FOR 200 YEARS Ada’s Legacy illustrates the depth and diversity of writers, things, and makers who have been inspired by Ada Lovelace, the English mathematician and writer. The volume commemorates the bicentennial of Ada’s birth in December 1815, celebrating her many achievements as well as the impact of her work which reverberated widely since the late 19th century. This is a unique contribution to a resurgence in Lovelace scholarship, thanks to the expanding influence of women in science, technology, engineering and mathematics.

ACM Books is a new series of high quality books for the computer science community, published by the Association for Computing Machinery with Morgan & Claypool Publishers. Marquette University’s Third Annual ETHICS OF BIG DATA SYMPOSIUM

The emerging world of big data brings with it ethical, social and legal issues. Are you prepared to navigate the challenges and the opportunities? Friday, April 27 8 a.m. – Noon Marquette University Milwaukee, Wisconsin For event information and registration, go to marquette.edu/ethics-of-big-data or contact Dr. Thomas Kaczmarek at [email protected] or 414.288.6734.

The deployment of big data brings desirable opportunities to understand, recommend and advise. But the sensitivity of personal data and unintended consequences of algorithmic decisions present us with ethical and moral decisions. The symposium will cover ethical and legal considerations for practitioners, including discussions and dilemmas of agency, fairness, public perception and privacy.

Hosted by Northwestern Mutual. COMMUNICATIONS OF THE ACM

Departments News Viewpoints, cont’d.

5 Editor’s Letter 27 Legally Speaking Here Comes Everybody … Will the Supreme Court Nix Reviews to Communications of Bad Patents? By Andrew A. Chien Considering the longer-term implications of a soon-to-be-decided 7 Cerf’s Up U.S. Supreme Court case. Unintended Consequences By Pamela Samuelson By Vinton G. Cerf 30 Computing Ethics 9 Vardi’s Insights Ethics Omission A Declaration of the Dependence Increases Gases Emission of Cyberspace A look in the rearview mirror at By Moshe Y. Vardi Volkswagon software engineering. By Simon Rogerson 10 Letters to the Editor 15 Keep the ACM Code of Ethics As It Is 33 The Profession of IT 15 In Pursuit of Virtual Life The Computing Profession 12 BLOG@CACM Scientists are simulating biological Taking stock of progress toward The Costs and Pleasures of organisms and replicating evolution a computing profession a Computer Science Teacher in the lab. How far can they expand since this column started in 2001. Mark Guzdial considers the boundaries of virtual life? By Peter J. Denning the enormous opportunity costs By Samuel Greengard of computer science teachers, 36 Viewpoint while Bertrand Meyer ponders 18 The Construction Industry Impediments with Policy the pleasures of arguing in the 21st Century Interventions to Foster Cybersecurity with graduate students. Three-dimensional printing A call for discussion of governmental and other new technologies investment and intervention in 43 Calendar are revitalizing the business support of cybersecurity. of building buildings. By Fred B. Schneider 117 Careers By Keith Kirkpatrick 39 Viewpoint 21 The State of Fakery Responsible Research with Crowds: Last Byte How digital media could be Pay Crowdworkers authenticated, from computational, at Least Minimum Wage 120 Q&A legal, and ethical points of view. High-level guidelines for the The Network Effect By Esther Shein treatment of crowdworkers. The developer of convolutional By M. Six Silberman, Bill Tomlinson, neural networks looks at Rochelle LaPlante, Joel Ross, their impact, today and Viewpoints Lilly Irani, and Andrew Zaldivar in the long run. By Leah Hoffmann 24 Privacy and Security 42 Viewpoint Making Security Sustainable Computational Social Science ≠ Can there be an Internet Computer Science + Social Data of durable goods? The important intersection By Ross Anderson of computer science and social science. By Hanna Wallach IMAGE: BLUE BRAIN PROJECT / EPFL ©2005 – 2018. ALL RIGHTS RESERVED. RESERVED. / EPFL ©2005 – 2018. ALL RIGHTS BRAIN PROJECT BLUE IMAGE:

2 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 03/2018 VOL. 61 NO. 03

Practice Contributed Articles Review Articles

86 How Can We Trust a Robot? If intelligent robots take on a larger role in our society, what basis will humans have for trusting them? By Benjamin Kuipers

Watch the author discuss his work in this exclusive Communications video. https://cacm.acm.org/videos/ how-can-we-trust-a-robot

Research Highlights

46 72 98 Technical Perspective A Graph-Theoretic Framework 46 Bitcoin’s Underlying Incentives 62 A Programmable Programming Traces Task Planning The unseen economic forces Language By Nicole Immorlica that govern the Bitcoin protocol. As the software industry enters By Yonatan Sompolinsky the era of language-oriented 99 Time-Inconsistent Planning: and Aviv Zohar programming, it needs programmable A Computational Problem programming languages. in Behavioral Economics 54 Operational Excellence By Matthias Felleisen, By Jon Kleinberg and Sigal Oren in April Fools’ Pranks Robert Bruce Findler, Matthew Flatt, Being funny is serious work. Shriram Krishnamurthi, 108 Technical Perspective By Thomas A. Limoncelli Eli Barzilay, Jay McCarthy, On Heartbleed: and Sam Tobin-Hochstadt A Hard Beginnyng Makth 58 Monitoring in a DevOps World a Good Endyng Perfect should never By Kenny Paterson Watch the authors discuss be the enemy of better. their work in this exclusive By Theo Schlossnagle Communications video. 109 Analysis of SSL Certificate https://cacm.acm.org/ videos/a-programmable- Reissues and Revocations Articles’ development led by programming-language in the Wake of Heartbleed queue.acm.org By Liang Zhang, David Choffnes, 72 The Wisdom of Older Technology Tudor Dumitras¸, Dave Levin, (Non)Users Alan Mislove, Aaron Schulman, Older adults consistently and Christo Wilson reject digital technology even when designed to be accessible and trustworthy. By Bran Knowles and Vicki L. Hanson

About the Cover: This month’s cover story 78 Evolution Toward Soft(er) Products explores the Racket As software becomes a larger part of language project, a programming language all products, traditional (hardware) designed to support manufacturers are becoming, in language-oriented programming. The story essence, software companies. of this project, now By Tony Gorschek celebrating 20 years, is told by its creators beginning on p. 62. Association for Computing Machinery Cover illustration Advancing Computing as a Science & Profession

IMAGES BY SERGEY ILYASOV; TTL MEDIA TTL MEDIA SERGEY ILYASOV; BY IMAGES by Chris Labrooy.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 3 COMMUNICATIONS OF THE ACM Trusted insights for computing’s leading professionals.

Communications of the ACM is the leading monthly print and online magazine for the computing and information technology fields. Communications is recognized as the most trusted and knowledgeable source of industry information for today’s computing professional. Communications brings its readership in-depth coverage of emerging areas of computer science, new trends in information technology, and practical applications. Industry leaders use Communications as a platform to present and debate various technology implications, public policies, engineering challenges, and market trends. The prestige and unmatched reputation that Communications of the ACM enjoys today is built upon a 50-year commitment to high-quality editorial content and a steadfast dedication to advancing the arts, sciences, and applications of information technology.

ACM, the world’s largest educational STAFF EDITORIAL BOARD ACM Copyright Notice and scientific computing society, delivers DIRECTOR OF PUBLICATIONS EDITOR-IN-CHIEF Copyright © 2018 by Association for resources that advance computing as a Scott E. Delman Andrew A. Chien Computing Machinery, Inc. (ACM). science and profession. ACM provides the [email protected] [email protected] Permission to make digital or hard copies computing field’s premier Digital Library of part or all of this work for personal and serves its members and the computing Deputy to the Editor-in-Chief or classroom use is granted without Executive Editor profession with leading-edge publications, Lihan Chen fee provided that copies are not made Diane Crawford conferences, and career resources. [email protected] or distributed for profit or commercial Managing Editor advantage and that copies bear this Thomas E. Lambert Acting Executive Director SENIOR EDITOR notice and full citation on the first Senior Editor Deputy Executive Director and COO Moshe Y. Vardi page. Copyright for components of this Andrew Rosenbloom Patricia Ryan work owned by others than ACM must Senior Editor/News Director, Office of Information Systems NEWS be honored. Abstracting with credit is Lawrence M. Fisher Wayne Graves Co-Chairs permitted. To copy otherwise, to republish, Web Editor Director, Office of Financial Services William Pulleyblank and Marc Snir to post on servers, or to redistribute to David Roman Darren Ramdin Board Members lists, requires prior specific permission Rights and Permissions Director, Office of SIG Services Monica Divitini; Mei Kobayashi; and/or fee. Request permission to publish Deborah Cotton Donna Cappo Michael Mitzenmacher; Rajeev Rastogi; from [email protected] or fax Editorial Assistant Director, Office of Publications François Sillion (212) 869-0481. Jade Morris Scott E. Delman VIEWPOINTS For other copying of articles that carry a Co-Chairs ACM COUNCIL Art Director code at the bottom of the first or last page President Tim Finin; Susanne E. Hambrusch; or screen display, copying is permitted Andrij Borys John Leslie King; Paul Rosenbloom Vicki L. Hanson Associate Art Director provided that the per-copy fee indicated Vice-President Board Members in the code is paid through the Copyright Margaret Gray Stefan Bechtold; Michael L. Best; Judith Bishop; Cherri M. Pancake Assistant Art Director Clearance Center; www.copyright.com. Secretary/Treasurer Andrew W. Cross; Mark Guzdial; Mia Angelica Balaquiot Richard Ladner; Carl Landwehr; Beng Chin Ooi; Elizabeth Churchill Production Manager Subscriptions Past President Francesca Rossi; Loren Terveen; An annual subscription cost is included Bernadette Shade Marshall Van Alstyne; Jeannette Wing Alexander L. Wolf Advertising Sales Account Manager in ACM member dues of $99 ($40 of Chair, SGB Board Ilia Rodriguez which is allocated to a subscription to Jeanna Matthews PRACTICE Communications); for students, cost Co-Chairs, Publications Board Chair is included in $42 dues ($20 of which Jack Davidson and Joseph Konstan Columnists Stephen Bourne and Theo Schlossnagle is allocated to a Communications Members-at-Large David Anderson; Phillip G. Armour; Board Members subscription). A nonmember annual Gabriele Anderst-Kotis; Susan Dumais; Michael Cusumano; Peter J. Denning; Eric Allman; Samy Bahra; Peter Bailis; subscription is $269. Elizabeth D. Mynatt; Pamela Samuelson; Mark Guzdial; Thomas Haigh; Terry Coatta; Stuart Feldman; Nicole Forsgren; Eugene H. Spafford Leah Hoffmann; Mari Sako; Camille Fournier; Benjamin Fried; ACM Media Advertising Policy SGB Council Representatives Pamela Samuelson; Marshall Van Alstyne Pat Hanrahan; Tom Killalea; Tom Limoncelli; Communications of the ACM and other Paul Beame; Jenna Neefe Matthews; Kate Matsudaira; Marshall Kirk McKusick; ACM Media publications accept advertising Barbara Boucher Owens CONTACT POINTS Erik Meijer; George Neville-Neil; in both print and electronic formats. All Copyright permission Jim Waldo; Meredith Whittaker advertising in ACM Media publications is BOARD CHAIRS [email protected] at the discretion of ACM and is intended Education Board Calendar items to provide financial support for the various Mehran Sahami and Jane Chu Prey CONTRIBUTED ARTICLES [email protected] Co-Chairs activities and services for ACM members. Practitioners Board Change of address Current advertising rates can be found Terry Coatta and Stephen Ibaraki James Larus and Gail Murphy [email protected] Board Members by visiting http://www.acm-media.org or Letters to the Editor William Aiello; Robert Austin; Elisa Bertino; by contacting ACM Media Sales at REGIONAL COUNCIL CHAIRS [email protected] Kim Bruce; Alan Bundy; Peter Buneman; (212) 626-0686. ACM Europe Council Carl Gutwin; Yannis Ioannidis; Chris Hankin WEBSITE Gal A. Kaminka; Ashish Kapoor; Single Copies ACM India Council http://cacm.acm.org Kristin Lauter; Igor Markov; Bernhard Nebel; Single copies of Communications of the Madhavan Mukund Lionel M. Ni; Adrian Perrig; ACM are available for purchase. Please ACM China Council Marie-Christine Rousset; Krishan Sabnani; contact [email protected]. Yunhao Liu AUTHOR GUIDELINES http://cacm.acm.org/about- m.c. schraefel; Ron Shamir; Alex Smola; COMMUNICATIONS OF THE ACM communications/author-center Josep Torrellas; Sebastian Uchitel; PUBLICATIONS BOARD Michael Vitale; Hannes Werthner; (ISSN 0001-0782) is published monthly Co-Chairs Reinhard Wilhelm by ACM Media, 2 Penn Plaza, Suite 701, Jack Davidson; Joseph Konstan ACM ADVERTISING DEPARTMENT New York, NY 10121-0701. Periodicals Board Members 2 Penn Plaza, Suite 701, New York, NY RESEARCH HIGHLIGHTS postage paid at New York, NY 10001, Phoebe Ayers; Anne Condon; Nikil Dutt; 10121-0701 Co-Chairs and other mailing offices. Roch Guerrin; Chris Hankin; T (212) 626-0686 Azer Bestavros and Shriram Krishnamurthi Yannis Ioannidis; XiangYang Li; F (212) 869-0481 Board Members POSTMASTER Sue Moon; Michael L. Nelson; Martin Abadi; Amr El Abbadi; Sanjeev Arora; Please send address changes to Sharon Oviatt; Eugene H. Spafford; Michael Backes; Maria-Florina Balcan; Communications of the ACM Stephen N. Spencer; Alex Wade; Advertising Sales Account Manager Andrei Broder; Doug Burger; Stuart K. Card; 2 Penn Plaza, Suite 701 Julie R. Williamson Ilia Rodriguez Jeff Chase; Jon Crowcroft; Alexei Efros; New York, NY 10121-0701 USA [email protected] Alon Halevy; Sven Koenig; Steve Marschner; ACM U.S. Public Policy Office Greg Morrisett; Tim Roughgarden; Printed in the USA. Adam Eisgrau, Media Kit [email protected] Guy Steele, Jr.; Margaret H. Wright; Director of Global Policy and Public Affairs Nicholai Zeldovich; Andreas Zeller 1701 Pennsylvania Ave NW, Suite 300, Washington, DC 20006 USA WEB T (202) 659-9711; F (202) 667-1066 Association for Computing Machinery Chair (ACM) James Landay

E R E C Computer Science Teachers Association 2 Penn Plaza, Suite 701 Board Members S Y A C E L L E Jake Baskin New York, NY 10121-0701 USA Marti Hearst; Jason I. Hong; P

T E H Executive Director T (212) 869-7440; F (212) 869-0481 Jeff Johnson; Wendy E. MacKay N I I S Z M A G A

4 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 editor’s letter

DOI:10.1145/3183638 Andrew A. Chien Here Comes Everybody. . . to Communications I am pleased to announce a new Communications of the ACM initiative with the ambitious goal of expanding the Communications community globally.

I hope it means “Here Comes Every- choices about security, privacy, free ments in computing. We hope to revisit body to Communications.”a Why bring speech, and control—reflect distinc- regions about once every two years. everybody to Communications? To in- tive regional, national, and commu- I am pleased to report that we clude important voices and perspec- nity culture. have already begun. The first special tives in the conversation about the Communications, the flagship pub- section will focus on China, where present and future of computing. lication of world’s leading computing we have convened an extraordinary With the proliferation of computing professional society, should be an in- team of industry and academic lead- into every industry, every product, clusive forum, spanning this commu- ers committed to attend the kick- and every aspect of society, not only nity. It should be a universal forum, an off meeting (set for March 8th at the has computing spread throughout inclusive, global community, with ac- UChicago Beijing Center), and we are the society and economy of every na- tive participation from everyone. actively planning successors in other tion, but the computing profession To that end, I am pleased to an- parts of the world. has spread into communities around nounce a Communications global initia- How we do this is instrumental. the globe. tive. Its goal is to give deeper insight, The special sections will be led by a One natural consequence is that focused coverage, and elevate distinc- regional team who will nominate, invention and innovation in comput- tive and compelling highlights of com- select, and drive authorship of the ing, once concentrated in a few re- puting drawn from regions around the section’s content. By design, this gions, is now a global enterprise. And, world. This initiative will add a 20–30 will encourage active participation while the technical foundations of page special section to a few issues of a growing global community in computing may be universal,b along of Communications each year. Each Communications. To drive creation of with technical challenges of function- special section will be a collection the regional teams and the entire ality, scale, reliability, and perhaps of short pieces, focused on a region series of special sections, we are usability; increasingly, the design of and chartered to represent the best of adding a new section to the Edito- many of a system’s most important computing leadership and distinctive rial Board of Communications. Ser- aspects—how they relate to society, development. We will bring a sharp endipitously, this creates new op- government, structure of commerce, focus on: portunities for you to volunteer and and individual enlightenment and ˲˲ Leading technical and research ad- contribute to the magazine. perspective as well as fundamental vances and activities; Expect to hear more about this ˲˲ Leading and emerging industry late in 2018 when we print the first a This title borrows from Shirky’s 2008 book and research players; special section! that described the growing power of groups ˲˲ Innovation and shape of comput- of individuals to organize large-scale activi- ing in the region; and, Andrew A. Chien, EDITOR-IN-CHIEF ties, using Internet tools, and without traditional corporate organizations. In fact, this is ˲˲ Unique challenges and opportuni- what the ACM has been doing successfully for ties … and by doing so enrich the entire Andrew A. Chien is the William Eckhardt Distinguished over 50 years. computing community’s perspective! Service Professor in the Department of Computer Science b More on this later, as growing excitement about Communications’ global initiative at the University of Chicago, Director of the CERES Center neuromorphic and quantum computing sug- for Unstoppable Computing, and a Senior Scientist at Argonne National Laboratory. gest we may soon see a proliferation of comput- will visit regions around the world in ing bases. Leading to a question, are we even turn, shifting its spotlight to match the engaged with the full breadth of computing? pace and impact of interesting develop- Copyright held by author.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 5 ACM Books. In-depth. Innovative. Insightful. ACM and Morgan & Claypool Publishers present ACM Books: an all-new series of educational , research and reference works for the computing community. Inspired by the need for high-quality computer science publishing at the graduate, faculty and professional levels, ACM Books is a ordable, current, and compre hensive in scope. ACM Books collections are available under an ownership model with archival rights included. We invite you to learn more about this exciting new program.

For more info please visit http://books.acm.org or contact ACM at [email protected]

Association for Computing Machinery 2 PennM Plaza, Suite 701 New York, NY 10121-0701, USA Phone:& +1-212-626-0658C Email: [email protected]

Morgan & Claypool Publishers 1210 Fifth Avenue, Suite 250 M San Rafael, CA 94901, USA Phone: +1-415-462-0004 &C Email: [email protected]

ACM_Books_LJ_FullPageAd_Winter2016_V01.indd 1 1/15/16 10:56 AM cerf’s up

DOI:10.1145/3184402 Vinton G. Cerf Unintended Consequences

HEN THE INTERNET was abound—some monetary, some for the ignore it. It is impossible to filter in real being developed, scien- sake of influence, some purely narcis- time. YouTube alone gets 400 hours of tists and engineers in sistic, some to share beneficial knowl- video uploaded per minute (that is 16.7 academic and research edge, to list just a few. A serious prob- days of a 24-hour television channel). settings drove the pro- lem is that the information comes in The platforms that support content Wcess. In their world, information was a all qualities, from incalculably valuable are challenged to cope with the scale medium of exchange. Rather than buy- to completely worthless and in some of the problem. Unlike other media ing information from each other, they cases seriously damaging. Even setting that have time and space limitations exchanged it. Patents were not the first aside malware, DDOS attacks, hacking (page counts for newspapers and choice for making progress; rather, and the like, we still have misinforma- magazines; minutes for television open sharing of designs and protocols tion, disinformation, “fake news,” and radio channels) making it more were preferred. Of course, there were “post-truth alternate facts,” fraudulent feasible to exercise editorial oversight, instances where hardware and even propositions, and a raft of other exe- the Internet is limitless in time and software received patent and licens- crable material often introduced cause space, for all practical purposes. ing treatment, but the overwhelm- deliberate harm to victims around the Moreover, automated algorithms are ing trend was to keep protocols and world. The vast choice of information subject to error or can be misled by the standards open and free of licensing available to readers and viewers leads action of botnets, for example, that pre- constraints. The information-sharing to bubble/echo chamber effects that tend to be human users “voting” in favor ethic contributed to the belief that reinforce partisan views, prejudices, of deliberate or accidental misinforma- driving down barriers to information and other societal ills. tion. Purely manual review of all the and resource sharing was an impor- There are few international norms incoming content is infeasible. The con- tant objective. Indeed, the Internet as concerning content. Perhaps child sumers of this information might be we know it today has driven the barrier pornography qualifies as one type of able to use critical thinking to reject to the generation and sharing of infor- content widely agreed to be unaccept- invalid content but that takes work and mation to nearly zero. Smartphones, able and which should be filtered and some people are often unwilling or laptops, tablets, Web cams, sensors, removed from the Internet. There are unable to do that work. If we are to cope and other devices share text, imagery, national norms that vary from country with this new environment, we are going video, and other data with a tap of a fin- to country regarding legitimate and to need new tools, better ways to validate ger or through autonomous operation. illegitimate/illegal content. The result sources of information and factual data, Blogs, tweets, social media, and Web is a cacophony of fragmentation and broader agreement on transnational page updates, email and a host of other misinformation that pollutes the vast norms all the while striving to preserve communication mechanisms course majority of useful or at least innocuous freedom of speech and freedom to through the global Internet in torrents content to be found on the Internet. The hear, enshrined in the Universal Decla- (no pun intended). Much, if not most, question before us is what to do about ration of Human Rights.a I hope our of the information found on the Inter- the bad stuff. It is irresponsible to computer science community will find net seems to me to be beneficial; a har- or invent ways to engage, using power- vest of human knowledge. But there ful computing, artificial intelligence, are other consequences of the reduced The question machine learning, and other tools to threshold for access to the Internet. enable better quality assessment of The volume of information is mind- before us is the ocean of content contained in our boggling. I recently read one estimate what to do about growing online universe. that 1.7 trillion images were taken (and many shared) in the past year. The Twit- the bad stuff. a http://www.un.org/en/universal-declaration- tersphere is alive with vast numbers human-rights/ of brief tweets. The social media have captured audiences and contributors Vinton G. Cerf is vice president and Chief Internet Evangelist at Google. He served as ACM president from 2012–2014. measured in the billions. Incen- tives to generate and share content Copyright held by author.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 7 Introducing ACM Transactions on Human-Robot Interaction

Now accepting submissions to ACM THRI

As of January 2018, the Journal of Human-Robot Interaction (JHRI) has become an ACM publication and has been rebranded as the ACM Transactions on Human-Robot Interaction (THRI).

Founded in 2012, the Journal of HRI has been serving as the premier peer-reviewed interdisciplinary journal in the eld.

Since that time, the human-robot interaction eld has experienced substantial growth. Research ndings at the intersection of robotics, human-computer interaction, arti cial intelligence, haptics, and natural language processing have been responsible for important discoveries and breakthrough technologies across many industries.

THRI now joins the ACM portfolio of highly respected journals. It will continue to be open access, fostering the widest possible readership of HRI research and information. All issues will be available in the ACM Digital Library.

Co-Editors-in-Chief Odest Chadwicke Jenkins of the University of Michigan and Selma Šabanović of Indiana University plan to expand the scope of the publication, adding a new section on mechanical HRI to the existing sections on computational, social/behavioral, and design-related scholarship in HRI.

The inaugural issue of the rebranded ACM Transactions on Human-Robot Interaction is planned for March 2018.

To submit, go to https://mc.manuscriptcentral.com/thri vardi’s insights

DOI:10.1145/3182625 Moshe Y. Vardi A Declaration of the Dependence of Cyberspace

OHN PERRY BARLOW, the down daily, suffering from “retail recently they have come to realize that a famed founder of the Elec- apocalypse.”a And while cyberattacks slightly depreciating currency value tronic Frontier Foundation, are now a daily occurrence, there are (about 2% per year) is better for eco- a digital-rights advocacy growing fears of a possible cyberattack nomic growth. To achieve this, central group, passed away on Feb. that will knock out U.S. power grids.b bankers use a variety of sophisticated J6, 2018. In 1996, Barlow published “A What happens in cyberspace does not monetary tools to manage the money Declaration of the Independence of stay in cyberspace! supply, taking into account a large Cyberspace.” It offered a rebuttal to To my mind, however, nothing number of economic indicators. This is Internet governance by national gov- epitomizes the hubris of the techno- an enormously complicated task chal- ernments, opening with “Govern- utopianists more than the idea of rein- lenging the best economic minds. In ments of the Industrial World, you venting money. In October 2008, the contrast, the supply of cryptocurrencies weary giants of flesh and steel, I come mysterious Satoshi Nakamoto posted a is a priori limited, and its gyrating value from Cyberspace, the new home of white paperc on the new domain of bit- is determined by trading decisions Mind. On behalf of the future, I ask you coin.org, in which he asserted, “What made by “investors.” So the idea that of the past to leave us alone.” is needed is an electronic payment sys- apolitical cryptocurrencies will replace It is hard to believe that such a naïve tem based on cryptographic proof in- political money is a delusion. view of cyberspace was taken seriously stead of trust.” Bitcoin is based on a Just like other speculative bubbles, just about 20 years ago, that people really P2P network where transactions are the cryptocurrency bubble will also blow believed simplistic statements such as cryptographically verified and recorded up at some point, though the timing is “We believe that from ethics, enlight- in a public distributed ledger based on quite unpredictable. But the risk is not ened self-interest, and the common- blockchain, a distributed-consensus only to gullible speculators. As time weal, our governance will emerge.” protocol. We now seem to be in the goes on, cryptocurrencies get more en- But we must remember that 20 years ago midst of a bitcoin mania, with the value meshed in our economic system and the Internet was indeed some kind of a of bitcoins gyrating wildly, making the risk of financial contagiongrows. New World, seemingly outside the double-digit moves in a single week. Financial contagion refers to the spread shackling legacy of traditional gover- There is also significant evidenced that of market disturbances, typically on the nance. That was also before the Internet the bitcoin-exchange markets are be- downside, between different economic and the World Wide Web became dom- ing manipulated. But bitcoin has only institutions and between different inated by giant corporations, and be- been the first of dozens of cryptocurren- countries. The cryptocurrency bubble fore Tim Berners-Lee, the recipient of cies, and initial coin offerings, which is, in my opinion, a growing systemic the 2016 ACM Turing Award for invent- raise funds for issuing new cryptocur- financial risk (and there are also the ing the World Wide Web, declared in rencies, and are growing in popularity. issues of susceptibility to cyberattacks 2017 that “The system is failing.” But even if bitcoin solved (quite im- and voracious energy appetite). What we have also learned in the past perfectly) the verifiability and distribut- Just as you cannot separate the mind 20 years that while cyberspace may be ed-trust issues, the idea of apolitical and the body, you cannot separate indeed “the new home of Mind,” it is in- money is a dangerous fantasy. Verifi- cyberspace and physical space. It is extricably connected to the physical ability and trust are only two require- time to accept this dependence and world. Indeed, the economic impact of ments from a currency. Other require- act accordingly. the Web has been and continues to be ments, which are intimately related, are Follow me on Facebook, Google+, profound. Newspapers are struggling to value and supply. Central banks used to and Twitter. survive because advertising income, strive for a stable currency value. More which has been their main source of rev- Moshe Y. Vardi ([email protected]) is the Karen Ostrum George Distinguished Service Professor in Computational enue, has migrated to the Web, with Engineering and Director of the Ken Kennedy Institute for a https://goo.gl/PEMP55 Google and Facebook as the main bene- Information Technology at Rice University, Houston, TX, USA. b https://goo.gl/esziCL He is the former Editor-in-Chief of Communications. ficiaries. While e-commerce escalates, c https://goo.gl/55N41V traditional retail outlets are shuttering d https://goo.gl/dP1GTz Copyright held by author.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 9 letters to the editor

DOI:10.1145/3183570 Keep the ACM Code of Ethics As It Is

HE PROPOSED CHANGES to professionalism. Computing It would have been instructive if the ACM Code of Ethics professionals should engage thoughtfully Subramanian had, say, compared and and Professional Conduct, and responsibly with the systems they contrasted LGI with blockchain-based as discussed by Don Got- create, maintain, and use. Lawyers, mechanisms for enforcing distributed terbarn et al. in “ACM Code politicians, and other members of society protocols, as they are two radically dif- of Ethics: A Guide for Positive Ac- do not always fully understand the ferent mechanisms for achieving es- T1 tion” (Digital Edition, Jan. 2018), are complexity of modern sociotechnical sentially the same objective. generally misguided and should be systems; computing professionals can rejected by the ACM membership. help this understanding. Humans References 1. Minsky, N.H. The imposition of protocols over open The changes attempt to, for example, understand such concepts as harm, distributed systems. IEEE Transactions on Software create real obligations on members dignity, safety, and well-being; computing Engineering 17, 2 (1991), 183–195. 2. Serban, C., Chen, Y., Zhang, W., and Minsky, N. The to enforce hiring quotas/priorities professionals can apply them in their concept of decentralized and secure electronic with debatable efficacy while ACM technical decisions. We invite Simonelis to marketplace. The Journal of Electronic Commerce Research 8, 1–2 (June 2008), 79–101. members are neither HR specialists read the Code and accompanying nor psychologists; create “safe spac- materials in more detail, as many of his Naftaly Minsky, Edison, NJ, USA es for all people,” a counterproduc- claims, in our opinion, misread the Code. tive concept causing problems in a We also invite everyone else to read it, number of universities; counter ha- too; https://ethics.acm.org/2018-code- Author Responds: rassment while not being lawyers or draft-3/ Comparing LGI and blockchain-based smart police officers; enforce privacy while Catherine Flick, Leicester, U.K., contracts would be a great idea, as Minsky not being lawyers; ensure “the public and Keith Miller, St. Louis, MO, USA says, as they are radically different good” while not being elected lead- approaches to decentralization. However, ers; encourage acceptance of “social from an adoption standpoint what matters responsibilities” while not defining ‘Law-Governed Interaction’ for most is mass adoption at scale. For that to them or being elected leaders or those Decentralized Marketplaces happen, the value created by decentralization charged with implementing govern- Given today’s sometimes gratuitous would have to be shared among all users in ment policy; and monitor computer efforts toward centralized control over some tangible way. Blockchain-based systems integrated into society for the Internet, I found it refreshing to decentralization, in addition to ensuring “fair access” while not being lawyers read Hemang Subramanian’s article secure low-cost distributed transactions, or part of the C-suite. “Decentralized Blockchain-Based Elec- could make network effects fungible through ACM is a computing society, not a tronic Marketplaces” (Jan. 2018) argu- the issuance of cryptocoins that can be society of activists for social justice, ing that applications like electronic exchanged for fiat currency; for example, community organizers, lawyers, po- marketplaces and social networks Steem is a popular social network that issues lice officers, or MBAs. The proposed would benefit from adecentralized im- virtual currency powered by the blockchain. changes add nothing related specifi- plementation, describing a mechanism Hemang Subramanian, Miami, FL, USA cally to computing and far too much based on Bitcoin’s concept of block- related to these other fields, and also chain imposing distributed protocols, fail to address, in any significant new or what is called “smart contracts” in Scant Evidence for Spirits way, probably the greatest ethical this context. Arthur Gardner’s letter to the editor hole in computing today—security Subramanian did not, however, “A Leap from Artificial to Intelli- and hacking. mention the existence of a different, gence” (Jan. 2018) on Carissa Schoe- If the proposed revised Code is ever older, technique for implementing nick et al.’s article “Moving Beyond submitted to a vote by the member- decentralized applications called the Turing Test with the Allen AI Sci- ship, I will be voting against it and urge “law-governed interaction,” or LGI, entific Challenge” (Sept. 2017) asked other members to do so as well. introduced in 1991 (under a different us to accept certain beliefs about arti- name) by Minsky.1 It was implement- ficial intelligence. Was he writing Reference ed at Rutgers University some 10 1. https://dl.acm.org/citation.cfm?id=3173016 that all rational beings are necessari- years later and is still under develop- ly spiritual? “That which actually Alexander Simonelis, Montreal, Canada ment. LGI can be used to implement knows, cares, and chooses is the spir- a range of decentralized applications, it, something every human being including decentralized marketplac- has,” he said. And that all humans Authors Respond: es2 and (in 2015) decentralized social are rational? Why and how would ACM promotes ethical and social networks, the very applications that someone (anyone) be convinced of responsibility as key components of attracted Subramanian’s interest. such a hypothesis?

10 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 letters to the editor

Not every human, to quote Gardner, lution has developed into what we to- security analysts being able to per- “knows, cares, and chooses.” One day call consciousness and rationality. form proper system assessments. It might suspect that no human infant Perhaps these are just emergent prop- is, in fact, some of the same micro ser- does, but may, in fact, learn and devel- erties of a murmuration of neurons. vices Andriole explored that can lead op them over time. Richard J. Cichelli, Nazareth, PA, USA to security flaws that are then avail- What evidence for spirits? Would able for exploitation by aspiring hack- Gardner accept an argument that ers with a library of scripts that can be there are no spirits? If not, would this Still Looking for Direction in run against the containers and the not be a rejection of the scientific Software Development host operating system. method and evidence-based reason- I have been in IT for 30 years, working Though I have great regard for ing? Scientific hypotheses are based on every kind of platform and thus cloud projects and the technology on experimental design. Valid experi- feel qualified to address several that allows faster and more-flexible mental designs always allow for “falsi- points about systems development solutions to address business needs, fiability,” as argued by philosopher of raised by Stephen J. Andriole in his IT managers must make sure they do science Karl Popper (1902–1994). Viewpoint “The Death of Big Soft- not lose the major benefits of enter- Falsifiability (sometimes called test- ware” (Dec. 2017). For example, I see prise resource planning products. I ability) is the capacity for some propo- in many current “agile” cloud-based spent the 1990s moving from piece- sition, statement, theory, or hypothesis projects a fundamental lack of direc- meal systems to a system where a to be proven wrong. That capacity is an tion. For projects that fail to perform business user can track raw materials essential component of the scientific as promised, the lack of a more in- all the way to the end product and method and hypothesis testing. depth requirements process can lead bought and sold with just a few clicks. Through it, we say what we know be- to missing critical integrations with I would hate to see IT managers lose cause we test our beliefs using observa- other systems. I have personally seen that by going back to disparate pro- tion, not faith. at least a dozen projects spiral out of cesses lacking the transparent inte- Humans are not rational by defi- control and never reach a real live hu- gration I know is possible. nition. They can think and behave ra- man user. For example, in 2016, I Reference tionally or not. Rational beings apply, worked with a U.S. government agen- 1. Library of Congress. Article 1—Legislative explicitly or implicitly, the strategy of cy on a very large project it had prom- Department. U.S. Constitution; https://www.congress. theoretical and practical rationality ised to deliver by 2020 but that failed gov/content/conan/pdf/GPO-CONAN-2017-9-2.pdf to the thoughts they accept and the a system test in the cloud because it Dan Lewis, Washington, D.C., USA actions they perform. A person who could not meet its own integration is not rational has beliefs that do and scalability goals. Even as the de- not fully use the information he or velopment team managed to occa- Author Responds: she has. sionally pick off relatively minor user The death of big software is attributable “Man is a rational animal—so at requests, it ignored the user-story re- to failure, control, governance, cloud, and least I have been told. Throughout a quirements with deeper technical monolithic software, and I thank Lewis long life I have looked diligently for evi- complexities, as in how to integrate for addressing failure, cloud, and dence in favour of this statement, but with other systems. Lack of integra- monolithic software. Cloud “containers” so far I have not had the good fortune tion led to missed deadlines for deliv- represent a first step toward hostage to come across it,” said British philoso- ering the key integrations by system prevention. I agree that cloud security pher Bertrand Russell (1872–1970), test dates, as mandated by Article I, due diligence should always be tongue firmly planted in cheek. Section 2 of the United States Consti- aggressive. I also agree that integration is One might believe, without evi- tution.1 always important but that monolithic dence, that “The leap from artificial to As far as how an organization can architectures do not guarantee integration intelligence could indeed be infinite,” get its data back if it moves from one (at the expense of flexibility) and that as Gardner claimed. However, every cloud provider to another, the con- micro-services-based architectures can day newspapers in 22 countries are de- tainer “solution” might sound nice to integrate and provide functional flexibility, signed by my company’s AI-based soft- users but can actually be worse than with the right tools. ware for classified pagination and dis- having table dumps from legacy sys- Stephen J. Andriole, play ad dummying. What was once tems. The lack of documentation Villanova, PA, USA done by rational, thinking human de- around containers, both architectur- signers is now done by even more ex- ally and with respect to how contain- pert computer programs. And I started ers function within the workflow Communications welcomes your opinion. To submit a Letter to the Editor, please limit yourself to 500 words or on this journey in 1973 by writing process and how the system will actu- less, and send to [email protected]. chess algorithms. ally process data, makes designing for To replace humans, these programs portability exceptionally difficult or have no need to know what a human is impossible for IT managers to main- or to care. tain during changes throughout the Our “clever code” may just be our systems life cycle. Another challenge DNA that through long biological evo- of working with containers involves ©2018 ACM 0001-0782/18/3

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 11 The Communications Web site, http://cacm.acm.org, features more than a dozen bloggers in the BLOG@CACM community. In each issue of Communications, we’ll publish selected posts or excerpts.

DOI:10.1145/3178118 http://cacm.acm.org/blogs/blog-cacm The Costs and Pleasures of a Computer Science Teacher Mark Guzdial considers the enormous opportunity costs of computer science teachers, while Bertrand Meyer ponders the pleasures of arguing with graduate students.

Mark Guzdial make more than half of what doctors cost. That may just be perception—the The Real Costs of do. In Finland, the opportunity cost of average starting salary for a certified a Computer Science becoming a teacher is not as great as teacher in Georgia is $38,925 (http:// Teacher are in the U.S. www.teachingdegree.org/georgia/salary/), Opportunity Costs, and The real problem of getting and the average starting salary for a Those Are Enormous enough computer science teachers is new software developer in the U.S. http://bit.ly/2AvL2fz the opportunity cost. We are strug- (not comparing to exorbitant possible December 1, 2017 gling with this cost at both the K–12 starting salaries) is $55,000 (http:// Imagine that you are an undergradu- (primary and secondary school) level bit.ly/2CFuQJL). That’s a big differ- ate who excels at science and math- and in higher education. ence, but it’s not the 3x differences of ematics. You could go to medical I have been exchanging email re- teachers vs. doctors. school and become a doctor, or you cently with Michael Marder of UTeach at We have a similar problem at the could become a teacher. Which University of Texas at Austin (http:// higher education level. The National would you choose? bit.ly/2CKwScu). UTeach (https://uteach. Academies of Sciences, Engineering, If you are in the U.S., most students utexas.edu/) is an innovative and suc- and Medicine just released a report: would not see these as comparable cessful program that helps science, Assessing and Responding to the Growth choices. The average salary for a gen- technology, engineering, and mathe- of Computer Science Undergraduate En- eral practitioner doctor in 2010 was matics (STEM) undergraduates be- rollments (you can read it for free or $161,000, and the average salary for a come teachers. They do not get a lot of buy a copy at http://bit.ly/2CWttnt), teacher was $45,226. Why would you computer science (CS) students who which describes the rapidly rising en- choose to make a third as much in sal- want to become CS teachers; CS is rollments in CS (also described in the ary? Even if you care deeply about ed- among the majors that provide the CRA Generation CS report, discussed ucation and contributing to society, smallest number of future teachers. A in a previous blog at http://bit. the opportunity cost for yourself and 2011 U.K. report (http://bit.ly/2CLviXF) ly/2qiMahP) and the efforts to manage your family is enormous. Meanwhile found that CS graduates are less like- them. The problem is basically too in Finland, the general practitioner ly to become teachers than other many students for too few teachers, makes $68,000 and the teacher makes STEM graduates. and one reason for too few teachers is $37,455. Teachers in Finland are not CS majors may be just as interested that computing Ph.D.’s are going into paid as much as doctors (http://bit. in becoming teachers. Why don’t they? industry instead of academia. ly/2m3ZnaK), but Finnish teachers My guess is the perceived opportunity Quoting from the report:

12 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 blog@cacm

CS faculty hiring has become who have been hired away on extend- beginning comparing doctors, teachers, a significant challenge ed leaves into industry. Those are CS and entry-level software developers. To nationwide. The number of new faculty not on hand to help carry the do that comparison properly, you have to CIS (computer and information teaching load for “Generation CS.” take into account the fact that doctors science and support ervices) Faculty do not have to leave cam- have to stay in school a long time. Ph.D.’s has increased by 21% pus to work with industry. Argo AI, Doctors don’t start making that kind of from 2009 (1,567 Ph.D.’s) to for example, makes a point of fund- money until after four years of medical 2015 (1,903 Ph.D.s), while CIS ing university-based research, of school and ~three years of residency. bachelor’s degree production keeping faculty on campus teaching Both teaching and software development has increased by 74%. During the growing load of CS majors (http:// can get jobs right out of undergrad. So that time, the percentage of bit.ly/2CuHA2r). Keeping the re- you have to factor in seven years of lost new Ph.D.’s accepting jobs in search on-campus also helps to fund wages for the doctor. At that point, the industry has increased graduate students (who may be future salary for the teacher has risen a little, somewhat, from 45% to 57% CS Ph.D.’s). There is likely an oppor- while that for the software developer according to the Taulbee tunity cost for Argo AI; by bringing has gone up quite a bit. So while I agree survey. Today, academia does the faculty off campus to Argo full- completely with the issue of opportunity not necessarily look attractive time, they would like get more re- cost, I think that this example needs to new Ph.D.’s: the funding search output. There is an associated more details to be complete. situation is tight and uncertain; opportunity cost for the faculty; go- —Mark Lewis the funding expectation of a ing on leave and into industry would department may be perceived likely lead to greater pay. Thanks, Mark! Great point about the as unreasonably high; the class On the other hand, industry that in- relationship between years of school and sizes are large and not every stead hires away the existing faculty salary. I agree. new hire is prepared to teach pays a different opportunity cost. When —Mark Guzdial large classes and manage TAs the faculty goes on leave, universities effectively; and the balance have fewer faculty to prepare the next Bertrand Meyer between building a research generation of software engineers. The Small and program and meeting teaching biggest cost is on the non-CS major. Big Pleasures obligations becomes more Here at Georgia Tech and elsewhere, it http://bit.ly/2Cz77eQ challenging. For the majority of is the non-CS majors who are losing the December 19, 2017 new CS Ph.D.’s, the research most access to CS classes because of One of the small plea- environment in industry is too few teachers. We try hard to make sures of life is to win a technical ar- currently more attractive. sure that the CS majors get access to gument with a graduate student. You classes, but when the classes fill, it is feel good, as well you should. It is The opportunity cost here influ- the non-CS majors who lose out. only human to want to be right. Be- ences the individual graduate’s That is a real cost to industry. A resides, if you ended up being wrong choice. The report describes new CS cent report from Burning Glass (http:// all or most of the time, you should Ph.D. graduates looking at industry bit.ly/2EdZvL5) documents the large start questioning your sanity: Why vs. academia, seeing the challenges number of jobs that require CS skills, are they the students and you the of academia, and opting for indus- but not a CS major. When we have too supervisor, rather than the other try. This has been described as the few CS teachers, those non-CS majors way around? (One of the most hypo- “eating the seed corn” problem suffer the most. critical lies in the world is the cliché (http://bit.ly/2CtEo79). (Eric Roberts In the long run, which is more pro- “I make sure to hire people who are has an origin story for the phrase at ductive: Having CS faculty working full- smarter than I am.” Sure. So obvious- his website on the capacity crisis, at time in industry today, or having a ly uttered—unless the person you are http://stanford.io/2CXoQtx.) steady stream of well-prepared com- hiring is your successor—for the sole That is a huge problem, but a simi- puter science graduates and non-CS purpose of making you look whip- lar and less well-documented prob- majors with computer science skills smart. If it were sincere, why then lem is when existing CS faculty take for the future? would you stay on?) leaves to go to industry. I do not One of the big pleasures of life is to know of any measures of this, but it Comments lose an argument with a graduate stu- certainly happens a lot—existing CS Great article. The first part of this dent. Then you have learned something. faculty getting scooped up into in- highlights one of my major complaints dustry. Perhaps the best-known ex- about teacher’s unions in the U.S. They Mark Guzdial is a professor in the College of Computing at the Georgia Institute of Technology in Atlanta, GA, USA. ample was when Uber “gutted” push very hard to keep the pay for all Bertrand Meyer is professor of Software Engineering at CMU’s robotics lab (see the descrip- teachers, regardless of subject taught, ETH Zurich, the Swiss Federal Institute of Technology; research professor at Innopolis University (Kazan, Russia), tion at http://bit.ly/2qw2wVh). It hap- equal. In the case of CS, they are clearly and chief architect of Eiffel Software (based in Goleta, pens far more often at the individual hurting education by doing so. CA, USA.). level. I know several robotics, AI, ma- I also have a comment on the chine learning, and HCI researchers opportunity cost analysis at the © 2018 ACM 0001-0782/18/3 $15.00

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 13 SHAPE THE FUTURE OF COMPUTING. JOIN ACM TODAY.

ACM is the world’s largest computing society, offering benefits and resources that can advance your career and enrich your knowledge. We dare to be the best we can be, believing what we do is a force for good, and in joining together to shape the future of computing. SELECT ONE MEMBERSHIP OPTION ACM PROFESSIONAL MEMBERSHIP: ACM STUDENT MEMBERSHIP:

q Professional Membership: $99 USD q Student Membership: $19 USD q Professional Membership plus q Student Membership plus ACM Digital Library: $42 USD ACM Digital Library: $198 USD ($99 dues + $99 DL) q Student Membership plus Print CACM Magazine: $42 USD q ACM Digital Library: $99 USD q Student Membership with ACM Digital Library plus (must be an ACM member) Print CACM Magazine: $62 USD q Join ACM-W: ACM-W supports, celebrates, and advocates internationally for the full engagement of women in computing. Membership in ACM-W is open to all ACM members and is free of charge. Priority Code: CAPP Payment Information Payment must accompany application. If paying by check or money order, make payable to ACM, Inc., in U.S. dollars Name or equivalent in foreign currency.

ACM Member # q AMEX q VISA/MasterCard q Check/money order

Mailing Address Total Amount Due

Credit Card # City/State/Province Exp. Date ZIP/Postal Code/Country Signature Email

Return completed application to: Purposes of ACM ACM General Post Office ACM is dedicated to: P.O. Box 30777 1) Advancing the art, science, engineering, and New York, NY 10087-0777 application of information technology Prices include surface delivery charge. Expedited Air 2) Fostering the open interchange of information Service, which is a partial air freight delivery service, is to serve both professionals and the public available outside North America. Contact ACM for more 3) Promoting the highest professional and information. ethics standards Satisfaction Guaranteed!

BE CREATIVE. STAY CONNECTED. KEEP INVENTING.

1-800-342-6626 (US & Canada) Hours: 8:30AM - 4:30PM (US EST) [email protected] 1-212-626-0500 (Global) Fax: 212-944-1318 acm.org/join/CAPP news

Science | DOI:10.1145/3178122 Samuel Greengard In Pursuit of Virtual Life Scientists are simulating biological organisms N N and replicating evolution in the lab. How far can they expand the boundaries of virtual life?

T FIRST GLANCE, the creature known as Caenorhab- ditis elegans—commonly referred to as C. elegans, a type of roundworm— Aseems remarkably simple; it is comprised of only 959 cells and approximately 302 neurons. In contrast, the human body contains somewhere around 100 trillion cells and about 100 billion neurons in the brain. Yet decod- ing the genome for this worm and digitally reproducing it—something that could spur enormous advances in the understanding of life and how organisms work—is a challenge for the ages. “The project will take years to complete. It involves enormous time and resources,” says Stephen Larson, project coordinator for the Open- Simulation of electrical activity in a “virtual brain slice” formed from seven unitary digital Worm Foundation. Larson, a neuro- reconstructions of neocortical microcircuits. scientist who is CEO of data software firm MetaCell, is not the only person Ultimately, this research could lead Says Larson: “Understanding how or- focused on digitally reproducing life, to a far greater understanding of how ganisms function would unlock many of or replicating evolution inside a com- neurons fire and brains and entire the secrets of nature and change the way puter. Researchers from a variety of organisms function. This knowledge we view and interact with the world.” fields are now attempting to decode would likely lead to new therapies worms, fly brains, and evolutionary and drugs for treating sickness and Beyond Biology processes in order to create virtual disease, but it could also produce The idea of developing virtual organ- organisms and simulations of living new biofuels, as well as entirely new isms and simulating physical systems creatures. It is safe to say that the field computing frameworks. It also raises through computing is nothing new. In of executable biology—constructing questions about what constitutes life the late 1940s, John von Neumann be- computational models of biological and whether living things can be engi- gan exploring the concept of creating a

IMAGE: BLUE BRAIN PROJECT / EPFL ©2005 – 2018. ALL RIGHTS RESERVED. RESERVED. / EPFL ©2005 – 2018. ALL RIGHTS BRAIN PROJECT BLUE IMAGE: systems—is coming to life. neered inside a computer. computer virus modeled after a biologi-

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 15 news

cal virus; he eventually developed the and silos is a complex endeavor. Ulti- first self-replicating program. In 1970, mately, “We may need to get to a new mathematician John Horton Conway in- Mountains of data type of computing process to under- troduced a cellular automation system produce stand the exotic dynamics of natural called Conway’s Game of Life, in which a neural systems,” he says. person configures a set of circles, then incremental gains, the computer embarks on a rudimen- and coordinating Cracking the Code tary evolutionary process. The OpenWorm project is one of sev- By the 1990s, a number of research- all the research eral current attempts to unravel the ers had begun to explore the idea of pro- groups and silos is mysteries of living things. ducing digital representations of biolog- For example, Virtual Fly Brain—a ical creatures. a complex endeavor. joint effort involving the University of For scientists, the idea of creating Edinburgh, University of Cambridge, virtual life and artificial worlds inside MRC Laboratory of Molecular Biology, a computer is rooted in practicality: it Cambridge, and the European Bioin- makes possible the study of the genet- functions in the worm’s body, develop- formatics Institute—is mapping the ic information of an organism, or the ing software to run simulations, build- physiology of the household fly. creation of a virtual space to study how ing a digital model of C. elegans, and In 2012, Jonathan Karr at the Insti- evolution and adaption take place. “Re- constructing an algorithm that simu- tute for Genomics & Multiscale Biol- searchers can run thousands and thou- lates the worm’s muscle movements— ogy Institute at the Mt. Sinai School of sands of replicates simultaneously. Ev- including how electrical signals travel Medicine in New York City assembled ery computer is essentially a Petri dish,” through its brain and nervous system. the first whole-cell model of Mycoplas- explains Christoph Adami, professor So far, researchers at Caltech have ma genitalium, a pathogenic bacterium of microbiology and molecular genet- developed the OpenWorm Browser, that resides in humans. The model ics at Michigan State University. This which relies on a Web or iOS interface succeeded in predicting the viability of approach also allows researchers to to display a three-dimensional anatom- cells after genetic mutations. isolate specific components, including ical model and actions for C. elegans. In 2016, Stanford University bio- genetic coding, and “very carefully tease The browser displays different layers, engineering professor Markus W. apart the different elements that go into including the skin, alimentary system, Covert and a team of researchers de- the evolutionary process,” he says. nervous system, reproductive system, veloped a whole-cell computational OpenWorm is an example of how and body wall muscles. In addition, a model; they used detailed information this new frontier of biology and com- program called Sibernetic uses a C++ from more than 900 scientific journals puting is unfolding. So far, “several algorithm to model and simulate con- to gain insights into previously unob- hundred people” have contributed tractile matter and membranes within served cellular behaviors. to the project in some way. This in- the muscle tissue of the C. elegans. An- There’s also the work of Henry cludes computer scientists, math- other platform, Gepetto, provides an Markram, a professor of neuroscience ematicians, biologists, and experts in open source Web-based neuroscience at the École Polytechnique Fédérale de neuroscience. Among the core partic- simulation and visualization environ- Lausanne in Switzerland, director of ipants are more than a dozen academ- ment that simulates complex biologi- that institution’s Laboratory of Neural ic and research luminaries, including cal systems and their surrounding en- Microcircuitry, and founder and di- C. elegans biologists Sreekanth Cha- vironment using multiple algorithms. rector of the Swiss Blue Brain Project lasani at the Salk Institute, Michael Not surprisingly, the data process- national brain initiative. His research Francis at the University of Massachu- ing challenges related to OpenWorm has focused on synaptic plasticity and setts Medical School, William Schafer and developing a life-like digital mod- the microcircuitry of the neocortex. at University of Cambridge, and An- el are enormous; the overall task of In 2005, he launched the initiative in drew Leifer at Princeton University. In understanding things like synthesis, order to reconstruct and simulate the addition, the organization has received reproduction, and digestion will likely mammalian brain, starting with the ro- computing input from the likes of Net- take several more years. For now, re- dent neocortical column. Markram and ta Cohen at Leeds University and Chris- searchers rely on a combination of fellow researchers are now attempting tian Grove at the California Institute of classical mathematical and analytics to reverse-engineer the circuitry of the Technology (CalTech). tools along with machine learning to brain—something that could radically The OpenWorm project is now near- decode functions at the level of ion redefine health and medicine. ly seven years old. Larson estimates the channels and cells. “Cells have a lot Make no mistake, these projects ex- project is 80% of the way toward achiev- of extra machinery in them that is dif- tend far beyond a basic understanding ing its first goal: assembling a digital ficult to detect, and many of these ac- of physiological mechanisms. An organ- model of the worm that allows research- tivities and processes are completely ism’s behavior is affected by numerous ers to simulate movement through sim- ignored by artificial neural nets,” Lar- factors, ranging from its environment ulated viscous fluids. The team hopes to son says. Simply put: mountains of to its genetics. This means that even achieve this milestone by late this year. data produce incremental gains, and when scientists decode the genome of This has involved mapping cells and coordinating all the research groups a creature such as C. elegans, it remains

16 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 news incredibly challenging to understand are doing when you consider them as how cells function alone and together, biological cells. There are more exotic ACM and how they interact with the envi- dynamics taking place, but we cannot ronment to adapt, adjust, and evolve. see them,” he says. However, “It may Member “Achieving a complete understanding be possible to develop more advanced of a worm requires incredible resourc- types of computing systems based on es. Understanding the mechanisms in biology. It appears that there is more News more advanced lifeforms is still very far we can do with neural nets than we have UNDERSTANDING off into the future,” says Herbert Sauro, been doing with deep learning.” THE POWER COMPUTING associate professor of bioengineering at Of course, the ultimate questions for CAN PROVIDE the University of Washington. researchers and society are where will Allison Druin is the inaugural “It’s painstaking and arduous work all of this lead, and how exactly do we Associate to put all the pieces together,” says Alex- define life? At some point, digital code Provost for ander Hoffman, professor of immunol- could replicate biological code for en- Research & Strategic ogy and microbiology at the University tire creatures, and researchers in syn- Partnerships at of California, Los Angeles. “It’s neces- thetic biology might compile code to the Pratt Institute in Brooklyn, sary to pull together research from a very engineer new types of organisms—or NY. Her early education as a large pool of existing literature, code all autonomous devices, such as robots, graphic designer, combined with her technology the information in a set of equations that use biological models to think. background, provides her with and parameters, and then work with This may bring the world to the highly a skill set well suited for this computing software to relate model discussed state of “singularity,” where new role. simulations to all the data. The problem intelligence becomes increasingly non- Druin earned her undergraduate degree in for now is that there’s often not enough biological and humans transcend their Graphic Design at the Rhode existing knowledge to deliver an accu- biological origins. Island School of Design, her rate simulation of a phenotype—and Concludes Sauro: “In the future, we Master’s degree in Media Arts & Sciences from the so you wind up with gaps in knowledge could see very different definitions of Massachusetts Institute of that require further experimentation.” life. Once you start creating and evolv- Technology (MIT) Media ing life-like behaviors in computer Lab, and her Ph.D. from the Mind Games code or in synthetic biological systems University of New Mexico’s College of Education. During A primary goal of these projects, and ex- and then applying them to the physical her time at MIT, developing ecutable biology in general, is to pro- world, it’s possible to produce a very new technologies for children duce reliable computer models that different reality.” became Druin’s personal research focus. “When you ultimately can be used to understand want to make new technology the behavior of cancer cells and ad- for children, there is no Further Reading dress other debilitating or life-threat- straight line from here to there,” ening diseases, ranging from multi- Adami, C., Hintze, A., Edlund, J.A., Olson, R.S., Druin explains. During two decades as a ple sclerosis and amyotrophic lateral Knoester, D.B., Schosau, J., Albantakis, L., Tehrani-Saleh, A., Kvam, P., Sheneman, L., professor at the University of sclerosis to heart disease and arthri- Goldsby, H., and Bohm, C. Maryland (UM), Druin served in tis, says Sauro. “Understanding the Markov Brains: A Technical Introduction. numerous roles, including lab internal machinery of cells and how Sept. 17, 2017. https://arxiv.org/ director, associate dean, and pdf/1709.05601.pdf. chief futurist. She describes they successfully orchestrate cellular her experience as giving voice remodeling in a way that doesn’t harm Basak, S., Behar, M., and Hoffmann, A. to users in the innovation them could lead to faster and better Lessons from Mathematically Modeling process, and in developing an understanding of the impact of ways to develop therapies and drugs.” the NFkB Pathway. Immunological Reviews, 2012. 246, pp.221-38. PMID: 22435558 new innovations. A deeper understanding of cellular PMCID: PMC3343698. In 2015, Druin took leave activity also could help researchers from UM to work as Special Palyanov, A., Khayrulin, S., and Larson, S.D. engineer and reengineer organisms Advisor for National Digital Application of smoothed particle Strategy for the National to produce biofuels and other chemi- hydrodynamics to modeling mechanisms Park Service, where she led cal substances, or to produce entirely of biological tissue. Advances in strategic planning to bring new categories and types of antibiot- Engineering Software. 2016. 98, 1-11. digital experiences to park doi: 10.1016/j.advengsoft.2016.03.002. visitors, and to also enhance ics and other medicines. digital preservation. This field of research may also have Mısırlı, G., Hallinan, J., Pocock, M., Lord, P., In her new role, Druin will enormous implications for comput- McLaughlin, J.A., Sauro, H., and Wipat, A. lead the initiative to expand Data Integration and Mining for Synthetic ing, Larson says. Today, computational research within Pratt, as well Biology Design. ACS Synthetic Biology, April as with the Institute’s external neuroscience attempts to faithfully re- 25, 2016. Vol. 5, Issue 10, pp. 1086-1097. partners. “You don’t have to produce the activity of neurons. Howev- http://eprints.keele.ac.uk/3637/. be a computer scientist to er, neural nets do not exactly replicate understand the power that the actions and behaviors of biologi- Samuel Greengard is an author and journalist based in computing can bring to the West Linn, OR, USA. world,” she says. cal cells and neurons. “It’s not obvious — John Delaney what information processing neurons © 2018 ACM 0001-0782/18/3 $15.00

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 17 news

Technology | DOI:10.1145/3178312 Keith Kirkpatrick The Construction Industry in the 21st Century Three-dimensional printing and other new technologies are revitalizing the business of building buildings.

HE CONSTRUCTION OF New York’s Empire State Build- ing is often seen as the figu- rative and literal pinnacle of construction efficiency, Trising 1,250 feet and 102 stories from the ground to its rooftop spire in just over 13 months’ time, at a human cost of just five lives. Indeed, most of today’s construction projects would be lucky to come close to that level of speed, regardless of the building’s size. While the construction industry traditionally has been slow to change the way it operates, several new technologies are poised to usher in a new era of faster and more automated construction practices. Three-dimensional (3D) printing is among the key technologies that are expected to change the way structures are built in the future, as construction engineers and contractors seek meth- A concrete villa in Binzhou, China, that was produced by a 150-meter-tall 3D printer. ods for completing buildings more quickly, more efficiently, and, in many productivity, as [the construction cause it can print to an exact shape cases, with a greater attention paid to schedule] will be more predictable.” with micron-level accuracy, and the sustainability. Large printers that can The most common commercial wax molds can be melted down for re- print construction materials such as use of 3D printing today is to create use again and again. foam or concrete into specific shapes the molds that are used to cast con- The completed panels are shipped can drastically speed up the creation of crete panels for use in a building or and installed into the passenger tun- walls, decorative or ornamental pieces, tunnel, which is how the technology nels of London’s Crossrail railway and even certain structural elements. is being used by Australia’s Laing construction project. Crossrail will Furthermore, in some scenarios, cus- O’Rourke, a multinational construc- be a high-frequency, high-capacity tom-built or unique items can be cre- tion company. Laing O’Rourke’s railway serving London and the ated onsite or in a factory, at a much FreeFAB technology employs employs Southeast U.K. According to James lower cost than by using traditional, a 100x25x15-foot robotic 3D printer Gardiner, CTO and cofounder of one-off casting techniques. that prints molds measuring 6x4.5 FreeFAB, a spinout of Laing O’Rourke, “If you can focus on [printing] the feet from a specially designed wax. utilizing 3D printing to create the more labor-intensive components of These molds are used to cast large molds, rather than the panels them- the structure, then the productivity will concrete panels at an offsite factory selves, incorporates all of the bene- increase,” says Pelin Gultekin-Bicer, that are unique, in terms of size or fits (speed and quick customization) Building Information Modeling (BIM)/ shape, and likely would require a of 3D printing, while minimizing the Virtual Design and Construction (VDC) large traditional wood or polystyrene technology’s weaknesses. project manager at VIATechnik LLC, a mold to be created for each panel. Gardiner notes that 3D concrete construction and engineering services According to the company, the printers require very specific tempera- firm based in Chicago. In addition, she FreeFAB technology is more eco-friend- ture and humidity levels in order for says, “if you can integrate the 3D print- ly and efficient than creating conven- concrete to set properly. In addition, if

ers onsite, then you can improve the tional wood or polystyrene molds, be- a 3D printer pauses or stops, the homo- VCG/GETTY IMAGES BY PHOTO

18 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 news geneity of each layer of concrete, and Nantes Métropole, Nantes Métropo- unique panels or parts, or will be de- the bonds between each layer, will be le Habitat (NMH), and Ouest Valori- ployed onsite to print unique or be- compromised. This introduces ques- sation, with help from teams at the spoke architectural or decorative tions about whether the material and Nantes Digital Science Laboratory pieces of a structure, rather than en- building process will be able to be cer- and the Institute of Research in Civil tire houses, in the near future. By ad- tified by local building authorities, giv- and Mechanical Engineering, are dressing the labor-intensive pieces en that any material used in construc- working to create an industrial 3D via onsite printing, 3D printing likely tion must meet load-handling, printer that will be able to build a will improve the predictability of weatherproofing, and other specific demonstration house in only a few scheduling of material delivery and building code issues. days. Called BatiPrint3D, the printer improve worker productivity, which “[Research universities] are start- was completed in the fall of 2017. together are responsible for much of ing to focus on the characterization of The device prints the home in three the costs related to construction. 3D materials, trying to develop specif- layers, including two foam layers for “The biggest problem of the con- ic materials that are particularly well- insulation, and a third concrete layer. struction industry is the variance in suited for 3D printing and under- Of course, the technology is simply schedule and cost estimation,” says standing those already used,” being demonstrated, and is not yet Gultekin-Bicer, explaining that these in- Gardiner says. “But I’d say that it’s still ready for mass commercialization. clude material costs, shipping costs, got a little way to go.” Similar to the WinSun house, a real and labor costs. “Cost estimations are Other organizations, however, have functioning house would need to have derived from the schedule, so if you ex- taken to using 3D printing to construct its infrastructure and finishes added tend the schedule of the building, you entire buildings. Chinese company Win- after printing, thereby putting real con- are also extending the cost,” she says, Sun claims that in 2008, it printed an en- struction times in line with those of noting that the variability in the sched- tire house using 3D printing technology traditionally built houses. ule is largely due to the human labor in just two days. The company has since That is why it is most likely that 3D factor. “If you can integrate the 3D 3D-printed larger structures, such as a printing will be primarily used either printers onsite, then we can improve five-story section of a city block, as well offsite, to facilitate the faster creation the productivity, as [the schedule] will as one of its own manufacturing plants of molds to create one-off castings of be more predictable.” in Suzhou Industrial Park. Each of these projects took about a month, far less time than would be required with traditional construction techniques. Autonomous Vehicles According to industry professionals, however, WinSun’s 3D-printed build- Help Build the Future ing technology still requires that the individual walls, floors, and roof be fas- It is likely autonomous vehicles that can be operated within an enclosed setting, such as vehicles used on construction sites, will gain traction before self-driving cars do. tened together using traditional San Francisco-based Built Robotics is equipping construction vehicles with methods, including bolting walls and technology that allows them to do work that is dangerous, repetitive, or both, with roofs together, and much of the inter- little human intervention. Late last year, the company launched an autonomous track loader (ATL) that uses a combination of LIDAR sensors, inertial measurement units, nal elements of a house (ductwork, and global positioning system (GPS) technology to handle basic construction site tasks, electrical conduit, plumbing, and other such as digging foundation holes. finishes) must still be installed, rather Caterpillar, one of the world’s largest manufacturers of heavy equipment, has than printed. deployed more than 100 autonomous haul trucks to mines throughout the world. According to the company, Caterpillar worked with Blacksburg, VA-based Torc “They’re not printing an entire six- Robotics for a decade to develop its RemoteTask skid steer remote control system. story apartment building in one go,” Caterpillar also partnered with Torc to develop a system (due for release early explains Casey Mahon, digital practice next year) in which a Komatsu 930E haul truck can be operated by the Caterpillar autonomous haul truck system. manager at Carrier Johnson, an archi- “I think the biggest challenge to autonomy is going to be on the security side,” tectural, interior design, and branding says Bob Schena, chairman, CEO, and co-founder of Rajant Corp., a wireless mesh practice based in San Diego, CA. networking company that works with construction equipment manufacturers. “If “They’re printing it in pieces. Those governments and regulators believe that equipment could be hacked, taken over, and controlled by bad actors, they’re not going to allow autonomy. So, this is an pieces are coming out to the job site, issue that will have to be addressed, and we’re very involved in that.” and they’re getting assembled. Then “Right now, we are working with construction companies that are going to use they’re getting clad with some tile or our solutions in order to securely connect their equipment,” says Moshe Shlisel, CEO of Ramle, Israel-based GuardKnox. Shlisel says many construction companies stone on the exterior. It’s rare that you are using relatively weak connectivity software to actively monitor and control heavy ever see an interior photograph of one equipment, such as cranes 200 feet tall. of those projects. When you do, you “The manufacturer of the construction equipment would like to be able to can see that, it’s clear, all the conduit monitor and to predict some malfunctions [by keeping an active and constant wireless connection to the crane], Shlisel says. “But you have to do that securely, is surface-run, all the fixtures are sur- otherwise you’re having a connected motor or a connected crane that is exposed face-mounted fixtures.” to every hacker that, just for fun, says to himself, ‘well, let’s see if I can control this Meanwhile, researchers in France crane remotely’.” —K.K. from the University of Nantes,

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 19 news

space constraints, and the changing lay- out of the site as construction progress- A bricklaying robot es as factors that can make a construc- called SAM100 can tion site less than ideal for automation. “There’s a shortage of skilled workers lay 3,000 bricks that are going into the construction a day, six times trades, so what the SAM does is allow each worker to be more productive.” as many as a typical Podkaminer notes that while the ro- human bricklayer. bot is expensive, there has been a lot of interest from contractors and masons in renting the robot, which allows it to be used by contractors that normally wouldn’t have enough work to amor- In the future, Gardiner notes, 3D tize its approximately $500,000 pur- printing will allow for a more stream- chase price over a longer time horizon. Advertise with ACM! lined use of materials, thanks to the Still, despite the increasing use of ability to print both structural and automation and 3D printing on the job decorative pieces of a building that site, the construction industry is notori- Reach the innovators place material only where needed for ously slow when it comes to adopting and thought leaders maximum strength, energy efficiency, new technology. This is largely due to or form, or to help them fit into unusu- the highly regulated nature of construc- working at the al or tight site constraints. tion, as well as the high cost of adopting cutting edge “At the moment, if you build a con- new technology, as Gardiner notes that crete wall or a brick wall, you’re putting the 3D printer used to create the mas- of computing the same bulk material uniformly, re- sive molds can cost $1 million. and information gardless of where the stresses are on that “There are inexpensive 3D construc- technology through particular element,” Gardiner says, ob- tion [concrete] printers around,” Gar- serving that 3D printing will allow archi- diner says, “but these machines are ACM’s magazines, tects to design building components generally trading off accuracy and re- websites that put material only where they are ab- liability to achieve their low cost. So, solutely needed, rather than simply ad- one of the things that I see is that a and newsletters. hering to a mass-produced shape. good concrete printer, a good con- It is not just 3D printing that is struction 3D printer, will need to be poised to disrupt the construction in- highly reliable, fast, and accurate. And ◊◆◊◆◊ dustry. Created by New York-based the problem with that is that a ma- company Construction Robotics, a chine that has those things is generally bricklaying robot called SAM100 can more expensive.” Request a media kit lay 3,000 bricks per day, effectively multiplying a typical human bricklay- with specifications Further Reading er’s productivity of about 500 bricks and pricing: per day by six. The SAM100 (whose Gardiner, J. name stands for “semi-automated 2011, Exploring the Emerging Design Territory of Construction 3D printing – Ilia Rodriguez mason”) uses a conveyor belt, robotic Project Led Architectural Research, arm, and concrete pump to lay bricks. Architecture & Design, RMIT University. +1 212-626-0686 The robot’s software ensures the ro- https://researchbank.rmit.edu.au/view/ [email protected] bot can quickly choose between types rmit:160277/Gardiner.pdf of bricks, quickly lay bricks in compli- Building Codes, Underwriters Laboratories cated patterns, and strictly adhere to https://www.ul.com/code-authorities/ the building plan. However, the tech- building-code/ nology still requires a human operator Laing O’Rourke’s FreeFAB Technology to smooth the concrete before placing https://www.youtube.com/ watch?v=alAHME_A8VY additional layers of bricks. “Automation is popular in a con- SAM100 OS 2.0 https://www.youtube.com/ trolled environment, but construction watch?v=tK9oBQyR9KY sites are not very controlled environ-

ments,” says Zak Podkaminer, a market- Keith Kirkpatrick is principal of 4K Research & ing executive with Victor, NY-based Con- Consulting, LLC, based in Lynbrook, NY, USA. struction Robotics. He cites weather (such as rain, snow, and humidity), © 2018 ACM 0001-0782/18/3 $15.00

20 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 news

Society | DOI:10.1145/3178125 Esther Shein The State of Fakery How digital media could be authenticated, from computational, legal, and ethical points of view.

ACK IN 1999, Hany Farid was finishing his postdoctoral work at the Massachusetts Institute of Technology (MIT) and was in a library Bwhen he stumbled on a book called The Federal Rules of Evidence. The book caught his eye, and Farid opened to a random page, on which was a section entitled “Introducing Photos into a Court of Law as Evidence.” Since he was interested in photography, Farid wondered what those rules were. While Farid was not surprised to learn that a 35mm negative is consid- A real dog (left), and an image of a dog created by a deep convolutional generative ered admissible as evidence, he was adversarial network (GAN) algorithm. surprised when he read that then-new digital media would be treated the Farid. Videos can go viral in a matter sponsibility … and do more to prevent same way. “To put a 35mm file and a of minutes; coupled with the pace of this dangerous practice.” digital file on equal footing with re- technological advance and the ability People need to be skeptical of what spect to reliability of evidence seemed to easily deceive someone online, how they see and hear, says David Schub- problematic to me,’’ says Farid, now a is it possible to trust what we’re see- mehl, research director, Cognitive/AI computer science professor at Dart- ing coming out in the world? Systems and Content Analytics at mar- mouth College and a leading expert on It is a troubling question, and no ket research firm IDC. digital forensics. “Anyone could see one, it seems, is immune from being “People have to get used to the idea where the trends were going.” targeted. Virgin Group founder Sir that images, voice recordings, video, That led Farid on a two-decades- Richard Branson revealed he has and any other forms of media that can long journey to consider about how been the target of scams using his im- be represented as digital data can be digital media could be authenticated age to impersonate him. In a blog manipulated and changed in one or from the computational, legal, and eth- post in January 2017, Branson noted more ways through the use of software,’’ ical points of view. He and others have that “the platforms where the fake Schubmehl says. their work cut out for them; there are stories are spreading need to take re- Because machine learning is becom- dozens of ways a digital image can be ing so prevalent, experts say we need manipulated, from doing something ways to improve its ability to spot decep- as simple as cropping or lightening it, Images, voice tion. For example, Schubmeil says, “To- to something more nefarious. day, researchers are experimenting with As fake news dominates headlines recordings, video, generative adversarial networks (GANs) and the use of artificial intelligence (AI) and other forms that can be used to combine two differ- to alter images, video, or photographs ent types of images or video together to is rampant, media outlets, political of media can be create a merged third type of video.” campaigns, ecommerce sites, and even changed through The idea behind GANs is to have two legal proceedings are being called into neural networks, one that acts as a “dis- question for the work they generate. the use of software, criminator” and the other a “genera- This has led to various efforts in govern- so we need better tor,” which compete against each other ment, academia, and technological to build the best algorithm for solving a realms to help identify such fakery. ways to spot problem. The generator network uses “More and more, we’re living in a deception. feedback it receives from the discrimi- digital world where that underlying nator to learn to produce convincing digital media can be manipulated and images that can’t be distinguished altered, and the ability to authenti- from real images; this helps it get better

IMAGE COURTESY OF INTEL AI BLOG BY URS KÖSTER BY OF INTEL AI BLOG COURTESY IMAGE cate is incredibly important,” says at detecting something false.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 21 news

Of course, image altering has its observes Farid, and there are a lot of (legal) benefits. It has allowed the gray areas. He advocates that publish- movie industry to produce spectacu- France enacted ing and media outlets, as well as courts lar action movies, since the vast ma- legislation requiring of law and scientific journals, should jority of that action is computer-gen- adhere to a policy of “you don’t have to erated content, Farid points out. On images showing tag the images, you simply have to the consumer side, image altering models whose show me the original.” This, he says, lets people create aesthetically pleas- “keeps people honest.” That approach ing photos, so everyone looks good in appearances have “allows we, the consumers, to make the the same photo. been altered to be determination, and bypasses the com- Schubmehl agrees. “Hollywood has plexity of defining what’s appropriate been creating fake worlds for people for clearly, prominently and what isn’t,” Farid says. decades and now regular people can do labeled to that effect. Right now, we are unable to know it as well,’’ he says. “There’s a tremen- for sure when a photo has been altered dous use for tools like Photoshop to cre- when sophisticated manipulations are ate exactly the right type of image that being used, says Nightingale. She adds, someone wants for whatever purposes.” however, that there are signs of image Whether manipulated images manipulation that can be identified. should be identified as such is the sub- “Perhaps not too surprisingly, adver- “Computer scientists working in ject of much debate. While it would tisers and publishers continue to resist digital forensics and image analysis seem to be a no-brainer when it comes such legislation and criticize the limita- have developed a suite of programs that to their use in legal cases and photo- tions it places on free expression and detect inconsistencies in the image, journalism, there are mixed opinions artistic freedom,’’ says Nightingale, who perhaps in the lighting,’’ she says. Her about the use of manipulated images completed image manipulation studies work includes conducting research to in industries such as fashion, enter- as part of her Ph.D. at the University of see whether people can make use of tainment, and advertising. France re- Warwick in Coventry, U.K. She adds that these types of inconsistencies to help cently passed a law stipulating that any “many photographers believe that the identify forgeries. image showing a model whose appear- use of image manipulation techniques Other more general tips Nightingale ance has been altered must feature a is a positive thing that allows creative suggests for spotting fake photos in- clear and prominent disclaimer label freedom; in fact, some suggest the abil- clude using reverse image searches to to indicate this is the case, notes So- ity to manipulate images makes a pho- find the image source, looking for re- phie Nightingale, a postdoctoral teach- tographer more akin to a painter who peating patterns in the image (since ing associate at Royal Holloway Univer- takes something that is real and puts repetition might be a sign that some- sity of London. Those who do not their own artistic spin on it.” thing has been cloned) and checking comply with the law are subject to a One tricky thing is that “manipu- the metadata, which provides details fine of 30% of the advertising cost. lated” is not an easy word to define, such as the date and time the photo

Milestones BBVA Recognizes Turing Laureates

The BBVA Foundation, and secure transmission of the secure protocols that Berkeley, when they embarked a Spanish organization electronic data, ranging from defined the face of modern on a collaboration whose that promotes research, email to financial transactions. cryptography,” as the jury first big result would lay the advanced training, and the In addition, their work provides terms it. RSA is a “public-key” theoretical foundations of transmission of knowledge the underpinning for digital encryption system because the field—the mathematical to society, has given its 10th signatures, blockchains, and each user has two keys: a demonstration of when Frontiers of Knowledge cryptocurrencies.” public key, used to encrypt an encryption method is Award in the Information The work of Goldwasser, the message; and another genuinely unbreakable. and Communication Micali, Rivest, and Shamir, known only to the receiver. The BBVA jury said Technologies category to the citation adds, “is crucial The encryption process is Goldwasser and Micali, ACM A.M. Turing Award to the fabric of our connected based on a mathematical together and separately, recipients Shafi Goldwasser, digital society. Every time problem intractable for today’s “have expanded the scope Silvio Micali, Ronald Rivest, we log in to social media, computers without the aid of of cryptography beyond its and Adi Shamir for their purchase goods online, or the other, private key. RSA is traditional goal of secure “fundamental contributions vote or sign electronically, still a widely used protocol, communication,” with to modern cryptology, an area we leverage the technology particularly in combination developments that have helped of a tremendous impact on our developed by their research.” with other techniques. build today’s flourishing digital everyday life,” in the words of In 1978, Shamir and Goldwasser and Micali society by allowing users to the jury’s citation. Rivest, together with Leonard in 1982 were working on collaborate, share information, “Their advanced crypto- Adleman, created the RSA a doctorate course at the and shop online without protocols enable the safe algorithm, the “first of University of California, sacrificing security.

22 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 news

read by a moderator. To protect the Real privacy of the conversation, the moder- Samples ator can only read the suspected fraud- ster’s messages, explains David Minns, founder and CEO of software developer DM Cubed, which devel- Latent Space oped the SaucyDates tool. He adds the AI tool also warns the potential D Is D victim of fraudulent content. Correct? Discriminator Farid says we should absolutely be alarmed by the growth of software that G enables digital media to be manipulated Generator Generated into fakes for nefarious purposes. Fake Samples “There’s no question that from the field of computer vision to computation- Fine Tune Training al photography to computer graphics to Noise software that is commercially available, we can continue to be able to manipu- Architecture of a generative adversarial network. late digital content in ways that were un- imageable a few years ago,’’ he says. was taken, camera settings used, and or has been manipulated,’’ he says. “And that trend is not going away.” location coordinates. “The stakes can be very, very high, and While digital forensic techniques that’s something we have to worry a Further Reading are a promising way to check the au- great deal about.” Physical Unclonable Functions and thenticity of photos, for now “using That is because a growing number Applications: A Tutorial. Proceedings of these techniques requires an expert of AI tools are increasing the ability for the IEEE, Volume: 102, Issue: 8, Aug. 2014 and can be time-consuming,’’ Nightin- fakery to flourish, regardless of how Pappu, S.R. gale says. “What’s more, they don’t they are being used. In 2016, Adobe an- Physical One-Way Functions. Pappu, 100% guarantee that a photo is real or nounced VoCo (voice conversion), es- Ravikanth. 2001. Massachusetts Institute of fake. That said, digital forensic tech- sentially a “Photoshop of speech” tool Technology, http://cba.mit.edu/docs/theses/01.03. niques and our work, which is trying to that lets a user edit recorded speech to pappuphd.powf.pdf improve people’s ability to spot fake replicate and alter voices. images, does at least make it more dif- Face2Face is an AI-powered tool that Thies, J., Zollhofer, M., Samminger, M., Theobalt, C., and Niessner, M. ficult for forgers to fool people.” can do real-time video reenactment. Face2Face: Real-time Face Capture and Farid has developed several tech- The technology lets a user “animate the Reenactment of RGB Videos, 2016 IEEE niques for determining whether an facial expressions of the target video by Conference on Computer Vision and Pattern image has been manipulated. One a source actor and re-render the ma- Recognition (CVPR) http://www.graphics.stanford. method looks at whether a JPEG im- nipulated output video in a photoreal- edu/~niessner/papers/2016/1facetoface/ age has been compressed more than istic fashion,” according to its creators thies2016face.pdf once. Another technique detects im- at the University of Erlangen-Nurem- Denton, E., Gross, S., and Fergus, R. age cloning, which is done when try- berg, the Max Planck Institute for In- Semi-Supervised Learning with Context- ing to remove something from an im- formatics, and Stanford University. Conditional Generative Adversarial age, he says. In addition, Schubmehl When someone moves their mouth Networks. 2016 https://arxiv.org/ cites the development of machine and makes facial expressions, those abs/1611.06430 learning algorithms by researchers at movements and expressions will be Sencar, H. T., and Memon, N. New York University to spot counter- tracked and then translated onto Digital Image Forensics: There is More to a feit items. someone else’s face, making it appear Picture than Meets the Eye. ISBN-13: 978-1461407560 The mission of a five-year U.S. De- that the target person is making those https://www.amazon.com/Digital-Image- fense Advanced Research Projects exact movements. Forensics-There-Picture/dp/1461407567/ Agency (DARPA) program called Medi- On the flip side is software that helps ref=reg_hu-rd_add_1_dp For (media forensics) is to use digital users take preventative measures The National Academy of Science. forensics techniques to build an auto- against being duped. One is an AI tool Strengthening Forensic Science in mated system that can accurately an- called Scarlett that was recently intro- the United State: A Path Forward. 2009. alyze hundreds of thousands of im- duced by adult dating site Saucy- The National Academies Press. https://www.ncjrs.gov/pdffiles1/nij/ ages a day, says Farid, who is Dates, with the goal of reducing fraud grants/228091.pdf participating in the program. “We’re and scams in the dating industry.

now in the early days of figuring out Scarlett acts as a virtual assistant and Esther Shein is a freelance technology and business how to scale [the system] so we can do as people are having live conversa- writer based in the Boston area. things quickly and accurately to stop tions, it scores users; when the score

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 23 viewpoints

DOI:10.1145/3180485 Ross Anderson VPrivacy and Security Making Security Sustainable Can there be an Internet of durable goods?

S WE START to connect du- German government banned the Cay- many regulated industries; it is not rable goods such as cars, la doll whose voice-recognition sys- just the obvious ones like aircraft and medical devices, and elec- tem could be abused in the same way.3 railway signals, but even kids’ toys— tricity meters to the Inter- Yet privacy may change less than we they must not have lead paint, and if net, there will be at least think. Your car knows your location you pull a teddy bear’s arms off, they Athree big changes. First, security will history, sure, but your phone knows must not leave sharp spikes. be more about safety than privacy. that already. It also knows where you So what is the strategic approach? Certification will no longer mean test- walk, and it is already full of adware. We looked at three verticals—road ing a car once before selling it for 10 Denial of service has also been in vehicles, medical devices, and smart years; safety will mean monthly soft- the news. In October 2016, the Mirai meters. Cars are a good example to il- ware updates, and security will be botnet used 200,000 CCTV cameras lustrate what we learned—though the an integral part of it. Second, we will (many of them in Brazil and Vietnam) lessons apply elsewhere too. have to reorganize government func- to knock out Twitter in the Eastern U.S. tions such as safety regulators, stan- for several hours. ISPs know they may Security and Safety for Cars dards bodies, testing labs, and law en- have to deal with large floods of traffic Car safety has been regulated for forcement. Finally, while you might from senders with whom they cannot years. In the U.S., the National High- get security upgrades for your phone negotiate, and are starting to get wor- ways Transportation and Safety Ad- for two or three years, cars will need ried about the cost. ministration was established in the safety and security patches for 20 But the most important issue 1960s following Ralph Nader’s cam- years or more. We have no idea how to in the future is likely to be safety. paigning; Europe has an even more patch 20-year-old software; so we will Phones and laptops do not kill a lot complex regulatory ecosystem. Reg- need fresh thinking about compilers, of people, at least directly; cars and ulators discovered by the 1970s that verification, testing, and much else. medical devices do. simply doing crash tests and pub- In 2016, Éireann Leverett, Richard lishing safety data were not enough Privacy, Availability, or Safety? Clayton, and I conducted a research to change industry behavior. They The early security scares about the project for the European Commission had to set standards for type approv- “Internet of Things” have mostly been on what happens when devices that al, mandate recalls when needed, about privacy. There have been re- are subject to safety regulation start and coordinate car safety with road ports of the CIA and GCHQ turning to contain both computers and com- design and driver training. Insurers smart TVs into room bugs, while the munications.5 There are surprisingly do some of the regulatory work, as

24 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 V viewpoints

do industry bodies; but governments publicized public demonstration that has started over-the-air upgrades, provide the foundation. a Jeep Cherokee could actually be run and other vendors are following suit. This ecosystem faces a big change off the road—by Charlie Miller and This will move safety regulation from of tempo. At present, a carmaker Chris Valasek, in 2015—showed that pre-market testing to a safety case builds a few hundred prototypes, the public will not tolerate the risk.4 maintained in real time—a chal- sends some for crash testing, has the While previous academic papers on lenge for both makers and regula- software inspected, and gets certifi- car hacking had been greeted with a tors. We will need better, faster, and cation under more than 100 regula- shrug, press photos of the Cherokee more transparent reporting of safety tions. Then it ramps up production in a ditch forced Chrysler to recall 1.4 and security incidents. We will need and sells millions of cars. Occasion- million vehicles. responsible disclosure—so people ally carmakers have to change things Cars, like phones and laptops, will who report problems are not afraid quickly. When a Swedish journal- get monthly software upgrades. Tesla of lawsuits. We will need to shake ist found that avoiding an elk could up existing regulators, test labs, and cause an A-class car model to roll standards bodies. Over two dozen over, Mercedes had to redesign its Over two dozen European agencies have a role in car suspension and fit a stability control safety, and none of them have any system, which delayed the product European agencies cybersecurity expertise yet. We will launch at a cost of $200 million.2 But have a role in car need to figure out where the security most of the safety case is a large up- engineers are going to sit. front capital cost, while the time con- safety, and none We may need to revisit the argu- stant for the design, approval, and of them have any ment between intelligence agencies testing cycle is five years or so. who want “exceptional access” to sys- In the future, a vulnerability in a cybersecurity tems for surveillance, and security ex- car will not need a skillful automotive expertise yet. perts who warn that this is hazardous.1 journalist to exploit it. Malware can do The Director of the FBI and the U.K. that. So if a car can be crashed by com- Home Secretary both argue that they mands issued remotely over the Inter- should be able to defeat encryption; net, it will have to be fixed. Although they want a golden master key to your we have known for years that car soft- phone so they can listen in. But how

IMAGE BY DJ TAYLOR BY IMAGE ware could be hacked, the first widely many would agree to a golden master

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 25 viewpoints

key that would let government agents will be legal; there have been regula- crash their car? tions in Europe since 2010 that force There are opportunities too. Your Perhaps carmakers to provide technical in- monthly upgrade to your car software the biggest formation to independent garages will not just fix the latest format string and spare-parts manufacturers. It vulnerability, but safety flaws as well. challenge will is tempting to hope that a free/open The move to self-driving cars will lead be durability. source approach might do some of to rapid innovation with real safety the heavy lifting, but many critical consequences. At present, product re- components are proprietary, and calls cost billions, and manufacturers need specialist test equipment for fight hard to avoid them; in the future, software development. We also need software patches will provide a much incentives for minimalism rather cheaper recall mechanism, so we can than toolchain bloat. We do not really remove the causes of many accidents leaving options such as keeping the know how to allocate long-term own- with software, just as we now fix dan- original development environment ership costs between the different gerous road junctions physically. of computers and test rigs, but not stakeholders so as to get the socially But cars will still be more difficult connecting it to the Internet. Could optimal outcome, and we can expect to upgrade than phones. A modern we develop on virtual platforms that some serious policy arguments. But car has dozens of processors, in every- would support multiple versions? whoever pays for it, dangerous bugs thing from engine control and navi- That can be more difficult than it have to be fixed. gation through the entertainment initially appears. Toolchain upgrades Once software becomes pervasive system to the seats, side mirrors, and already break perfectly functional soft- in devices that surround us, that are tire-pressure sensors. The manufac- ware. A bugbear of security developers online, and that can kill us, the soft- turer will have to coordinate and drive is that new compilers may realize that ware industry will have to come of the process of updating subsystems the instructions you inserted to make age. As security becomes ever more and liaising with all the different sup- cryptographic algorithms execute in about safety rather than just privacy, pliers. Its “lab car”—the rig that lets constant time, or to zeroise crypto- we will have sharper policy debates test engineers make sure everything graphic keys, do not affect the output. about surveillance, competition, works together—is already complex So they optimize them away, leaving and consumer protection. The no- and expensive, and the process is your code suddenly open to side-chan- tion that software engineers are not about to get more complex still. nel attacks. (In separate work, Laurent responsible for things that go wrong Simon, David Chisnall, and I have will be put to rest for good, and we will Sustainable Safety and Security worked on compiler annotations that have to work out how to develop and Perhaps the biggest challenge will be enable a security developer’s intent to maintain code that will go on work- durability. At present most vendors be made explicit.) ing dependably for decades in envi- won’t even patch a three-year-old Carmakers currently think their li- ronments that change and evolve. phone. Yet the average age of a U.K. car ability for upgrades ends five years af- at scrappage is 14.8 years, and rising ter the last car is sold. But their legal References 1. Abelson, H. et al. Keys under doormats: Mandating all the time; cars used to last 100,000 obligation to provide spare parts lasts insecurity by requiring government access to all data and communications. Commun. ACM 58, 10 (Oct. miles in the 1980s but now keep go- for 10 years in Europe; and most of 2015). ing for nearer 200,000. As the embed- the cars in Africa arrive in the country 2. Andrews, E.L. Mercedes-Benz tries to put a persistent Moore problem to rest. New York Times (Dec. 11, ded carbon cost of a car is about equal secondhand, and are repaired for as 1997). to that of the fuel it will burn over its long as possible to keep them oper- 3. German parents told to destroy Cayla dolls over hacking fears. BBC News (Feb. 17, 2017). lifetime, a significant reduction in ve- able. Once security patches become 4. Greenberg, A. Hackers remotely kill a jeep on the hicle durability will be unacceptable necessary for safety, who is going to highway—with me in it. Wired (July 21, 2015). 5. Leverett, É., Clayton, R., and Anderson, R. on environmental grounds. be writing the patches for today’s cars Standardisation and certification of the Internet As we build more complex arti- in Africa in 25 years’ time? of Things. In Proceedings of WEIS 2017; http:// weis2017.econinfosec.org/program/. facts, which last longer and are more This brings us to the business

safety critical, the long-term main- side—to the question of who will Ross Anderson ([email protected]) tenance cost may become the limit- pay for it all. Markets will provide is Professor of Security Engineering at Cambridge University, U.K. He is a Fellow of the Royal Society and ing factor. Two things follow. First, part of the answer; insurance pre- the Royal Academy of Engineering, and author of Security software sustainability will be a big miums are now rising because low- Engineering—A Guide to Building Dependable Distributed research challenge for computer sci- speed impacts now damage cam- Systems. entists. Second, it will also be a major eras, lidars, and ultrasonic sensors, business opportunity for firms who so that a damaged side mirror can can cut the cost. cost $1,000 rather than $100. The On the technical side, at present firms that earn money from these it is hard to patch even five-year-old components have an incentive to software. The toolchain usually will help maintain the software that not compile on a modern platform, uses them. And part of the answer Copyright held by author.

26 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 viewpoints

VDOI:10.1145/3180488 Pamela Samuelson Legally Speaking Will the Supreme Court Nix Reviews of Bad Patents? Considering the longer-term implications of a soon-to-be-decided U.S. Supreme Court case.

“ AD” PATENTS HAVE been a plague to many in the software industry. Patents can be “bad” for numerous reasons. Although patents are Bsupposed to be available only for new and inventive advances, sometimes it is difficult for examiners to locate the most relevant prior art. Not knowing about this art may cause them to approve patents that should not have been issued. Sometimes claims are too abstract or vague to be eligible for patents, or are deficient in other ways. To address the bad patent problem, most developed countries have created administrative procedures so that third parties can challenge the validity of patents by asking a patent office tribunal to re-examine the patents. Post- grant review procedures are cost-effective ways to get rid of bad patents without having to go through full- dress, multiyear, very expensive litiga- The statue The Contemplation of Justice, created by the sculptor James Earle Fraser, tion and appellate review. Patents that flanks the U.S. Supreme Court building in Washington, D.C. survive post-grant reviews are “stronger” for having gone through this ex- PTAB rulings can ask the Court of Ap- lack of novelty grounds. Oil States ap- tra scrutiny. peals for the Federal Circuit (CAFC) to pealed this decision to the CAFC, in In 2011, the U.S. Congress enacted review them. part by attacking the constitutionality the America Invents Act (AIA), which, Greene’s Energy Group was among of the PTAB review process. among other things, gave the Patent the firms that asked PTAB to review a The CAFC thought so little of Oil Trial & Appeal Board (PTAB) the pow- patent being asserted against it. Oil States’ appeal that it did not even issue er to review issued patents and extin- States Energy Services had sued it in an opinion to explain its denial. But Oil guish claims that the board deems federal court for infringement of this States was not ready to give up. It asked deficient for lack of novelty or inven- patent. PTAB agreed with Greene’s that the U.S. Supreme Court to review the

IMAGE BY KEVIN HARBER/FLICKR (CC BY-NC-ND 2.0) KEVIN HARBER/FLICKR (CC BY-NC-ND BY IMAGE tiveness. Those dissatisfied with the Oil States patent was invalid on constitutional question.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 27 viewpoints

To everyone’s astonishment, the gress, including the power to pass laws Court decided in June 2017 to hear Oil that grant exclusive rights for limited States’ appeal. All of a sudden, the very times to inventors for their discoveries ACM Transactions useful PTAB tool that Congress created in the useful arts (that is, patents). to enable challenges to “bad” patents Article II establishes the Executive on Accessible was at risk of being struck down on Branch of the U.S. government and hoary old constitutional grounds. sets forth powers of the President and Computing those who work within that branch of What’s at Stake? the government. Oil States is obviously upset at the in- Article III establishes and gives cer- ACM TACCESS is a validation of its patent on a device and tain powers to federal courts. Section 1 quarterly journal that method to cap well-heads during hy- states: “The judicial Power of the United draulic fracturing (aka fracking) proce- States, shall be vested in one supreme publishes refereed dures so the well-heads can withstand Court, and in such inferior Courts as considerable pressure caused when the Congress may from time to time or- articles addressing firms pump fluids into oil and gas wells dain and establish.” Section 2 provides issues of computing to stimulate or increase the production. in relevant part that “[t]he judicial Pow- The Oil States lawsuit against er shall extend to all Cases, in Law and as it impacts the Greene’s for infringing this patent was Equity, arising under this Constitu- stayed during the PTAB review. Unless tion, the Laws of the United States, and lives of people with the Supreme Court invalidates the Treaties made, or which shall be made, disabilities. The whole PTAB review procedure on con- under their Authority.” stitutional grounds, Oil States will lose Also at issue in Oil States is the Sev- journal will be of this lawsuit. enth Amendment to the Constitution, The stakes are, of course, much larg- one of the 10 amendments to the Con- particular interest to er than the loss that Oil States would stitution (widely known as the Bill of SIGACCESS members sustain if the Supreme Court rules Rights), which were promulgated dur- against it. In its petition asking the ing the process of state ratification of and delegates to its Court to hear its appeal, Oil States em- the Constitution to address several civ- phasized that PTAB had overturned il liberties concerns raised in debates affiliated conference nearly 80% of the patent claims it had about the original document. (i.e., ASSETS), as well reviewed. (This affected almost 10,000 The Seventh Amendment provides: patents as of March 2016.) That may “In Suits at common law, where the val- as other international sound like a lot, but consider the PTAB ue in controversy shall exceed twenty only reviews patents when it has decid- dollars, the right of trial by jury shall be accessibility ed the challenges are likely to succeed.) preserved, and no fact tried by a jury, conferences. The Oil States brief cited one source es- shall be otherwise re-examined in any timating that PTAB reviews had “de- Court of the United States, than accord- stroyed” $546 billion in value to the ing to the rules of the common law.” U.S. economy and “wiped out” $1 trillion in valuation of companies whose Oil States’ Constitutional Attack patents had been invalidated. Oil States’ argument, boiled down to If PTAB was right that these patents its essence, is this: Congress created should not have issued, one might think it is good that companies sued for infringement did not have to pay Virtually everyone $546 billion to license invalid patents. The $1 trillion in lower valuation for who pays attention firms whose patents were struck down to patent law was may also be well deserved. That, however, is not directly relevant to the con- surprised by the stitutional issue the Court will be grap- Court’s decision to pling with in the Oil States case. For further information review the CAFC’s or to submit your Relevant Constitutional Provisions decision in Oil States. Explaining the legal issue before the manuscript, Supreme Court in Oil States requires a brief review of key parts of the U.S. visit taccess.acm.org Constitution. Article I confers numerous legislative powers on the U.S. Con-

28 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 viewpoints the PTAB as an administrative tribunal office could not cancel a patent be- within of the U.S. Patent and Trade- cause Congress had not given the office mark Office (PTO). Congress gave If the only way power to do so. By enacting the AIA, PTAB the power to adjudicate certain a bad patent can get however, Congress conferred such challenges to the validity of issued pat- power on the PTO. The sentences from ent claims. Under its theory, Oil States killed is through McCormick on which Oil States relied contends that only Article III courts a $5–$10 million were only “dicta.” have the constitutional authority to adjudicate patent validity. It further federal court lawsuit, Conclusion claims that under the Seventh Amend- lots of bad patents Constitutional challenges to Congres- ment, it has the right to a jury trial on sional enactments are rarely success- its patent claims. will go unchallenged. ful. The Supreme Court generally gives Oil States relies on an 1898 Su- a broad reading to the constitutional preme Court opinion, McCormick powers that Congress has to accom- Harvesting Machine Co. v. C. Aultman plish its goals, including the estab- & Co., which says, among other lishment of administrative tribunals. things, that a patent “is not subject to Even so, there is reason for PTAB-pro- being revoked or cancelled by the SA v. Nordberg, which opined that ponents to be worried about the Oil President, or any other officer of the Congress has the power to decline to States case. The Supreme Court’s juris- government” because “it has become provide for jury trials to resolve cer- prudence on the powers of Article III the property of the patentee, and as tain types of disputes between pri- courts versus the powers of Congress such is entitled to the same legal pro- vate parties over statutory rights as to establish tribunals to resolve private tection as other property.” Because long as the rights at issue are integral disputes is very arcane, convoluted, courts had, prior to adoption of the to a public regulatory scheme whose and far from consistent. Constitution, the power to review pat- operations Congress has assigned to The Court’s job is to interpret the ent validity and infringement, the an administrative agency. Constitution. So the Justices cannot Constitution should be understood to Greene’s also cited precedents hold- just decide that Congress had a good have vested Article III courts, and only ing that patents are “public rights” and reason to establish PTAB and give it Article III courts, with the power to integrally related to the complex regu- power to extinguish erroneously decide whether patents are valid. latory scheme that Congress assigned granted patents. The Justices will have Oil States is not the first firm to have to the PTO. Because PTAB’s reviews are to carve a careful path through the raised this particular constitutional part and parcel of that scheme, thicket of their past decisions to artic- challenge. Three times previously the Greene’s argued that Congress had ulate standards to uphold PTAB’s pat- Supreme Court denied petitions to re- power to assign these reviews to the ex- ent reviews, and give guidance about view the constitutionality of the PTAB pert agency in charge of this scheme, Congress’ powers to establish other power to extinguish patent claims. namely, the PTO. administrative tribunals, such as a This explains why virtually everyone The PTO also filed a brief in re- small claims tribunal within the Copy- who pays attention to patent law was sponse to Oil States’ petition, point- right Office. surprised by Court’s decision to review ing out that Congress intended for the One can hope the Justices will con- the CAFC’s decision in Oil States. AIA to “establish a more efficient and sider the constitutional issue in Oil The Oil States case has many owners streamlined patent system that will States in light of another constitution- of weak patents hoping the Court will improve patent quality and limit un- al purpose, that which underlies the find merit in Oil States’ arguments. necessary and counterproductive liti- patent system: promoting the prog- Many of others are worried that the gation” in response to a “growing ress of useful arts. If the only way a bad Court will get tangled up in abstract ar- sense that questionable patents are patent can get killed is through a $5 guments about whether patents are too easily obtained and are too diffi- million–$10 million federal court law- “private” or “public” rights under the cult to challenge.” The PTO brief ar- suit, lots of bad patents will go unchal- Court’s complicated jurisprudence gued that the Court should not thwart lenged. PTAB is an efficient mecha- about powers of Article III courts and this laudable goal. nism to extinguish erroneously issued powers that Congress has to create ad- It noted that Congress had given patents. It will be unfortunate indeed ministrative tribunals to handle cer- several agencies besides the PTO the if the Court feels so caught in the web tain kinds of claims. power to review and correct errors in of its constitutional precedents that it their past decisions. PTO examiners, cannot find a way to uphold the good Responses to Oil States’ Claims like many other federal agents, some- work the PTAB has been doing. Greene’s had hoped to ward off Su- times make mistakes. The PTAB review preme Court review by relying on process enables correction of those Pamela Samuelson ([email protected]) is the Richard M. Sherman Distinguished Professor of Law and the Court’s prior lack of interest in mistakes. Information at the University of California, Berkeley, USA, hearing this kind of challenge. On The PTO challenged Oil States’ read- and a member of the ACM Council. the merits, it relied upon a 1989 Su- ing of the McCormick decision. The preme Court opinion, Gianfinaciera holding in that case was that the patent Copyright held by author.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 29 viewpoints

VDOI:10.1145/3180490 Simon Rogerson Computing Ethics Ethics Omission Increases Gases Emission A look in the rearview mirror at Volkswagon software engineering.

HE VOLKSWAGEN EMISSIONS scandal came to light in Sep- tember 2015. The company installed software into millions of vehicles with diesel Tengines so that impressive emission readings would be recorded in laboratory conditions even though the reality is that the diesel engines do not comply with current emission regulations. Volkswagen is a global organization headquartered in Germany; its subsid- iaries adhere to common policies and a corporate culture. This worldwide scandal broke first in the U.S. with ongoing investigation and legal action there and in other countries including Germany, Italy, and the U.K. Combustion engines are the source of pollution and therefore have been subjected to emission control. The formation of NOx (nitrogen oxides) through combustion is a significant A wall of rear-view mirrors at the entrance to Seat Pavilion in Volkswagen Group’s Autostadt contributor to ground-level ozone and (“Car City”) visitor attraction adjacent to the Volkswagen factory in Wolfsburg, Germany. fine particle pollution that is a health risk. On this basis, the use of software the design of a new diesel engine to use of the defeat device software to control emissions must be defined meet stricter U.S. emission standards would enable VW diesel vehicles to as safety critical for, if it fails or mal- to take effect in 2007. The goal was to pass U.S. emissions tests.” functions, it can cause death or serious market new vehicles as meeting the Drawing upon the Statement of injury to people. There does not appear stricter standards and attract U.S. buy- Facts, Leggett3 reported that although to be any acknowledgement of this ers. Being unable to accomplish this, there had been some concerns over across vehicle manufacturing. the engineers working under Hadler the propriety of the defeat software all The statement from the U.S. De- and Dorenkamp, developed software those involved in the discussions in- partment of Justice16 details the facts that allowed vehicles to distinguish cluding engineers were instructed not of the VW emissions case. Two senior test mode from drive mode thus satis- to get caught and furthermore to de- managers, Jens Hadler and Richard fying the emissions test while allow- stroy related documents. According to Dorenkamp appear to be at the center ing much greater emissions when ve- Mansouri,4 Volkswagen is an autocratic of the so-called defeat software’s on- hicles were on the road. “Hadler company with a reputation for avoid- going design and implementation authorized Dorenkamp to proceed ing dissent and discussion. It has a

processes. These began in 2006, with with the project knowing that only the compliant business culture where em- GEOGIF/SHUTTERSTOCK.COM BY IMAGE

30 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 viewpoints viewpoints

ployees are aware that underperfor- has been translated into Arabic, Croa- mance can result in replacement and tian, French, German, Hebrew, Italian, so management demands must be Software engineers Mandarin, Japanese, and Spanish (see met to ensure job security. Three state- at Volkswagon faced http://bit.ly/2nlOYro). ments in particular in the Volkswagen Software engineers and software Group Code of Conduct: Promotion of In- ethical and legal engineering educators have a respon- terests, Secrecy, and Responsibility for issues that are easy sibility to be cognizant of the code and Compliance,17 align with the ongoing its requirements. Public Interest, conduct encouraged during the emis- to identify. which is apposite for safety-critical V 15 sions debacle. Trope and Ressler ex- software, is central to the code. Al- plain that as an autocratic book of though education can influence the rules, the group code supports and courage and capability to act in accor- even promotes dishonest dysfunc- dance with the code, that result de- tional behavior that includes the cre- pends on structural and psychological ation of software to cheat, rather than tleblowing as the reason it did not hap- supports within the environment in solve, engineering problems and to pen. Rhodes9 adds a second factor, ar- which engineers practice. protect that software from disclosure guing that corporate business ethics is The actions of VW managers and as if it were a trade secret. very much a pro-business stance that is software engineers violated the follow- In January 2017, the U.S. Justice De- implemented through corporate con- ing principles of the code: partment announced that, “Volkswa- trol and compliance systems, and in- Principle 1.03 “approve software gen had agreed to plead guilty to three struments of managerial coordination. only if they have a well-founded belief criminal felony counts, and pay a $2.8 This can enable the pursuit of business that it is safe, meets specifications, billion criminal penalty, as a result of self-interest through organized wide- passes appropriate tests, and does not the company’s long-running scheme spread conspiracies involving lying, diminish quality of life, diminish pri- to sell approximately 590,000 diesel cheating, fraud, and lawlessness. This vacy, or harm the environment. The ul- vehicles in the U.S. by using a defeat is what happened at Volkswagen. timate effect of the work should be to device to cheat on emissions tests man- Queen7 concurs, explaining that Volk- the public good.” The defeat software dated by the Environmental Protection swagen intentionally deceived those to is clearly unsafe given NOx pollution Agency (EPA) and the California Air Re- whom it owed a duty of honesty. The damages both health and the environ- sources Board (CARB), and lying and ob- pressure for continuous growth and the ment. The public were under the mis- structing justice to further the scheme.” perception that failure was not an op- apprehension that VW cars were emit- tion8 created a culture where corporate ting low levels of NOx and therefore not Business Analysis secrecy was paramount—which in turn a health risk. Thus, software engineers Many of the accounts about the Volk- implicitly outlawed whistleblowing. installed unethical software. swagen emissions case focus on busi- Principle 1.04 “disclose to appro- ness ethics with only a few touch- The Role of Software Engineering priate persons or authorities any ac- ing upon the role of the software If one has a responsibility for the plan- tual or potential danger to the user, engineers in this situation. These ning, design, programming, or imple- the public, or the environment, that accounts at times are repetitive but mentation of software then that aspect they reasonably believe to be associ- intertwine to provide a rich view. The of one’s work falls within the scope of the ated with software or related docu- widespread unethical actions across Software Engineering Code of Ethics and ments.” There is no evidence that any Volkswagen can be described as a new Professional Practice regardless of one’s software engineer disclosed. Com- type of irresponsible behavior, namely job title. In that sense software engineer- mercial software is usually developed deceptive manipulation.12 The detail of ing pervades this debacle and is there- in teams and in this case it is likely this and the associated corporate re- fore worthy of further investigation. this was a large team spanning all as- percussions are discussed further by So what was the role of software engi- pect of software development. Stanwick and Stanwick.14 neers in the creation and installation of Principle 1.06 “be fair and avoid de- Software engineers at Volkswagen VW’s defeat software? This question ception in all statements, particularly faced ethical and legal issues that are can be addressed using the Software En- public ones, concerning software or re- easy to identify. Plant6 suggests that gineering Code of Ethics and Professional lated documents, methods and tools.” they should have alerted external bod- Practice. The code is long established, The emissions software was heralded ies since the internal lines of reporting documenting the ethical and profes- publically as a success when internally were compromised. Merkel5 concurs, sional obligations of software engineers there was widespread knowledge that citing the Software Engineering Code of and identifying the standards society this claim was fraudulent. Software en- Ethics and Professional Practice (see expects of them.2 The code translates gineers were likely to have been privy to http://www.acm.org/about/se-code) by ethical principles into practical guid- this cover-up. way of justification, and adds that the ance. It encourages positive action and Principle 2.07 “identify, docu- lack of whistleblowers in such a large resistance to act unethically. It has ment, and report significant issues of group is surprising. Both authors point been adopted by many professional social concern, of which they are to the potential personal cost of whis- bodies and companies worldwide and aware, in software or related docu-

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 31 viewpoints

ments, to the employer or the client.” on rigor and justification.11 They must There is some evidence that concern possess practical skills to address the was raised about the efficacy of the Unethical actions complex ethical and societal issues defeat software but it seems those in related to that surround evolving and emerging dissent allowed themselves to be technology. Such education should managed towards deception. software engineering be based on a varied diet of participa- Principle 3.03 “identify, define and can be addressed tive experiential learning delivered address ethical, economic, cultural, le- by those who have a practical under- gal and environmental issues related from two sides. standing of the design, development, to work projects.” The EPA regulations and delivery of software. Contrasting are explicit and are legally binding. the Volkswagen Group Code of Con- From the evidence accessed it is un- duct with the Software Engineering clear as to whether software engineers Code might provide one means for knew of the illegality of their actions. experiential learning. Such educated Nevertheless ignorance cannot and paramount. Practice comprises proc- software engineers might find ways to must not be a form of defence. ess and product. Process concerns vir- prevent the installation of unethical Principle 6.06 “obey all laws govern- tuous conduct of software engineers, software of the future. ing [the] work, unless, in exceptional whereas product concerns whether circumstances, such compliance is in- software is deemed to be ethically vi- References 1. Catalogue of Catastrophe, International Project consistent with the public interest.” able. Actions and outcomes in the Leadership Academy; http://bit.ly/2FvCtkl. 2. Gotterbarn, D., Miller, K. and Rogerson, S. Software This relates to the analysis under prin- Volkswagen case appear to have failed engineering code of ethics is approved. Commun. ACM ciple 3.03. Compliance to further the on both counts. 42, 10 (Oct. 1999), 102–107 and Computer (Oct. 1999), 84–89; http://bit.ly/2nlOYro. prosperity of Volkswagen was at the ex- These serious issues related to pro- 3. Leggett, T. VW papers shed light on emissions scandal. pense of legal compliance. fessional practice must be addressed. BBC News (Jan. 12, 2017); http://bbc.in/2AWMtzW. 4. Mansouri, N. A case study of Volkswagen unethical Principle 6.07 “be accurate in stat- It is hoped such issues are exceptional practice in diesel emission test. International Journal ing the characteristics of software on but sadly it is likely they are common- of Science and Engineering Applications 5, 4 (2016), 211–216. which they work, avoiding not only place given the ongoing plethora of 5. Merkel, R. Where were the whistleblowers in the false claims but also claims that might software disasters (see, for example, Volkswagen emissions scandal? The Conversation 30, 1 (Sept. 30, 2015); http://bit.ly/2AUeGac. reasonably be supposed to be specula- Catalogue of Catastrophe and Soft- 6. Plant, R. A software engineer reflects on the VW tive, vacuous, deceptive, misleading, or ware Fail Watch13). Unethical actions scandal. The Wall Street Journal (Oct. 15, 2015); http://on.wsj.com/2FsKruB. doubtful.” Software engineers could related to software engineering can be 7. Queen, E.L. How could VW be so dumb? Blame argue internally that the software in- addressed from two sides. One side fo- the unethical culture endemic in business. The Conversation (Sept. 26, 2015); http://bit.ly/1LZV0jO. deed performed as it was designed to. cuses on resisting the temptation to 8. Ragatz, J.A. What can we learn from the Volkswagen However, the design was to achieve perform unethical practice while the scandal? Faculty Publications, 2015, Paper 297; http://bit.ly/2CW8XTr. regulatory and public deception. other focuses on reducing the oppor- 9. Rhodes, C. Democratic business ethics: Volkswagen’s tunity of performing unethical prac- emissions scandal and the disruption of corporate Principle 6.13 “report significant vi- sovereignty. Organization Studies 37, 10 (2016), olations of this Code to appropriate tice. Society at large needs competent, 1501–1518. 10. Rogerson, S. Ethics and ICT. In R. Galliers and W. authorities when it is clear that consul- ethical, and altruistic professionals to Currie, Eds. The Oxford Handbook on Management tation with people involved in these deliver societally acceptable, fit-for- Information Systems: Critical Perspectives and New Directions. Oxford University Press, 2011, 601–622. significant violations is impossible, purpose software. Both of these can be 11. Rogerson, S. Future Vision, Special Issue—20 years of counterproductive or dangerous.” Giv- helped by education, but education ETHICOMP. Journal of Information, Communication and Ethics in Society 13, 3/4 (2015), 346–360. en the apparent corporate culture will not suffice without adequate so- 12. Siano, A. et al. More than words: Expanding the taxonomy within Volkswagen there was little cial supports. of greenwashing after the Volkswagen scandal. Journal of Business Research 71(C), (2017), 27–37. point in reporting concerns further up In order to fulfill software engi- 13. Software Fail Watch 2016, Quarter One, Tricentis; the line. In fact the corporate code neering duties, an individual must http://bit.ly/2r2nnzY. 14. Stanwick, P. and Stanwick, S. Volkswagen emissions seems at odds with the professional fully understand the professional re- scandal: The perils of installing illegal software. code regarding this point. Software en- sponsibilities and obligations of the International Review of Management and Business Research 6, 1 (2017), 18. gineers failed to report these breaches role. These are explicitly laid out in 15. Trope, R.L. and Ressler, E.K. Mettle fatigue: VW’s to appropriate authorities. the Software Engineering Code of Ethics single-point-of-failure ethics. IEEE Security & Privacy 14, 1 (2016), 12–30. and Professional Practice and as such 16. U.S. Department of Justice. Volkswagen AG agrees Conclusion individuals must know and apply it to to plead guilty and pay $4.3 billion in criminal and civil penalties and six Volkswagen executives and Professionals, who must have been their everyday work. To achieve this, employees are indicted in connection with conspiracy to cheat U.S. emissions tests. Justice News, (Jan. 11, party to this illegal and unethical act, the effective education of new profes- 2017); http://bit.ly/2j7mT5M. developed and implemented this soft- sionals is essential. Teaching technol- 17. Volkswagen. The Volkswagen Group Code of Conduct, ware. Those who undertake the plan- ogy in isolation is unacceptable and 2010; http://bit.ly/21IYPF8. ning, development, and operation of dangerous. Software engineers need Simon Rogerson ([email protected]) is Professor Emeritus software have obligations to ensure a broader education to gain the nec- of Computer Ethics in the Centre for Computing and Social integrity of output and overall to con- essary skills and knowledge to act in Responsibility at De Montfort University, The Gateway, Leicester, U.K. tribute to the public good.10 The ethi- a socially responsible manner not on cal practice of software engineers is the basis of instinct and anecdote but Copyright held by author.

32 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 viewpoints

VDOI:10.1145/3182108 Peter J. Denning The Profession of IT The Computing Profession Taking stock of progress toward a computing profession since this column started in 2001.

E STARTED THIS column in 2001 when ACM was re-envisioning itself as a society of a computing profession. ACM Wleaders and many members already thought of computing as a profession. They wanted ACM to strengthen its support of computing professionals and its commitment to the practitioners of a computing profession. How has all this progressed in the past 17 years? In my first column on the IT profession, my opening question was whether a profession is needed in the first place. I wrote: “To most of the hundreds of millions of computer users around the world, the inner workings of a computer are an utter mystery. Opening the box holds as much attraction as lifting the joint ventures including curriculum come close to covering all the orga- hood of a car. Users look to computing recommendations and accreditation. nized specialties in computing. There professionals to help them with their They have diverged on certification are two other categories—computing- needs for designing, locating, retriev- and licensing, which traditionally have intensive fields in science and engi- ing, using, configuring, programming, been eschewed by ACM leadership and neering where computing is a tool but maintaining and understanding com- embraced by IEEE-CS leadership. is not the focus of concern, and computers, networks, applications, and dig- The next question was what special- puting-infrastructure occupations, ital objects.”1 The need has intensified ties have professionals organized to where specialists operate and main- over the years because there are now deal with specific kinds of concerns— tain the infrastructures on which ev- billions of users and the technologies for example, specialists in program- eryone depends. Table 1 is an update they rely on are much more complex. ming languages, operating systems, of the original table,1,2 now showing The ACM and IEEE Computer Soci- networks, or graphics. The ACM SIGs 52 specialties. ety are the two main professional soci- and IEEE-CS hosted organizations to Table 1 reflects the interests of the eties in computing. They are compara- support these groups. ACM had (and members of ACM and IEEE-CS. How- ble in size with approximately 100,000 still has) around 40 SIGs in special- ever, this is not the only way to catego- members each. ACM has traditionally ized areas. The list of SIGs is a useful rize computing professionals. The U.S. emphasized the science-math side of guide to the organized core specialties Bureau of Labor statistics maintains computing, and IEEE-CS the engineer- of computing. a list of “computer and information ing-design side of computing. The two However, the list of ACM and IEEE- technology occupations” that spells

IMAGE BY LIGHTSPRING LIGHTSPRING BY IMAGE societies have cooperated on many CS specialty organizations does not out the kinds of jobs employers recruit

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 33 viewpoints

for. Table 2 summarizes the BLS infor- Table 1. Selected professional specialties of computing. mation about computing.a Table 1 can be seen as an enumeration of the spe- Computing-Core Computing-Intensive Computing-Infrastructure cialties covered by the groups listed in Disciplines Fields Occupations Table 2. No matter how you look at it, Artificial intelligence Aerospace engineering Blockchain administrator there is a huge market for computing Cloud computing Autonomous systems Computer technician professionals. Computer science Bioinformatics Data analyst Between them, the ACM, IEEE-CS, Computer engineering Cognitive science Data engineer and a large network of education insti- Computational science Cryptography Database administrator tutions provide an extensive support Database engineering Computational science Help desk technician structure for computing professionals, Computer graphics Data science Identity theft recovery agent including these elements:2,3 Cyber security Digital library science Network technician ˲˲ Curricula that grant entry into Human-computer interaction E-commerce Professional IT trainer profession Network engineering Genetic engineering Reputation manager ˲˲ Standards for curricula (body of Programming languages Information science Security specialist knowledge) Programming methods Information systems System administrator ˲˲ Standards for professional practice Operating systems Public Policy and Privacy Web identity designer ˲˲ Professional development (short Performance engineering Instructional design Web programmer courses, books) Robotics Knowledge engineering Web services designer ˲˲ Accreditation guidelines and eval- Scientific computing Management information uation systems ˲˲ Certification Software architecture Network science ˲˲ Licensing Software engineering Multimedia design ˲˲ Code of ethics Telecommunications ˲˲ Professional specialty groups The professional societies do a lot of work to support computing professionals! Table 2. BLS occupations.

ACM and the Profession Category Entry Degree Median Salary in 2016 The ACM’s founders in 1947 believed Computer and Information Research Scientists MS $112K that computers would be a permanent Software Developers BS $102K source of attention and concern, even- Computer Network Architects BS $101K tually permeating all fields. ACM’s Information Security Analysts BS $93K initial responses to this concern were Computer Systems Analysts BS $87K a computing research journal (Jour- Database Administrators BS $85K nal of the ACM), a newsletter (Com- Computer Programmers (coders) BS $80K munications of the ACM), and a net- Network and Computer System Administrators BS $80K work of local chapters. Beginning in Web Developers AS $66K the early 1960s, ACM developed and Computer Support Specialists BS or AS $52K maintained a code of ethics. The first Source: bls.gov computer science departments were founded in 1962 and ACM issued its first curriculum recommendations in Since the 1960s, ACM has hosted more sionals who provide advice and ideas to 1968. Around 1985, ACM began part- than 40 special interest groups (SIGs) the Board. The Practitioner Board part- nering with IEEE-CS on accreditation in various specialties of computing. ners with relevant ACM publications, and upgrades to curriculum recom- In 2002, ACM established a Profes- for example in Queue and in Ubiquity’s mendations. Over the years, ACM/ sion Board, later renamed the Practi- advisory panel of young professionals. IEEE-CS have evolved successively tioner Board, to establish and maintain more sophisticated curriculum recom- programs for the professional develop- Challenges mendations into a “computing body of ment of the 70% of members who are In 2001, I wrote that the biggest chal- knowledge.” ACM has supported pro- practicing, nonacademic profession- lenge for ACM in fostering a profession fessionals in developing competencies als. The Practitioner Board today of- would be academic computer scientists through its professional development fers an increasing range of programs to giving up the illusion they could either center and ongoing discussions about support professionals, including pro- control the profession or be seen as good professional practice in Commu- fessional development, books, market- its leaders. Computing was spreading nications and ACM Queue magazines. ing, distinguished speakers, practitio- prolifically and professionals in other ner-oriented conferences, and GPAC fields were independently organizing a https://www.bls.gov/ooh/computer-and-infor- (global practitioner advisory council), professional groups to support them. A mation-technology/home.htm a global network of about 100 profes- conspicuous example was the compu-

34 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 viewpoints tational sciences, such as computation- ing authors who understand both In my opinion, however, ACM will al physics, computational chemistry, worlds and can translate the key ideas not achieve its goal of supporting a or computational biology, where proof research into useful ideas for prac- computing profession and its practi- fessional groups were being organized tice. This is quite difficult because few tioners without a concerted effort to independent of ACM or IEEE. At the such authors exist. A fine example is the bring practitioners into ACM leader- time, computer scientists were not very The Morning Paper, a five-times weekly ship positions and give them more open to reaching out to other fields and blog by Adrian Colyer in the U.K. (http:// professional recognitions. IEEE-CS helping them meet their own needs in blog.acolyer.org); in each issue, Colyer has been more successful with this computing. Over the years, ACM has translates a research paper into terms than ACM. embraced this challenge and has be- and connections that practitioners can come much better at reaching out to use. Colyer has a relationship with ACM Self-Management other fields. ACM has settled into a role through the Practitioners Board. Individual members further their pro- as a “curator of the flame,” providing Another aspect of this challenge is fessional development by using the the definitions, bodies of knowledge, in leadership: ACM has lost its ability to services and support structures of ACM and standards of practice for comput- populate its leadership positions with and IEEE-CS, and by improving their ing wherever it appears. a mixture of academic-research and own personal practices in their pro- One of the consequences of the practitioner professionals so that these fessional relations with clients. I have spread of computing into everyone’s two worlds will get to know each other aimed the 64 The Profession of IT col- lives is that the public mood has been and work together for a stronger pro- umns published since 2001 to support shifting to the notion that essential fession. All members of the ACM Coun- the latter. These columns have exam- knowledge to support education and cil, and most of the SIG leadership, are ined various aspects of the profession advancement of the profession should from the academic-research sector. including the nature of the profession, be free to the public. The ACM Digital ACM has been particularly good at sup- education for professionals, innova- Library, which has been feeling the pres- porting its professional academic and tion, language-action, the Internet, sure of this mood for several years, now research members with first-rate, wide- software, moods, jobs, and time man- supports open access to research papers ly respected publications, conferences, agement. You can find them on my for which the authors have paid an up- and awards. ACM has been less atten- website.b Please contact me with your front open access fee. The library itself is tive to helping practitioners develop questions or issues that I can address available to more practicing profession- their professional skills, at articulating in future columns. als because ACM grants access through standards for essential professional Computing has come a long way in licenses to organizations. These chang- skills, or at developing awards and oth- the 70 years since its founders’ first in- es have reduced revenues for digital er recognitions for industry profession- klings that computing would become library and other publication subscrip- als. The ACM Nominating Committee a pervasive professional concern. ACM tions. While it does not have a final an- has its work cut out for it. has developed an impressive array of swer, ACM has been making good head- ACM could do much more in recog- offers in publications, a digital library, way toward finding revenues to support nizing practitioner members for their conferences, chapters, support for its knowledge base in a way that it can contributions. Most ACM awards to- professional education, support for ultimately be free to the public. day go to members of the academic-re- practitioners, and awards. Its biggest But ACM’s biggest challenge con- search sector. The Distinguished Service challenge is integrating its academic- cerns the relations between two major Award, initially chartered in the 1960s as research and practitioner sectors. I ex- sectors of its members. The “academ- an industry professional award co-equal pect significant progress on this chal- ic-research” sector is members who in prestige to the Turing Award, has lenge in the next decade. are on the faculty of universities and faded into semi-obscurity and is now the colleges or are employed by industry only ACM award with no purse; it could b http://denninginstitute.com/pjd/PUBS/CACMcols research labs. The “practitioner” sec- be rejuvenated as a major recognition tor is members who are practicing non- for senior practitioner members. ACM References 1. Denning, P.J. Who are we? Commun. ACM 44, 2 (Feb. academic professionals, either self could also set up new awards explicitly 2001), 15–19. employed or employed by a company. for the practitioner sector. 2. Denning, P.J. and Frailey, D.J. Who are we—now? Commun. ACM 54, 6 (June 2011), 25–27. ACM sometimes uses the term “indus- Although ACM has major programs 3. Ford, G. and Gibbs, N. A Mature Profession of try professional” for practitioner. for practitioners—including profes- Software Engineering. Software Engineering Institute, Carnegie-Mellon University (1996); http://www.sei. One aspect of this challenge is bridg- sional development, learning center, cmu.edu/library/abstracts/reports/96tr004.cfm ing the gap between ACM’s treasure and Queue magazine—practitioner trove of research papers (in the digital members frequently tell us that ACM Peter J. Denning ([email protected]) is Distinguished Professor of Computer Science and Director of the library) and the working worlds of prac- does not understand them. The Prac- Cebrowski Institute for information innovation at titioners. Most research papers are com- titioner Board, under the leadership the Naval Postgraduate School in Monterey, CA, USA, is Editor of ACM Ubiquity, and is a past president of ACM. munications among researchers that of Terry Coata and Stephen Ibaraki, The author’s views expressed here are not necessarily enable the advancement of a research has begun to turn this around, with those of his employer or the U.S. federal government. field. Many practitioners, however, find 50 practitioner volunteers helping the these papers difficult at best and opaque board and another 100 providing ad- at worst. Bridging the gap means find- vice through the GPAC network. Copyright held by author.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 35 viewpoints

VDOI:10.1145/3180493 Fred B. Schneider Viewpoint Impediments with Policy Interventions to Foster Cybersecurity A call for discussion of governmental investment and intervention in support of cybersecurity.

HE LIST OF cyberattacks having significant impacts is long and getting longer, well known, and regularly invoked in calls for ac- Ttion. Such calls are not misplaced, because society is becoming more dependent on computing, making cyberattacks more capable of widespread harm. Vardi’s recent call1 “it is time to get government involved, via laws and regulations” motivates this Viewpoint. Indeed, we do know how to build more-secure systems than we are deploying today. And governments can—through regulation or other mechanisms—incentivize actions that individuals and organizations are otherwise unlikely to pursue. However, a considerable distance must be traversed from declaring that government interventions are needed to deciding particulars for those interventions, much less intervening. To the trade-offs involved in the adop- ment. Somebody must pay. It could be start, we need to agree on specific goals tion of each. In so doing, I hope to fa- consumers (through higher prices), to be achieved. Such an agreement re- cilitate discussions that will lead to government (through tax credits or quires understanding monetary and agreements about goals and costs. It is grants), or investors (if developers will other costs that we as a society are will- premature to advocate for specific in- accept reduced profits). But realize that ing to incur, as well as understanding terventions, exactly because those dis- the consumers, taxpayers, and inves- the level of threat to be thwarted. Only cussions have yet to take place. tors are just us. So before mandating after such an agreement is reached, expenditures for enhanced cybersecu- does it make sense for policymakers to Secure Systems rity, we must decide that we are will- contemplate implementation details. Are More Expensive ing to pay and decide how much we are This Viewpoint reviews interven- Assurance that a system will do what it willing to pay. tions often suggested for incentiviz- should and will not do what it should Other priorities will compete. Some

ing enhanced cybersecurity. I discuss not requires effort during develop- will advocate using “return on invest- ASSOCIATES ANDRIJ BORYS BY IMAGE

36 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 viewpoints viewpoints

ment” (ROI) to set spending levels for to proceed. But user authentication re- cybersecurity versus other priorities. quires (tedious) user interactions with But ROI is problematic as a basis for The nature the system; program authentication justifying how much to spend here. of networked limits which software can be run on a ˲˲ There are no good ways to quantify system; and the role of context can lim- how secure a system is. Measuring cy- infrastructures it a user’s flexibility in how tasks might bersecurity can be as difficult as estab- makes it difficult be accomplished. lishing assurance for a system in the to characterize ˲˲ Another common approach to de- V first place, which we know to be a hard fense is isolation. Here, effects of ac- problem for real systems. who benefits from tions by users, programs, or machines ˲˲ There are no good ways to quantify are somehow contained. Isolation the costs of not investing in cybersecu- cybersecurity. might be employed to keep attackers rity. To tally lost business or the work out or to keep attackers in. In either to recover data and systems ignores case, communications is blocked, other, important harms from attacks. which makes orchestrating coopera- Disclosure of confidential information difficult. We might, for example, tion, for example, can destroy reputa- facilitate secure access to a bank actions, constrain future actions, or un- if successful, threaten the integrity of count by requiring use of a Web brows- dermine advantages gained through an election outcome? er that is running in a separate (real or technological superiority. Externali- Investments in cybersecurity will virtual) computer on which there is a ties also must be incorporated into have to be recurring. Software, like a separate file system and only certain a cost assessment—attacks can have new bridge or building, has both an “safe” application programs are avail- both local and remote impact, because initial construction cost and an ongo- able. The loss of access to other files or the utility of an individual computer ing maintenance cost. It is true that programs hinders attackers but it also often depends on, or is affected by, an software does not wear out. Neverthe- hinders doing other tasks. entire network. less, software must be maintained: These enforcement mechanisms We should be mindful, though, that ˲˲ Today’s approaches for establish- increase the chances that malicious ac- investments directed at other national ing assurance in the systems we build tions will be prevented from executing, priorities—defense, foreign aid, and have limitations. So some vulnerabili- because they also block some actions social programs—are also difficult ties are likely to remain in any system that are not harmful. And users typi- to evaluate in purely objective ways. that gets deployed. When these vulner- cally feel inconvenienced when limita- Yet governments routinely prioritize abilities are discovered, patches must tions are imposed on how tasks must across making such investments. Even be developed and applied to systems be accomplished. So nobody will be in smaller, private-sector institutions, that have been installed. surprised to learn that users regularly the “bottom line” is rarely all that mat- ˲˲ Unanticipated uses and an en- disable enforcement mechanisms— ters, so they too have experience in vironment that evolves by accretion security is secondary to efficiently get- making investment decisions when mean that assumptions a system devel- ting the job done. ROI or other objective measures are oper will have made might not remain not available. valid forever. Such assumptions con- Security Can Be in Tension Any given intervention to encour- stitute vulnerabilities, creating further with Societal Values age investing in cybersecurity will allo- opportunities for attackers. Enhanced level of cybersecurity can cate costs across various sectors and, Ideally, systems will be structured to conflict with societal values, such as therefore, across different sets of indi- allow patching, and software produc- privacy, openness, freedom of expres- viduals. A decision to invest in the first ers will engage in the continuing ef- sion, opportunity to innovate, and ac- place might well depend on specifics fort to develop patches. Some business cess to information. Monitoring can of that allocation. We often strive to models (for example, licensing) are undermine privacy; authentication of have those individuals who benefit the better than others (for example, sales) people can destroy anonymity; authen- most be the ones who pay the most. at creating the income stream needed tication of programs prevents change, But the nature of networked infra- to support that patch development. which can interfere with flexibility in structures makes it difficult to charac- innovation and can be abused to block terize who benefits from cybersecurity Cost Is Not the Only Disincentive execution of software written by com- and by how much. For instance, civil Secure systems tend to be less con- petitors. Such tensions must be re- government (and much of defense), venient to use, because enforcement solved when designing interventions private industry, and individuals all mechanisms often intrude on usability. that will promote increased levels of share the same networks and use the ˲˲ One common approach for ob- cybersecurity. same software, so all benefit from the structing attacks is based on monitor- Moreover, societal values differ same security investments. Externali- ing. The system authenticates each across countries. We thus should not ties also come into play. For example, request before it is performed and expect to formulate a single uniform set should only the targeted political party uses the context of past actions when of cybersecurity goals that will serve for be paying to prevent cyberattacks that, deciding what requests are authorized the entire Internet. In addition, the ju-

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 37 viewpoints

risdiction of any one government nec- terrence depends on being able to attri- essarily has a limited geographic scope. bute acts to individuals or institutions So government interventions designed Secure systems and then punish the offenders. to achieve goals in some geographic tend to be less ˲˲ Attribution of attacks delivered region (where that government has over a network is difficult, because jurisdiction) must also accommodate convenient to use packets are relayed through multiple the diversity in goals and enforcement because enforcement intermediaries and, therefore, pur- mechanisms found in other regions. ported sources can be spoofed or re- mechanisms often written along the way. Attribution thus Flawed Analogies Lead intrude on usability. requires time-consuming analysis of to Flawed Interventions information beyond what might be Long before there were computers, lia- available from network traffic. bility lawsuits served to incentivize the ˲˲ Punishment can be problematic delivery of products and services that because attackers can work outside would perform as expected. Insurance the jurisdiction of the government was available to limit the insured’s mented by repurposing functionality). where their target is located. To limit costs of (certain) harms, where the for- Insurance. Insurance depends for or monitor all traffic that is destined mulation and promulgation of stan- pricing on the use of data about past to the hosts within some govern- dards facilitated decisions by insurers incidents and payouts to predict fu- ment’s jurisdiction can interfere with about eligibility for coverage. Finally, ture payouts. But there is no reason societal values such as openness and people and institutions were discour- to believe that past attacks and com- access to information. Such monitor- aged from malicious acts because their promises to computing systems are ing also is infeasible, given today’s net- bad behavior would likely be detected a good predictor of future attacks or work architecture. and punished—deterrence. compromises. I would hope succes- Computers and software comprise sive versions of a given software com- Making Progress a class of products and services, at- ponent will be more robust, but that The time is ripe to be having discus- tackers are people and institutions. is not guaranteed. For example, new sions about investment and govern- So it is tempting to expect that liabil- system versions often are developed ment interventions in support of cyber- ity, insurance, and deterrence would to add features, and a version that security. How much should we invest? suffice to incentivize investments to adds features might well have more And how should we resolve trade-offs improve cybersecurity. vulnerabilities than its predecessor. that arise between security and (other) Liability. Rulings about liability for Moreover, software deployed in a large societal values? It will have to be na- an artifact or service involve compari- network is running in an environment tional dialogue. Whether or not com- sons of observed performance with that is likely to be changing. These puter scientists lead, they need to be some understood basis for acceptable changes—which might not be under involved. And just as there is unlikely behaviors. That comparison is not the control of the developer, the user, to be a single magic-bullet technology possible today for software security, the agent issuing insurance, or even for making systems secure, there is un- since software rarely comes with full any given national government— likely to be a magic-bullet intervention specifications of what it should and might facilitate attacks, and that fur- to foster the needed investments. should not do. Software developers ther complicates the use of historical and service providers shun provid- data for predicting future payouts. Reference 1. Vardi, M. Cyber insecurity and cyber libertarianism. ing detailed system specifications be- Companies that offer insurance can Commun. ACM 60, 5 (May 2017), 5. cause specifications are expensive to benefit from requiring compliance create and could become an impedi- with industrywide standards since the Fred B. Schneider ([email protected]) Fred B. Schneider ment to making changes to support domain of eligible artifacts is now nar- is Samuel B. Eckert Professor of Computer Science and chair of the at Cornell University computer science deployment in new settings and to rowed, which simplifies predictions department, Cornell University, USA. support new functionality. Having a about possible adverse incidents and single list that characterizes accept- payouts. Good security standards also The impetus for this Viewpoint was a series of discussions able behavior for broad classes of will reduce the likelihood of adverse with Communications Senior Editor Moshe Vardi during the two years preceding his May 2017 Communications systems (for example, operating sys- incidents. However, any security stan- Editor’s Letter. Susan Landau, Lyn Millett, and Deirdre Mulligan read an earlier version of this Viewpoint tems or mail clients) also turns out to dard would be equivalent to a list of ap- and provided helpful and timely feedback. I am also be problematic. First, by its nature, proved components or allowed classes grateful to the two reviewers for their comments, which such a list could not rule out attacks of behavior. Such a list only can rule resulted in this Viewpoint having a better-defined focus. to compromise a property that is spe- out certain attacks and it can limit op- The author’s work has been supported in part by AFOSR grant F9550-16-0250 and NSF grant 1642120. The views cific only to some element in the class. portunities for innovation, so security and conclusions contained in this Viewpoint are those of the author and should not be interpreted as necessarily Second, to the extent that such a list standards are unlikely to be popular representing the official policies or endorsements, rules out repurposing functionality with software producers. either expressed or implied, of these organizations (and thereby blocks certain attacks), Deterrence. Finally, deterrence is or the U.S. government. the list would limit opportunities for considerably less effective in cyber- innovations (which often are imple- space than in the physical world. De- Copyright held by author.

38 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 CREDIT TK V C quality controlprocesses. workers toimprovetaskdesigns and er inquiries,andcommunicate with when clientspaywell,respondtowork- are happierandproducebetterwork researchers havelearnedthatworkers computer withoutlivingexpenses.But interface tohumanworkersbutavast oversight, asiftheplatformwerenotan tionless” completionofworkwithout prime clientstoexpectcheap,“fric- these expectations.Someplatforms havior. Clientsmaybeunawareof pectations aboutappropriateclientbe- crowdwork arediverse—astheirex- they work.Workers’ relationships to clients maynotfullyunderstandhow Crowdsourcing platformsarecomplex; ing or desire toefficiently.funding use from malicebutmisunderstand- ed responsibly. Typically this results not sourced research,isnotalwaysconduct- (see theaccompanyingfigure). pre-2008 toover20,000papersin2016 from lessthan1,000papersper year term “crowdsourcing” hasgrown High-level guidelines for the treatment of crowdworkers. High-level guidelinesforthetreatmentofcrowdworkers. at LeastMinimumWage with Crowds:Pay Crowdworkers Responsible Research Viewpoint cated thanthe averageU.S. worker. (“MTurk”), for example,aremoreedu- on Amazon’sMechanicalTurk platform ognized expertiseandskills. Workers have varied but undervalued or unrec- DOI:10.1145/3180492 Crowdsourcing, includingcrowd- INGLY ROWDSOURCING ISINCREAS- of papersincludingthe Google Scholar,thenumber ic research.Accordingto importantinscientif viewpoints 6 Workers M.S. Silberman, B. Tomlinson, R.LaPlante, J. Ross, L.Irani, and A. Zaldivar

2 -

secondary usage: thepercentageof dom answers. Butrejectionalsohasa ample, completingasurvey withran- prevent workersfromcheating—for ex- work foranyreason.Rejection exists to refuse to pay for (“reject”) completed tionally underpayormistreat workers. tations, andskills,theymayuninten- man workerswithdiverseneeds,expec- complex, error-pronesystemwithhu- search, involvesinteractingthrougha that crowdsourcingwork,includingre- search. When clients do not understand of insightthatcanleadtobetterre- terchangeable subjectsbutassources perspective, treatingworkersnotasin- offer researchersanopportunitytoshift through workerforums.Workers’ skills Many adviseclientsontaskdesign Papers inGoogle Scholarthatusetheterm“crowdsourcing.” 20,000 10,000 25,000 15,000 5,000 On MTurk,forexample,clients may 0 2000 2002 2004 MARCH 2018 | 2004

VOL. 61| 2008 hear that peoplethroughliving a earn some readers maybesurprised to secondary sourceofincome. While crowdwork asaprimaryorsignificant relevant forworkerswho relyon “reputation,” butnotthereverse. screen workersbasedon a formof client’s rejectionhistory. Clientscan rejections andnoinformationabouta MTurk offersworkersnowaytocontest workers’ eligibilityforothertasks. rors inqualitycontrol,compromising can benegativelyaffectedbyclienter- A worker’s“approvalrate,”however, automatically screenworkersfortasks. a proxyforworkerquality,andusedto ents chosetopayfor—isinterpreted as that is,thepercentageoftaskstheircli- tasks a worker has had “approved”— These dynamicsseemespecially NO. 3 | 2010 COMMUNICATIONS OFTHEACM 2012 2014

2016 39 viewpoints

crowdwork, research shows this is and overtime pay—but contractors are increasingly common, even in rich not. (Many countries have similar dis- ACM Transactions countries. In a 2015 International La- tinctions.) While this legal classification bour Organization survey of MTurk is unclear and contested, and there is on Reconfigurable workers (573 U.S. respondents), 38% growing recognition that at least some of U.S. respondents said crowdwork crowdworkers should receive many or Technology and was their primary source of income, all protections afforded employees (in- with 40% of these (15% of U.S. respon- cluding Salehi et al.,9 Michelucci and Systems dents) reporting crowdwork as their Dickinson,8 and Berg2), these intentions only source of income.2 In a 2016 Pew have not yet been realized. ACM TRETS is a survey of 3,370 MTurk workers, 25% of Our own research, which we have U.S. respondents said that MTurk spe- asked researchers to stop citing10 and peer-reviewed and cifically was the source of “all or most” will therefore not cite here, has been archival journal that of their income.5 used to justify underpayment of While it is to our knowledge gener- workers. Reporting on MTurk demo- covers reconfigurable ally not possible to be certain how rep- graphics in 2008–2009, we reported resentative any survey of crowdworkers that workers responding to our survey technology, systems, is, these findings are consistent with earned on average less than $2/hour. and applications both other MTurk-specific research This figure has been cited by research- and recent national surveys of online ers to justify payment of similar wages. on reconfigurable labor platform activity broadly—which Our (now outdated) descriptive re- computers. Topics includes “microtasking” platforms search, which reported averages from a (such as MTurk), platforms for in-per- sizable but not necessarily representa- include all levels son work (such as Uber), and platforms tive sample of MTurk workers, was not of reconfigurable for remote work (such as Upwork). For an endorsement of that wage. Addi- example, Farrell and Greig3 found that tionally, eight years have passed since system abstractions overall the “platform economy was a that study—it should not be used to and all aspects of secondary source of income,” but that orient current practice. “as of September 2015, labor platform Therefore, we build on a long-run- reconfigurable income represented more than 75% of ning conversation in computing re- total income for 25% of active [labor search on ethical treatment of crowd- technology platform] participants,” or approxi- workers (for example, Bederson and including platforms, mately 250,000 workers.a Quinn1) by offering the following high- With crowdwork playing an eco- level guidelines for the treatment of programming nomically important role in the lives of paid crowdworkers in research. environments and hundreds of thousands—or millions— Pay workers at least minimum wage of people worldwide, we ask: What are at your location. Money is the primary application successes. the responsibilities of clients and plat- motivation for most crowdworkers form operators? (see, for example, Litman et al.6 for Crowdsourcing is currently largely MTurk). Most crowdworkers thus re- “outside the purview of labor laws”8— late to paid crowdwork primarily as but only because most platforms classify work, rather than as entertainment or a workers as “independent contractors,” hobby; indeed, as noted previously, a not employees. “Employees” in the U.S. significant minority rely on crowdwork are entitled to the protections of the Fair as a primary income source. Most de- Labor Standards Act—minimum wage veloped economies have set minimum wages for paid work; however, the common requirement (noted earlier) that a Farrell and Greig3 report that 0.4% of adults “actively participate in” (receive income from) workers agree to be classified as inde- labor platforms each month. (“Labor plat- pendent contractors allows workers to forms” here include both platforms for in-per- be denied the protections afforded em- son work such as Uber as well as platforms for ployees, including minimum wage. remote work such as MTurk and Upwork.) Per Ethical conduct with respect to re- the CIA World Factbook, the U.S. total popula- For further information tion is 321,369,000, with approximately 80.1% search subjects often requires re- “adult” (“15 years or older”). Therefore the searchers to protect subjects beyond or to submit your number of U.S. adults earning more than 75% the bare minimum required by law; giv- of their income from labor platforms is ap- en the importance of money as a motiva- manuscript, proximately 0.25 * 0.004 * 0.801 * 321369000, or 257,415. “Adults” is interpreted by Farrell tion for most crowdworkers, it is ethi- visit trets.acm.org and Greig as “18 years or older,” not “15 years cally appropriate to pay crowdworkers or older,” so we round down to 250,000. minimum wage. Further, workers have

40 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 viewpoints

Learn from workers. If workers tell crowdsourced research responsi- you about technical problems or un- ble, researchers and IRBs must de- To make clear instructions, address them velop ongoing, respectful dialogue crowdsourced promptly, developing workarounds as with crowdworkers. needed for workers who have complet- research possible, ed the problematic task. Especially if Further Reading researchers and you are new to crowdsourcing, you For detailed treatment of ethical issues may unknowingly be committing er- in crowdwork, see Martin et al.7 For al- IRBs must develop rors or behaving inappropriately due ternatives to MTurk, see Vakharia and ongoing, respectful to your study design or mode of en- Lease11 or type “mturk alternatives” gagement. Many workers have been into any search engine. Readers inter- dialogue with active for years, and provide excellent ested in ethical design of labor plat- crowdworkers. advice. Workers communicate with forms should seek recent discussions one another and with clients in forums on “platform cooperativism” (for ex- (as described earlier); MTurk workers ample, platformcoop.net). in particular have articulated best practices for ethical research in the Dy- References 1. Bederson, B. and Quinn. A.J. Web workers unite! 9 requested this (Salehi et al. ). Ethics de- namo Guidelines for Academic Re- Addressing challenges of online laborers. In mands we take worker requests seriously. Proceedings of CHI ’11 EA (2011), 97–106. questers (guidelines.wearedynamo. 2. Berg, J. Income security in the on-demand economy: While crowdworkers are often locat- org; Salehi et al.9). Findings and policy lessons from a survey of crowdworkers. Comparative Labor Law & Policy ed around the world, minimum wage Currently, the design of major Journal 37, 3 (2016). at the client’s location is a defensible crowdsourcing platforms makes it dif- 3. Farrell, D. and Greig, F. Paychecks, paydays, and the online platform economy: Big data on income lower limit on payment. If workers are ficult to follow these guidelines. Con- volatility. JP Morgan Chase Institute, 2016. underpaid, for example, due to under- sider a researcher who posts a task to 4. Harris, M. Amazon’s Mechanical Turk workers protest: ‘I am a human being, not an algorithm.’ The Guardian estimation of how long a task might MTurk, and after the task is posted, (Dec. 3, 2014); http://bit.ly/2EcZvMS. take, correct the problem (for instance, discovers that even expert workers 5. Hitlin, P. Research in the crowdsourcing age, a case study. Pew Research Center, July 2016. on MTurk, with bonuses). On MTurk, if take twice as long as expected. This is 6. Litman, L., Robinson, J., and Rosenzweig, C. The workers are refused payment mistak- unsurprising; recent research shows relationship between motivation, monetary compensation, and data quality among U.S.- and enly, reverse the rejections to prevent that task instructions are often un- India-based workers on Mechanical Turk. Behavior damage to the workers’ approval rat- clear to workers. If this researcher Research Methods 47, 2 (Feb. 2015), 519–528. 7. Martin, D. et al. Turking in a global labour market. ing. Note that fair wages lead to higher wishes to pay workers “after-the-fact” Computer Supported Cooperative Work 25, 1 (Jan. quality crowdsourced research.6 bonuses to ensure they are paid the in- 2016), 39–77. 8. Michelucci, P. and Dickinson, J.L. The power of crowds. Remember you are interacting with tended wage, this can only be done Science 351, 6268 (2016), 32–33. human beings, some of whom com- 9. Salehi, N. et al., Eds. Guidelines for Academic one-by-one or with command-line Requesters—WeAreDynamo Wiki (2014); http://bit. plete these tasks for a living. Treat them tools. The former is time-consuming ly/1q6pY33. 10. Silberman, M. et al. Stop citing Ross et al. 2010, ‘Who at least as well as you would treat an in- and tedious; the latter is only usable are the crowdworkers?’; http://bit.ly/2FkrObs. person co-worker. As workers them- for a relative minority of clients. The 11. Vakharia, D. and Lease, M. Beyond Mechanical Turk: An analysis of paid crowd work platforms. In Proceedings selves have gone to great lengths to ex- platform’s affordances (or non-affor- of iConference 2015. (2015). press to the public,4 crowdworkers are dances) are powerful determiners of not interchangeable parts of a vast how clients (are able to) treat workers. M. Six Silberman ([email protected]) computing system, but rather human works in the Crowdsourcing Project at IG Metall, 60329 We suggest platform operators would Frankfurt am Main, Germany. beings who must pay rent, buy food, do workers, clients, and themselves a Bill Tomlinson ([email protected]) is a Professor in the and put children through school—and service by making it easier for clients Department of Informatics at the University of California, who have, just like clients, career and to treat workers well in these cases. Irvine, CA, USA, and a Professor in the School of Information Management, Victoria University of life goals and the desire to be acknowl- Finally, we call on university Institu- Wellington, New Zealand. edged, valued, and treated with respect. tional Review Boards to turn their atten- Rochelle LaPlante ([email protected]) is a Respond quickly, clearly, concisely, tion to the question of responsible professional crowdworker, Seattle, WA, USA. and respectfully to worker questions crowdsourced research. Crowdworkers Joel Ross ([email protected]) is a Senior Lecturer in the Information School, University of Washington, Seattle, and feedback via both email and work- relate to their participation in crowd- WA, USA. er forums (for example, turkernation. sourced research primarily as workers. Lilly Irani ([email protected]) is an Assistant Professor in com, mturkcrowd.com). In addition to Thus the relation between researchers the Communication Department and Science Studies being a reasonable way to engage with and crowdworkers is markedly different Program, University of California, San Diego, CA, USA. Andrew Zaldivar ([email protected]) is a Researcher in human workers, this engagement may than researchers’ relation to study par- the Department of Cognitive Sciences, University of also improve the quality of the work ticipants from other “pools.” While California, Irvine, CA, USA (now at Google). you receive, since you may be informed there may be some exceptions, we thus of task design problems before a great believe researchers should generally pay This material is based upon work supported in part by National Science Foundation Grant CCF-1442749. The deal of work has been done—and be- crowdworkers at least minimum wage. authors thank Janine Berg and Valerio De Stefano for fore you have incurred a responsibility comments. This Viewpoint reflects the authors’ views, not We urge IRBs to consider this position. any official organizational position. to pay for that work, which was done in These suggestions are a start, not a good faith. comprehensive checklist. To make Copyright held by authors.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 41 viewpoints

VDOI:10.1145/3132698 Hanna Wallach Viewpoint Computational Social Science ≠ Computer Science + Social Data The important intersection of computer science and social science.

HIS VIEWPOINT IS about differences between computer science and social science, and their implications for computational social science. Spoiler Talert: The punchline is simple. Despite all the hype, machine learning is not a be- all and end-all solution. We still need social scientists if we are going to use machine learning to study social phenomena in a responsible and ethical manner. I am a machine learning researcher by training. That said, my recent work has been pretty far from traditional machine learning. Instead, my focus has been on computational social science—the study of social phenomena using digitized information and computational and statistical methods. For example, imagine you want to know how much activity on websites such as Amazon or Netflix is caused by based adjustments to each senator’s tional social science sits at the inter- recommendations versus other fac- ideological position using their con- section of computer science, statistics, tors. To answer this question, you gressional voting history and the corre- and social science. might develop a statistical model for sponding bill text.4,8 For me, shifting away from tradi- estimating causal effects from observa- Finally, imagine you want to study tional machine learning and into this tional data such as the numbers of rec- the faculty hiring system in the U.S. to interdisciplinary space has meant that ommendation-based visits and num- determine whether there is evidence of I have needed to think outside the algo- bers of total visits to individual product a hierarchy reflective of systematic so- rithmic black boxes often associated or movie pages over time.9 cial inequality. Here, you might model with machine learning, focusing in- Alternatively, imagine you are inter- the dynamics of hiring relationships stead on the opportunities and chal- ested in explaining when and why sen- between universities over time using lenges involved in developing and us- ators’ voting patterns on particular is- the placements of thousands of tenure- ing machine learning methods to sues deviate from what would be track faculty.3 analyze real-world data about society. expected from their party affiliations Unsurprisingly, tackling these kinds This Viewpoint constitutes a reflec- and ideologies. To answer this ques- of questions requires an interdisciplin- tion on these opportunities and chal-

tion, you might model a set of issue- ary approach—and, indeed, computa- lenges. I structure my discussion here EVANNOVOSTRO BY IMAGE

42 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 viewpoints viewpoints

around three points—goals, models, and data—before explaining how ma- Calendar chine learning for social science there- The goals typically fore differs from machine learning for pursued by of Events other applications. computer scientists March 5–8 Goals and social scientists HRI ‘18: ACM/IEEE When I first started working in compu- International Conference on fall into two very Human-Robot Interaction V tational social science, I kept overhear- Chicago, IL, ing conversations between computer different categories. Contact: Takayuki Kanda, scientists and social scientists that in- Email: [email protected] volved sentences like, “I don’t get it— March 7–11 how is that even research?” And I could IUI’18: 23rd International not understand why. But then I found Conference on Intelligent User Interfaces this quote by Gary King and Dan Hop- Models Tokyo, Japan, kins—two political scientists—that, I These different goals—prediction and Co-Sponsored: ACM/SIG, think, really captures the heart of this explanation—lead to very different Contact: Shlomo Berkovsky, disconnect: “[C]omputer scientists may modeling approaches. In many predic- Email: shlomo.berkovsky@ csiro.au be interested in finding the needle in tion tasks, causality plays no role. The the haystack—such as [...] the right Web emphasis is firmly on predictive ac- March 11–15 CHIIR ‘18: Conference on page to display from a search—but so- curacy. In other words, we do not care Human Information Interaction cial scientists are more commonly inter- why a model makes good predictions; and Retrieval ested in characterizing the haystack.”6 we just care that it does. As a result, New Brunswick, NJ, In other words, the conversations I models for prediction seldom need to Sponsored: ACM/SIG, Contact: Chirag Shah, kept overhearing were occurring be- be interpretable. This means that there Email: [email protected] cause the goals typically pursued by are few constraints on their structure. computer scientists and social scientists They can be arbitrarily complex black March 15 AID ‘18: AI Decentralized 2018 fall into two very different categories. boxes that require large amounts of Toronto, ON, Canada, The first category is prediction. Predic- data to train. For example, GoogLeNet, Sponsored: ACM/SIG, tion is all about using observed data to a “deep” neural network, uses 22 layers Contact: Toufi Saliba, reason about missing information or fu- with millions of parameters to classify Email: [email protected] ture, yet-to-be-observed data. To use King images into 1,000 distinct categories.10 March 15–16 and Hopkins’ terminology, these are In contrast, explanation tasks are TAU ‘18: ACM International Workshop on Timing Issues in “finding the needle” tasks. In general, it fundamentally concerned with causal- the Specification and Synthesis is computer scientists and decision mak- ity. Here, the goal is to use observed of Digital Systems ers who are most interested in them. Sure data to provide evidence in support or Monterey, CA, enough, machine learning has traditional- opposition of causal explanations. As a Sponsored: ACM/SIG, Contact: Athanasius Spyrou, ly focused on prediction tasks—such as result, models for explanation must be Email: [email protected] classifying images, recognizing handwrit- interpretable. Their structure must be ing, and playing games like chess and Go. easily linked back to the explanation March 18–21 TEI ‘18: 12th International The second category is explanation. of interest and grounded in existing Conference on Tangible, Here the focus is on “why” or “how” ques- theoretical knowledge about the Embedded, and Embodied tions—in other words, finding plausible world. Many social scientists therefore Interaction Stockholm, Sweden, explanations for observed data. These ex- use models that draw on ideas from Sponsored: ACM/SIG, planations can then be compared with Bayesian statistics—a natural way to Contact: Martin Jonsson, established theories or previous findings, express prior beliefs, represent uncer- Email: [email protected] or used to generate new theories. Expla- tainty, and make modeling assump- March 19–21 nation tasks are therefore “characteriz- tions explicit.7 CODASPY ‘18: 7th ACM ing the haystack” tasks and, in general, it To put it differently, models for pre- Conference on Data and is social scientists who are most inter- diction are often intended to replace hu- Application Security and Privacy Tempe, AZ, ested in them. As a result, social scien- man interpretation or reasoning, where- Sponsored: ACM/SIG, tists are trained to construct careful re- as models for explanation are intended Contact: Gail-Joon Ahn, search questions with clear, testable to inform or guide human reasoning. Email: [email protected] hypotheses. For example, are women March 19–23 consistently excluded from long-term Data DATE ‘18: Design, Automation strategic planning in the workplace? As well as pursuing different goals, and Test in Europe Dresden, Germany, Are government organizations more computer scientists and social scien- Contact: Jan Madsen, likely to comply with a public records tists typically work with different types Email: [email protected] request if they know that their peer or- of data. Computer scientists usually ganizations have already complied? work with large-scale, digitized data-

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 43 viewpoints

sets, often collected and made avail- say, “Why not use these large-scale, new to most computer scientists, but able for no particular purpose other social datasets in combination with they are not new to social scientists. than “machine learning research.” In the powerful predictive models devel- contrast, social scientists often use oped by computer scientists?” How- Conclusion data collected or curated in order to ever, unlike the datasets tradition- To me, then, this highlights an impor- answer specific questions. Because ally used by computer scientists, these tant path forward. Clearly, machine this process is extremely labor inten- new datasets are often about people learning is incredibly useful—and, in sive, these datasets have traditionally going about their everyday lives—their particular, machine learning is useful been small scale. attributes, their actions, and their infor social science. But we must treat But—and this is one of the driving teractions. Not only do these datasets machine learning for social science forces behind computational social sci- document social phenomena on a very differently from the way we treat ence—thanks to the Internet, we now massive scale, they often do so at the machine learning for, say, handwriting have all kinds of opportunities to ob- granularity of individual people and recognition or playing chess. We can- tain large-scale, digitized datasets that their second-to-second behavior. As not just apply machine learning meth- document a variety of social phenome- a result, they raise some complicated ods in a black-box fashion, as if com- na, many of which we had no way of ethical questions regarding privacy, putational social science were simply studying previously. For example, my fairness, and accountability. computer science plus social data. We collaborator Bruce Desmarais and I It is clear from the media that one of need transparency. We need to priori- wanted to conduct a data-driven study the things that terrifies people the most tize interpretability—even in predictive of local government communication about machine learning is the use of contexts. We need to conduct rigorous, networks, focusing on how political ac- black-box predictive models in social detailed error analyses. We need to tors at the local level communicate with contexts, where it is possible to do more represent uncertainty. But, most im- one another and with the general pub- harm than good. There is a great deal of portantly, we need to work with social lic. It turns out that most U.S. states concern—and rightly so—that these scientists in order to understand the have sunshine laws that mimic the fed- models will reinforce existing structur- ethical implications and consequences eral Freedom of Information Act. These al biases and marginalize historically of our modeling decisions. laws require local governments to ar- disadvantaged populations.

chive textual records—including, in In addition, when datapoints are References many states, email—and disclose them humans, error analysis takes on a 1. Barocas, S. and Selbst, A.D. Big data’s disparate impact. California Law Review 104 (2016), 671–732. to the public upon request. whole new level of importance because 2. ben-Aaron, J. et al. Transparency by conformity: Desmarais and I therefore issued errors have real-world consequences A field experiment evaluating openness in local governments. Public Administration Review 77, 1 (Jan. public records requests to the 100 that involve people’s lives. It is not 2017), 68–77. county governments in North Carolina, enough for a model to be 95% accu- 3. Clauset, A., Arbesman, S., and Larremore, D.B. Systematic inequality and hierarchy in faculty hiring requesting all non-private email mes- rate—we need to know who is affected networks. Science Advances 1, 1 (Jan. 2015). sages sent and received by each coun- when there is a mistake, and in what 4. Gerrish, S. and Blei, D. How they vote: Issue-adjusted models of legislative behavior. In Advances in Neural ty’s department managers during a ran- way. For example, there is a substantial Information Processing Systems Twenty Five (2012), domly selected three-month time difference between a model that is 95% 2753–2761. 5. Hardt, H. How big data is unfair; http://bit.ly/1BBglLr. frame. Out of curiosity, we also decided accurate because of noise and one that 6. Hopkins, D.J. and King, G. A method of automated to use the process of requesting these is 95% accurate because it performs nonparametric content analysis for social science. American Journal of Political Science 54, 1 (Jan. email messages as an opportunity to perfectly for white men, but achieves 2010), 229–247. conduct a randomized field experiment only 50% accuracy when making pre- 7. Jackman, S. Bayesian Analysis for the Social Sciences. Wiley, 2009. to test whether county governments are dictions about women and minorities. 8. Lauderdale, B.E. and Clark, T.S. Scaling politically meaningful dimensions using texts and votes. more likely to fulfill a public records re- Even with large datasets, there is al- American Journal of Political Science 58, 3 (Mar. quest when they are aware that their ways proportionally less data available 2014), 754–771. 9. Sharma, A., Hofman, J., and Watts, D. Estimating peer governments have already fulfilled about minorities, and statistical pat- the causal impact of recommendation systems from the same request. terns that hold for the majority may be observational data. In Proceedings of the Sixteenth ACM Conference on Economics and Computation On average, we found that counties invalid for a given minority group. As a (2015), 453–470. who were informed that their peers result, the usual machine learning ob- 10. Szegedy, C. et al. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer had already complied took fewer days jective of “good performance on aver- Vision and Pattern Recognition (2015). to acknowledge our request and were age,” may be detrimental to those in a more likely to actually fulfill it. And we minority group.1,5 Hanna Wallach ([email protected]) is a Senior Researcher at Microsoft Research and an Adjunct ended up with over half a million Thus, when we use machine learn- Associate Professor at the University of Massachusetts email messages from 25 different ing to reason about social phenome- Amherst. county governments.2 na—and especially when we do so to This article is based on an essay that appeared on Medium draw actionable conclusions—we —see http://bit.ly/13QlExf. This work was supported in Challenges have to be exceptionally careful. part by NSF grant #IIS-1320219. Any opinions, findings and conclusions, or recommendations expressed in this Clearly, new opportunities like this More so than when we use machine material are those of the author and do not necessarily are great. But these kinds of opportu- learning in other contexts. But here is reflect those of the sponsor. nities also raise new challenges. Most the thing: these ethical challenges are conspicuously, it is very tempting to not entirely new. Sure, they may be Copyright held by author.

44 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 ACM Welcomes the Colleges and Universities Participating in ACM’s Academic Department Membership Program

ACM now offers an Academic Department Membership option, which allows universities and colleges to provide ACM Professional Membership to their faculty at a greatly reduced collective cost.

The following institutions currently participate in ACM’s Academic Department Membership program:

• Appalachian State University • Northeastern University • University of Colorado Denver • Armstrong State University • Ohio State University • University of Connecticut • Ball State University • Old Dominion University • University of Illinois at Chicago • Berea College • Pacific Lutheran University • University of Jamestown • Bryant University • Pennsylvania State University • University of Memphis • Calvin College • Regis University • University of Nebraska at Kearney • Colgate University • Roosevelt University • University of Nebraska Omaha • Colorado School of Mines • Rutgers University • University of North Dakota • Edgewood College • Saint Louis University • University of Puget Sound • Franklin University • San José State University • University of the Fraser Valley • Georgia Institute of Technology • Shippensburg University • University of Wyoming • Governors State University • St. John’s University • Harding University • Trine University • Virginia Commonwealth University • Hofstra University • Trinity University • Wake Forest University • Howard Payne University • Union College • Wayne State University • Indiana University Bloomington • Union University • Western New England University • Mount Holyoke College • University of California, Riverside • Worcester State University

Through this program, each faculty member receives all the benefits of individual professional membership, including Communications of the ACM, member rates to attend ACM Special Interest Group conferences, member subscription rates to ACM journals, and much more.

For more information: www.acm.org/academic-dept-membership

academic-dept-membership-cacm-ad.indd 1 12/22/17 10:35 AM practice

DOI:10.1145/3152481

Article development led by queue.acm.org

The unseen economic forces that govern the Bitcoin protocol.

BY YONATAN SOMPOLINSKY AND AVIV ZOHAR Bitcoin’s Underlying Incentives

BEYOND ITS ROLE as a protocol for managing and transferring money, the Bitcoin protocol creates a complex system of economic incentives that govern its inner workings. These incentives strongly impact the protocol’s capabilities and security guarantees, and the path of its future development. This article explores these economic undercurrents, their strengths and flaws, and how they influence the protocol. Bitcoin, which continues to enjoy growing popularity, is built upon an open peer-to-peer (P2P) network of this, the protocol requires nodes that 9 nodes. The Bitcoin system is “permissionless”—anyone participate in the system to show proof can choose to join the network, transfer money, and that they exerted computational effort to solve hard cryptographic puzzles even participate in the authorization of transactions. Key (proof-of-work) in order to participate to Bitcoin’s security is its resilience to manipulations actively in the protocol. by attackers who may choose to join the system under Nodes that engage in such work are called miners. The system rewards min- multiple false identities. After all, anyone can download ers with bitcoins for generating proof- the open-source code for a Bitcoin node and add as of-work, and thus sets the incentives for such investment of efforts. many computers to this network as they like, without The first and most obvious effect of having to identify themselves to others. To counter participants getting paid in bitcoins for

46 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 running software on their computers the protocol (machines with current A Quick Primer on was that, once bitcoins had sufficient ASICs are about a million times fast- the Bitcoin Protocol value, people started mining quite er than regular PCs when perform- Users who hold bitcoins and wish to a lot. In fact, efforts to mine intensi- ing this work). The Bitcoin network transfer them send transaction mes- fied to such a degree that most min- quickly grew and became more se- sages (via software installed on their ing quickly transitioned to dedicated cure, and competition for the pay- computer or smartphone) to one of the computer farms that used specialized ments given out periodically by the nodes on the Bitcoin network. Active gear for this purpose: first, GPUs that protocol became fierce. nodes collect such transactions from were used to massively parallelize Before discussing the interplay be- users and spread them out to their the work; and later, custom-designed tween Bitcoin’s security and its eco- peers in the network, each node inform- chips, or ASICs (application-specific nomics, let’s quickly look at the rules ing other nodes it is connected to about integrated circuits), tailored for the of the protocol itself; these give birth to the requested transfer. Transactions are

IMAGE BY SERGEY ILYASOV SERGEY ILYASOV BY IMAGE specific computation at the core of this complex interplay. then aggregated in batches called blocks.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 47 practice

Blocks, in turn, are chained together ficult because of the proof-of-work to create the blockchain, a record of all required for each block’s creation. In accepted bitcoin transactions. Each fact, as long as the attacker has less block in the chain references its pre- computational power than the entire decessor block by including a crypto- Bitcoin network put together, blocks graphic hash of that block—effectively The key to and transactions in the blockchain be- a unique identifier of that predecessor. Bitcoin’s operation come increasingly harder to replace as A complete copy of the blockchain is the chain above them grows. kept at every node in the Bitcoin net- is to get all nodes This difficulty in replacing the chain work. The process of block creation to agree on implies that it takes many attempts be- is called mining. One of its outcomes fore an attacker can succeed in doing (among others) is the printing of fresh the contents so. These failed attempts supposedly coins, which we call minting. impose a cost on attackers—mining The rules of the protocol make of the blockchain, blocks off of the longest chain without block creation extremely difficult; a which serves getting the associated mining rewards. block is considered legal only if it con- Naïve attacks are indeed costly for at- tains the answer to a hard cryptograph- as the record tackers (more sophisticated attacks are ic puzzle. As compensation, whenever of all transfers discussed later). miners manage to create blocks they The accompanying figure on page 49 are rewarded with bitcoins. Their rein the system. shows the evolution of the blockchain: ward is partly made up of newly mint- forks appear and are resolved as one of ed bitcoins and partly of mining fees the branches becomes longer than collected from all of the transactions the other. Blocks that are off the lon- embedded in their blocks. The rate of gest chain are eventually aban- minting is currently 12.5 bitcoins per doned. They are no longer extended, block. This amount is halved approxi- their contents (transactions colored mately every four years. As this amount in red) are ignored, and the miners decreases, Bitcoin begins to rely more that created them receive no reward. and more on transaction fees to pay At point 1 there are two alternative the miners. chains resulting from the creation of The key to Bitcoin’s operation is to a block that did not reference the latest get all nodes to agree on the contents tip of the blockchain. At point 2 the of the blockchain, which serves as the fork is resolved, as one chain is longer record of all transfers in the system. than the other. At point 3 there is Blocks are thus propagated quickly another fork that lasted longer, and to all nodes in the network. Still, it at point 4 the second fork is resolved. is sometimes possible for nodes to receive two different versions of the Bitcoin Economics 101: blockchain. For example, if two nodes Difficulty Adjustment and manage to create a block at the same the Economic Equilibrium time, they may hold two different ex- of Mining tensions to the blockchain. These Bitcoin’s rate of block creation is kept blocks might contain different sets of roughly constant by the protocol: payments, and so a decision must be Blocks are created at random intervals made on which version to accept. of roughly 10 minutes in expectation. The Bitcoin protocol dictates that The difficulty of the proof-of-work re- nodes accept only the longest chain as quired to generate blocks increases the correct version of events, as shown automatically if blocks are created too in the accompanying figure. (To be quickly. This mechanism has been put more precise, nodes select the chain in place to ensure that blocks do not that contains the most accumulated flood nodes as more computational computational work. This is usually the power is added to the system. The sys- longest chain.) This rule, often called tem thus provides payments to miners the “longest chain rule,” provides Bit- at a relatively constant rate, regardless coin with its security. An attacker who of the amount of computational power wishes to dupe nodes into believing invested in mining. that a different set of payments has Clearly, as the value (in U.S. dollars) occurred will need to produce a lon- of bitcoins rises, the mining business ger chain than that of the rest of the (which yields payments that are de- network—a task that is incredibly dif- nominated in bitcoins) becomes more

48 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 practice lucrative. More participants then find quires a proof-of-work that is useful to miner grows and pushing the other it profitable to join the group of min- society at large but cannot provide val- smaller (and, hence, less profitable) ers, and, as a consequence, the difficul- ue to the individual miner. (For some miners out of the game. The resulting ty of block creation increases. With this attempts at using other problems as a winner-takes-all dynamic inevitably increase in difficulty, mining blocks basis for proof-of-work, see Ball et al.,2 leads to centralization within the sys- slowly becomes more expensive. In the Miller et al.,8 and Zhang et al.13) tem, which is then at the mercy of the ideal case, the system reaches equilib- prevailing miner, and no security prop- rium when the cost of block creation Mining Decentralization erties can be guaranteed. equals the amount of extracted re- The key aspect of the Bitcoin protocol ASICs mining. The appearance of wards. In fact, mining will always be is its decentralization: no single entity ASICs initially alarmed the Bitcoin slightly profitable—mining is risky, has a priori more authority or control community. ASICs were orders of and also requires an initial investment over the system than others. This pro- magnitude more efficient at mining in equipment, and some surplus in motes both the resilience of the sys- bitcoins than previous systems. As the rewards must compensate for this. tem, which does not have a single an- this special hardware was not initially Hence, Bitcoin’s security effectively ad- chor of trust or single point of failure, easy to acquire, it provided its own- justs itself to match its value: A higher and competition among the different ers with a great advantage over other value also implies higher security for participants for mining fees. miners—they could mine at a much the protocol. To maintain this decentralization, lower cost. Those with this advantage As mining rewards continue to it is important that mining activity in would add ASIC-based proof-of-work decline (as per the protocol’s min- Bitcoin be done by many small entities to the system until the difficulty level ing schedule), the incentive to cre- and that no single miner significantly would be so high that everyone else ate blocks is expected to rely more on outweigh the others. Ideally, the re- would quit mining. The risk was then transaction fees. If a sudden drop in wards that are given to miners should that a single large miner would have bitcoin transaction volume occurs, reflect the amount of effort they put in: sole access to ASICs and would come these fees might be insufficient to a miner who contributes an α-fraction to dominate the Bitcoin system. Con- compensate miners for their compu- of the computational resources should cerns subsided after some time, as tational resources. Some miners might create an α-fraction of the blocks on av- ASICs became commercially available then halt their block creation process, erage, and as a consequence extract a and more widely distributed. temporarily. This may compromise the proportional α-fraction of all allocated In fact, ASIC mining actually intro- system, as the security of transactions fees and block rewards. duces long-term effects that contrib- depends on all honest miners actively In practice, some participants can ute to security. Later this article looks participating. (For additional work on benefit disproportionately from min- at how a miner can carry out profitable the incentives in Bitcoin after mining ing, for several different reasons. An double spending and selfish mining declines, see Carlsten et al.3) unbalanced reward allocation of this attacks. One can argue, however, that Many complain that the computa- sort creates a bias in favor of larger even selfish and strategic miners are tion required to create blocks wastes miners with more computational better off avoiding such attacks. In- resources (especially electricity) and power, making them more profitable deed, a miner who invested millions of has no economic goal other than im- than their smaller counterparts and dollars in mining equipment such as posing large costs on would-be attack- creating a constant economic under- ASICs is heavily invested in the future ers of the system. The proof-of-work is current toward the centralization of value of Bitcoin: the miner’s equip- indeed a solution to a useless crypto- the system. Even slight advantages ment is expected to yield payments of graphic puzzle—except, of course, that can endanger the system, as the miner bitcoins over a long period in the fu- this “useless” work secures the Bitcoin can use additional returns to purchase ture. Should the miner then use this network. But what if some of the work more and more computational power, gear to attack the system, confidence could be useful? Or could be produced raising the difficulty of mining as the in the currency would drop, and with more efficiently? If mining does not entail a waste of resources for each The evolution of the blockchain. node, then it also costs nothing for attackers to attack the system. In fact, if (eventually abandoned) the proof-of-work is less costly to solve, more honest participants join mining (to collect the rewards), and soon the difficulty adjustment mechanism raises the difficulty again. Hence, in a sense, the Bitcoin proof-of-work is built to spend a certain amount of re- (eventually sources no matter how efficient an abandoned) individual miner becomes. To derive time 1 2 3 4 substantial benefits from mining without an offsetting increase in costs re-

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 49 practice

it the value of bitcoins and future re- graphic hash function. Solving the large entity, professional miners may wards. The interests of miners are puzzle is mostly done via brute-force enjoy the economic benefits of size. thus, in some sense, aligned with the enumeration of different inputs. With a larger mining operation, such overall health of the system. A miner can gain an advantage by cre- miners are much more likely to invest All in all, ASIC mining introduces ating blocks using more efficient meth- in different optimizations, such as a barrier-to-entry to the system, as or- ods than his or her counterparts. In ad- finding sources of somewhat cheaper dinary people cannot simply join the dition to better hardware, an advantage electricity, or placing their equipment mining efforts; it thus reduces decen- can take a more algorithmic form. In in cooler regions to provide more effi- tralization. On the other hand, it intro- fact, an algorithmic “trick” nicknamed cient cooling to their machines (min- duces a form of barrier-to-exit, as min- ASICBoost has recently made head- ing usually consumes a great deal of ers cannot repurpose their equipment lines. ASICBoost enables the miner to electricity, and cooling the machines to other economic activities; it there- reuse some of the computational work presents a real challenge). Large min- fore contributes to security. performed during the evaluation of ers can also purchase ASICs in bulk for The appearance of competing cryp- one input for the evaluation of another. better prices. All of this translates to tocurrencies (for example, Litecoin, This algorithm is proprietary, patent natural advantages to size, a phenome- which essentially cloned Bitcoin), pending, and it is unclear who is and non that is not specific to Bitcoin but in some of which use the same proof-of- who is not using it. Such an algorithmic fact appears in many industries. These work as Bitcoin, offers alternatives for advantage can be translated to lower effects give large miners an advantage miners who wish to divert their mining power consumption per hash. Bitmain, and slowly pull the system toward a power elsewhere. This introduces com- a large manufacturer of ASICs for bit- centralized one. plex market dynamics. For example, coin mining that also operates some Many have raised concerns that when a specific currency loses some mining pools, was recently accused by most of today’s Bitcoin mining is done value, miners will divert their mining some of secretly deploying a hardware by Chinese miners. They enjoy better power to another cryptocurrency until variant of ASICBoost to increase its access to ASICs, cheaper electricity, the difficulty readjusts. This can cause profits. Allegations were made that this and somewhat lower regulation than fluctuations in block creation that de- company was politically blocking some similar operations in other locations. stabilize smaller cryptocurrencies. protocol improvements that would co- The Chinese government, which tight- Alternative systems with no ASIC incidentally remove their ability to use ly controls Internet traffic in and out mining. Interestingly, some crypto- ASICBoost. of China, could choose to disrupt the currencies use different proof-of-work Communication. Yet another meth- system or even seize the mining equip- puzzles that are thought to be more od for a miner to become more effi- ment that is within its borders. resistant to ASIC mining, that is, they cient is to invest in communication choose puzzles for which it is difficult infrastructure. By propagating blocks Mining Pools and Risk Aversion to design specialized hardware; for faster, and by receiving others’ blocks Bitcoin’s mining process yields very example, Ethereum uses the Ethash faster, a miner can reduce the chances high reward but with very low prob- puzzle (https://github.com/ethereum/ that their blocks will not belong to the ability for each small miner. A single wiki/wiki/Ethash). This is often longest chain and will be discarded ASIC that is running full time may achieved by designing algorithmic (“orphaned”). As off-chain blocks re- have less than a 1-in-600,000 chance problems that require heavy access to ceive no rewards, a better connection of mining the next block, which im- other resources, such as memory, and to the network translates to reduced plies that years can go by without find- that can be solved efficiently by com- losses. Admittedly, with Bitcoin’s cur- ing a single block. This sort of high- mercially available hardware. rent block creation rate, this advantage risk/high-reward payoff is not suitable These alternative systems are in is rather marginal; blocks are created for most. Many would prefer a small, principle more decentralized, but on infrequently, and speeding up delivery constant rate of income over long pe- the flip side they lack the barrier-to-exit by just a few seconds yields relatively riods of time (this is essentially risk effect and its contribution to security. little advantage. Nonetheless, better aversion).6 A constant income stream A similar effect occurs when cloud connectivity is a relatively cheap way to can be used, for example, to pay the mining becomes highly available. become more profitable. electric bills for mining. Some mining entities offer their equip- Furthermore, the effects of com- The formation of pools. Mining ment for rental over the cloud. The cli- munication become much more pro- pools are coalitions of miners that ents of these businesses are effectively nounced when the protocol is scaled combine their computational resourc- miners who do not have a long-term up and transaction processing is ac- es to create blocks together and share stake in the system. As such services celerated. Today, Bitcoin clears three the rewards among members of the become cheaper and more accessible, to seven transactions per second on pool. Since the pool’s workers together anyone can easily become a temporary average. Changing the parameters of find blocks much more often than each miner, with similar effects on security. Bitcoin to process more transactions miner alone, they are able to provide ASICBoost. Recall that creating a per second would increase the rate of small continuous payments to each block requires solving a cryptograph- orphan blocks and would amplify the worker on a more regular basis. ic puzzle unique to that block. This advantage of well-connected miners. From the perspective of the Bitcoin involves guessing inputs to a crypto- Economies of scale. As with any network, the pool is just a single min-

50 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 practice ing node. Pool participants interact among all workers in proportion to the with the pool’s server, which sends number of partial solutions each work- the next block header that the pool is er submitted. The number of shares working on to all workers. Each mem- was measured from the previous block ber tries to solve the cryptographic created by the same pool. puzzle corresponding to this block Mining pools Some workers came up with a way (in fact, they use small variants of the are coalitions to improve their rewards: if a pool was same block and work on slightly dif- unlucky and did not find a block for a ferent proof-of-work puzzles to avoid of miners while, many partial solutions (shares) duplicating work). Whenever a work- that combine would accumulate. If a block was then er finds a solution, it is sent to the found by the pool, its reward would be pool manager, who in turn publishes their computational split among many shares. Working to the block to the network. The block generate additional shares is just as provides a reward to the pool, which resources costly as before but yields low expected the manager then distributes among to create blocks rewards for this very reason. Instead, the all of the pool’s workers (minus some worker could just switch to another pool small fee). together and in which a block had been found more Reward distribution within pools share the rewards recently, and in which each additional and possible manipulations. Many share granted a higher expected reward. pools are public and open to any will- among members If many adopt this behavior, a pool that ing participant. Obviously, such pools of the pool. is temporarily unsuccessful should, in must take measures to ensure that fact, be completely abandoned by all only members who truly contribute to rational miners. Pool-hopping-resistant the pool’s mining efforts enjoy a por- reward schemes were quickly developed tion of the rewards. To that end, every and adopted by most mining pools.10 pool member sends partial solutions Block-withholding attacks. While a of the proof-of-work to the pool— miner cannot steal the block reward these are solutions that came “close” of a successful solution, he or she can to being full blocks. Partial solutions still deny the rewards from the rest of are much more common than full so- the miners in the pool. The miner can lutions, and anyone working on the choose to submit only partial solutions problem can present a steady stream to the pool’s manager but discard all of such attempts that fall short of the successful solutions. The miner thus target. This indicates that the worker receives a share of the rewards when is indeed engaged in work, and can be others find a solution, without provid- used to assess the amount of compu- ing any actual contribution to the pool. tational power each worker dedicates Discarding the successful solution sab- to the pool. Pools thus reward work- otages the pool, and causes a small loss ers in some proportion to the num- of income to the attacker. ber of shares that they earn (a share is In spite of the losses to an attack- granted for every partial solution that er, in some situations it is worthwhile is submitted). for mining pools to devote some of Fortunately, a pool member who their own mining power to sabotage has found a valid solution to the puzzle their competitors: the attacker pool cannot steal the rewards. The crypto- infiltrates the victim pool by register- graphic puzzle depends on the block ing some of its miners as workers in header, which is under the control of the victim pool. These workers then the pool’s manager. It encodes a com- execute a block-withholding attack. mitment to the contents of the block Careful calculations of the costs and itself (via a cryptographic hash), in- rewards show that, in some scenar- cluding the recipient of the block’s ios (depending on the sizes of the rewards. After finding a valid solution attacker and victim pools), the at- for a specific block header, one cannot tack is profitable.4 To prevent such tamper with the header without invali- schemes, a slight modification of the dating the solution. mining protocol has been proposed. In Nonetheless, pools are susceptible to the modified version, workers would some manipulations by strategic miners: not be able to discern between partial Pool hopping. In the early days of and full solutions to the proof-of-work Bitcoin, mining pools would simply puzzle and would not be able to selec- divide the reward from the latest block tively withhold full solutions.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 51 practice

Eliminating pools. While pools are valid transactions and to validate every good for small miners, mitigating new block before accepting it. their risk and uncertainty, they intro- Interestingly, despite this logic, duce some centralization to the system. sometimes miners mine on top of a The pool operator is essentially con- block without fully validating it. This trolling the combined computational The existence practice is known as SPV mining (sim- resources of many miners and is there- of selfish mining plified payment verification usually fore quite powerful. Some researchers refers to the use of thin clients that do proposed a technical modification to strategies implies not read the full contents of blocks). the mining protocol that undermines that there is Why would miners engage in the existence of public pools altogeth- building on top of an unvalidated er.7 Under this scheme, after finding something block? The answer again lies in in- a valid solution to the block, the pool centives. Some miners apply meth- member who mined it would still be fundamentally ods to learn about the hash ID of a able to redirect the rewards to them- broken in newly created block even before re- selves (without invalidating the solu- ceiving its entire contents. One such tion). Assuming many miners would the protocol’s method, known as spy-mining, in- claim the rewards for themselves, incentive structure: volves joining another mining pool pools would not be profitable and as a worker to detect block creation would therefore dissolve. rational events. Even when the block is re- profit-maximizing ceived, it takes time to validate the The Economics of Attacks transactions it contains. During this and Deviations from the Rules participants time, the miner is aware that the Earlier, this article described methods will not follow it. blockchain is already longer by one by which a miner can become more block. Therefore, rather than letting dominant within the protocol—both to the mining equipment lie idle until profit more than his or her fair share and the block is validated, the miner de- to generate more of the blocks in the cides to mine on top of it, under the chain. The methods discussed thus far assumption that it will most likely be do not violate any of the protocol’s rules; valid. To avoid the risk that the next in some sense, miners are expected to block will contain conflicts with the make the most of their hardware and transactions of the unverified block, infrastructure. This section discusses the miner does not embed new trans- direct violations of the rules of the pro- actions in the next block, hoping still tocol that allow miners to profit at the to collect the block reward. expense of others. In a sense, the ex- There is indeed evidence that min- istence of such strategies implies that ers are taking this approach. First, there is something fundamentally bro- some fraction of the blocks being ken in the protocol’s incentive struc- mined is empty (even when many trans- ture: rational profit-maximizing par- actions are waiting to be approved). An- ticipants will not follow it. other piece of evidence is related to an Informally, the protocol instructs unfortunate incident that took place in any node to: validate every new mes- July 2015. An invalid block was (unin- sage it receives (block/transaction); tentionally) mined due to a bug, and SPV propagate all valid messages to its miners added five additional blocks on peers; broadcast its own new blocks top of it without validating. Of course, immediately upon creation; and, build other validating miners rejected that its new blocks on top of the longest block and any block that referenced it, chain known to that node. Attacks on resulting in a six-block-long fork in the the protocol correspond to deviations network. Blocks that were discarded in from one or more of these instructions. the fork could have contained double- Validation. A miner who does not spent transactions. validate incoming messages is vulner- This event shows the dangers of SPV able—the next block might include an mining: it lowers the security of Bit- invalid transaction that he or she did coin and may trigger forks in the block- not verify, or reference an invalid pre- chain. Fortunately, miners have vastly decessor block. Other nodes will then improved the propagation and valida- consider this new block as invalid and tion time of blocks, so SPV mining has ignore it. This sets a clear incentive for less and less effect. The planned de- miners to embed in their blocks only cline in the minted reward given to

52 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 practice empty blocks will also lower the incen- There is no definite method to verify Cryptocurrency protocols should be tive to engage in such behavior. whether miners are engaging in self- placed on stronger foundations of in- Transaction propagation. A second ish mining or not. Given that very few centives. There are many areas left to important aspect of the Bitcoin pro- blocks are orphaned, it seems like this improve, ranging from the very basics tocol pertains to information propa- practice has not been taken up, at least of mining rewards and how they inter- gation: new transactions and blocks not by large miners (who would also act with the consensus mechanism, should be sent to all peers in the net- have the most to gain from it). One way through the rewards in mining pools, work. Here the incentive to comply to explain this is that miners who at- and all the way to the transaction fee with the protocol is not so clear. Miners tempt such manipulation over the long market itself. may even have a disincentive to share term may suffer loss to their reputation unconfirmed transactions that have and provoke outrage by the commu- Related articles yet to be included in blocks, especially nity. Another explanation is that this on queue.acm.org transactions that offer high fees.1 Min- scheme initially requires losing some ers have strong incentive to keep such of the selfish miner’s own blocks, and Research for Practice: Cryptocurrencies, Blockchains, and Smart Contracts transactions to themselves until they it becomes profitable only in the long Arvind Narayanan and Andrew Miller manage to create a block. Sending a run (it takes around two weeks for the http://queue.acm.org/detail.cfm?id=3043967 transaction to others allows them to protocol to readjust the difficulty level). Bitcoin’s Academic Pedigree snatch the reward it offers first. Thus Double spending is the basic attack Arvind Narayanan and Jeremy Clark far, most transaction fees have been against Bitcoin users: the attacker pub- http://queue.acm.org/detail.cfm?id=3136559 relatively low, and there is no evidence lishes a legitimate payment to the net- Certificate Transparency that transactions with high fees are be- work, waits for it to be embedded in the Ben Laurie ing withheld in this way. blockchain and for the victim to confirm http://queue.acm.org/detail.cfm?id=2668154 Next, let’s turn our attention to devia- it, and then publishes a longer chain of tions from the mining protocol intended blocks mined in secret that do not con- References 1. Babaioff, M. et al. On Bitcoin and red balloons. In explicitly to manipulate the blockchain. tain this payment. The payment is then Proceedings of the 13th ACM Conference on Electronic Commerce, 2012, 56–73. Selfish mining. Whenever a miner no longer part of the longest chain and, 2. Ball, M. et al. Proofs of Useful Work. IACR Cryptology creates a new block, the protocol says it effectively, “never happened.” ePrint Archive, 2017, 203. 3. Carlsten, M. et al. On the instability of Bitcoin without should be created on top of the longest This attack incurs a risk: the attack- the block reward. In Proceedings of the ACM SIGSAC chain the miner observes (that is, to er could lose the rewards for his or her Conference on Computer and Communications Security, 2016, 154–167. reference the tip of the longest chain blocks if they do not end up in the longest 4. Eyal, I. The Miner’s dilemma. In Proceedings of the as its predecessor) and that the miner chain. Surprisingly, and unfortunately, IEEE Symposium on Security and Privacy, 2015. 5. Eyal, I. and Sirer, E.G. Majority is not Enough: should send the new block immedi- a persistent attacker can eliminate this Bitcoin mining is vulnerable. In Proceedings of the ately to network peers. risk by following more sophisticated at- International Conference on Financial Cryptography 12 and Data Security. Springer, Berlin, 2014. Unfortunately, a miner can benefit tack schemes. The idea is to abandon 6. Fisch, B.A., Pass, R., Shelat, A. Socially Optimal Mining by deviating from these rules and act- the attack frequently, publish the secret Pools. arXiv preprint, 2017. 5,11 7. Miller, A. et al. Nonoutsourceable scratch-off puzzles ing strategically. The miner’s gen- attack chain, and collect rewards for its to discourage Bitcoin mining coalitions. In Proceedings eral strategy is to withhold the blocks’ blocks. By resetting the attack whenever of the 22nd ACM SIGSAC Conference on Computer and Communications Security. 2015. publication and keep the extension of the risk of losing block rewards is too 8. Miller, A. et al. Permacoin: Repurposing Bitcoin work the public chain secret. Meanwhile, high, the attacker can eliminate the at- for data preservation. In Proceedings of the IEEE Symposium on Security and Privacy, 2014. the public chain is extended by other tack cost and even be profitable in the 9. Nakamoto, S. Bitcoin: A peer-to-peer electronic cash system. Bitcoin.org, 2008; https://bitcoin.org/bitcoin.pdf. (honest) nodes. The strategic miner long term. These schemes are in essence 10. Rosenfeld, M. Analysis of Bitcoin Pooled Mining publishes the chain only when the risk a combination of selfish mining and Reward Systems. arXiv preprint, 2011. 11. Sapirshtein, A., Sompolinsky, Y. and Zohar, A. Optimal that it will not prevail as the longest double-spending attacks. selfish mining strategies in Bitcoin. InProceedings chain is too high. When the miner does Currently, double spending is not of the International Conference on Financial Cryptography and Data Security. Springer, Berlin, 2016. so, all nodes adopt the longer exten- observed often in the network. This 12. Sompolinsky, Y. and Zohar, A. Bitcoin’s security model sion that the miner suddenly released, could be because executing a success- revisited. In Proceedings of the International Joint Conference on Artificial Intelligence, Workshop on A.I. as dictated by the protocol, and they ful double spend is difficult, or because in Security, 2017. Melbourne. discard the previous public extension. the very miners who could execute 13. Zhang, F. et al. REM: Resource-Efficient Mining for Blockchains. Cryptology ePrint Archive. https://eprint. Importantly, this behavior increas- such attacks successfully also have a iacr.org/2017/179. es the miner’s share in the longest heavy stake in the system’s reputation. chain—meaning, it increases the per- Yonatan Sompolinsky is a Ph.D. student at Conclusion the School of Computer Science and Engineering at centage of blocks on the eventual lon- the Hebrew University of Jerusalem. He is founding gest chain that the miner generates. Incentives do indeed play a big role in scientist of DAGlabs. Recall that Bitcoin automatically ad- the Bitcoin protocol. They are crucial Aviv Zohar is a faculty member at the School of justs the difficulty of the proof-of-work for its security and effectively drive its Computer Science and Engineering at the Hebrew University of Jerusalem, and a cofounder and so as to keep the block creation rate daily operation. As argued here, min- chief scientist of QED-it. He has been researching the scalability, security, and underlying incentives constant. Thus, in the long run, a larg- ers go to extreme lengths to maximize of cryptocurrencies for several years. er relative share of blocks in the chain their revenue and often find creative translates to an increase in the miner’s ways to do so that are sometimes at Copyright held by authors/owners. absolute rewards. odds with the protocol. Publication rights licensed to ACM. $15.00.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 53 practice

DOI:10.1145/3152489 Was Stack Overflow lucky that the Article development led by queue.acm.org engineers had designed the prank so that it could be easily disabled? No, it was not luck. It was all in the playbook Being funny is serious work. for operational excellence in AFPs (April Fools’ pranks). BY THOMAS A. LIMONCELLI A successful AFP depends on many operational best practices. In this article, I will share some of the key ones.

What Makes an April Operational Fools’ Prank Funny? Before discussing the technical details, let’s look at what makes an AFP funny. The best AFPs are topical and absurdist. Excellence Topical means it refers to current events or trends. This makes it relevant and “a thinker.” Topical would be dis- playing your website upside-down after in April Fools’ a large and highly publicized acquisi- tion by a major Australian competitor. (Australians tell me that kind of joke never gets old.) Doing that to your web- Pranks site otherwise just announces that your Web developers finally read that part of the CSS3 spec. Secondly, it must be so absurd that it reveals a hidden truth. Absurdist humor is not simply silly for silliness’ sake. Absurdism acts as a crucible that burns away all lies to get to the truth. Stack Overflow’s 2017 prank, “Dance AT 10:23 UTC on April 1, 2015, stackoverflow.com enabled Dance Authentication,” was both topi- 1 cal and absurdist.3 The prank was a an April Fools’ prank called StackEgg. It was a simple blog post and accompanying dem- Tamagotchi-like game that appeared in the upper-right onstration video for Stack Overflow’s corner of the company’s website. Though it had been new (fictional) authentication system. Rather than the usual 2FA (two-factor tested, we did not account for the additional network authentication) system that requires activity it would generate. By 13:14 UTC the activity had an authenticator app or key fob, this grown to the point of overloading the company’s load system required users to turn on their webcams and dance their password. balancers, making the site unusable. All of the company’s This was topical because recent growth Web properties were affected. The prank had, essentially, in 2FA adoption meant many Internet users were experiencing 2FA for the created a self-inflicted denial-of-service attack. first time. It was absurdist because it The engineers involved in the prank didn’t panic. took the added burden and nuisance of They went to a control panel and disabled the feature. 2FA to an extreme. It revealed the truth that badly implemented security sacri- Network activity returned to normal, and the site fices convenience. was operating again by 13:47 UTC. The problem was Inspiration for absurdity should come from reality. For example, the diagnosed, fixed, and new code was pushed into Go programming language is an in- production by 14:56 UTC. The prank was saved! tentionally minimalistic language—

54 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 A scene from Stack Overflow’s 2017 AFP: Dance Dance Authenication (https://www.youtube.com/watch?v=VgC4b9K-gYU).

a reaction against bloated languages An AFP should not mock a particu- As with any feature, user acceptance such as C++ and Java. It seems like lar person (that is just mean) or group testing should be done with a wide va- every C++ or Java programmer who of people (that is just hateful). The ex- riety of users. Be sure to include some learns Go posts to forums demanding ception to this is that it is always OK to nonusers. You might consider doing dozens of features that are “missing.” mock people more powerful than you. user experience testing, but since most This leads to a discussion about why Punch up, not down. companies don’t, why start now? those features are intentionally miss- ˲˲ Punch up: Mock elite people who ing from Go. This discussion seems don’t realize how privileged they are; Engineer It Like Any Other Feature to happen on a weekly basis. A good mock the CEO who bragged he’s sav- The end-to-end process of creating and AFP for Go would be a blog post an- ing the company money by using his launching the prank should be the same nouncing that Go 2.0 will include all private jet. as any other feature. It should start with those “missing” features and, in fact, ˲˲ Don’t punch down: Do not mock a concept, then have a design and execu- they have been implemented and are the less fortunate—for example, don’t tion plan, launch plan, and operational ready for use. The article would then mock homeless people or any group runbook. Involve product management. link to the download page for Java. of powerless people in society; racist, Have requirements, specifications, a sexist, or homophobic humor is not project schedule, testing, and so on. If What Makes an April Fools’ funny because it is inherently punch- it is a big prank, beta testing with users Prank Un-Funny? ing down. sworn to secrecy may be required. A prank should not get in the way of busi- An AFP should be funny to the au- Like any major feature, the earlier ness or harm customers. For example, dience, not just the people who creat- you involve operations, the better. Op- a 2016 Gmail prank called “Drop the ed it. Every year plenty of companies erations’ worst nightmare is to be told Mic” gave users a button that would produce AFPs that fall flat because that a major feature is being launched send a farewell message to someone, they are inside jokes that everyone in tomorrow … “Would you please set up then block all email from that person the company finds hiii-larious. That 10 new servers and find a petabyte of … forever. There was no “Are you sure?” is all well and good, but if the inten- disk space?” April Fools’ pranks are prompt. As you can guess, this disrupt- tion was to make customers laugh, no different. They often require extra ed actual customers trying to do actual it really should not depend on them bandwidth, isolated servers, firewall business.2 Google disabled the prank a knowing that Larry in accounting rules, and other tasks that take days or

SCREEN CAPTURE FROM STACK OVERFLOW’S “DANCE DANCE AUTHENTICATION” VIDEO BY ARCADIA CREATIVE BY VIDEO AUTHENTICATION” DANCE “DANCE OVERFLOW’S FROM STACK SCREEN CAPTURE few hours later. loves World of Warcraft. weeks to complete.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 55 practice

Feature Flags During the prank, plausible deni- surdist. Obviously, they did not have The prank should be easy to enable ability is important. Act like it is real, or the benefit of reading this article. and disable. Hide the feature behind act like you don’t see it, or act like you I thought for a moment. What was a “feature flag.” With the flag off, the were not involved. Do, however, include the most recently controversy? Well, feature is in the code but dormant. a link to a page that explains that this facial-recognition software was be- Enabling the prank in production is a is just a joke. They say a joke isn’t fun- coming good enough and computa- matter of turning the flag on. Disabling ny if you have to explain it; if someone tionally inexpensive enough that it it is a simple matter of turning the flag doesn’t realize it is a joke, it can lead to was making the news and starting a off. Developers can test the feature by unfunny situations and hurt feelings. lot of ethical debates. enabling the flag in the development This is the Internet, not Mensa. I blurted out, “Hey, didn’t we just pur- and test environments. Some flag sys- Perform a project retrospective.5 Af- chase a company that makes facial-rec- tems can automatically be on for cer- ter the prank, sit down with everyone ognition software? You’d think a smart tain user segments. involved and reflect on what went well, bunch of people like that would be able Some companies can launch or dis- what didn’t go well, what should be to accurately place mustaches on all the able a feature only by rolling out new done the same way next time, and what photos in the corporate directory.” code into production. This is bad for should have been done differently. Pub- There was a short pause in the con- many reasons. It is riskier than fea- lish this throughout the organization. It versation. Then one manager said, “We ture flags: if the release that removes not only makes everyone feel included, just moved those people into my build- a prank is broken, do you revert to but it also educates people about how to ing. They sit down the hall from me.” the previous release (with the prank) do better next time. Yes, you may have Another manager chimed in that he or the prior release (which may be overloaded the network and created an manages the team that runs the cor- too old to deploy into production)? outage, but if everyone in the organiza- porate directory. Another manages the Code pushes are difficult to coordi- tion learned from this experience, your operations people for it. Another man- nate with PR, blog posts, and so on: organization is now smarter. Every out- ages the helpdesk most likely to receive they might take minutes or hours, not age that results in organizational learn- any complaints. seconds, like flipping a feature flag. ing is a blessing. If you hide informa- Soon, we had a plan. Code pushes require more skill: in tion, the organization stays ignorant. We started meeting weekly. We wrote many environments, code pushes are a design doc that spelled out how the done by specific people, who might Case Study: The Mustache Prank AFP would work, how we would shut it be asleep. In an emergency you want One of the most successful AFPs I was off after 24 hours, and, most important- to empower anyone to shut off the involved with was at a previous employ- ly, how individual people could opt out prank. The process should be quick er. Managers had been on a teleconfer- if they complained. A project manager and easy. Lastly, if the prank has over- ence for an hour brainstorming ideas was assigned to coordinate people on loaded the network, it may also affect for an AFP. They wanted one that would three different continents to make it all the systems that push new code. Mean- be visible only to employees. There is happen as expected. HR and executive while, a feature “flag flip” is simpler nothing less funny than managers try- management signed off on the project. and more likely to just plain work. ing to write a joke, so they turned to me. This was long before social media The way you structure an AFP project I was a half-manager so they assumed apps were doing this kind of thing, so is unusual in that the deadline cannot I’d have a half-funny suggestion. the primary question we kept getting change. There are three levers available After listening to the ideas they had was, “Is this really possible?” to managers: deadline, budget, and so far, I was not impressed. They were Was it technically feasible? Yes. It features. If a project is going to be late, irrelevant, not topical; silly, not ab- turns out the free software develop- management must adjust one of those ment kit that the company provided three. An AFP, however, cannot adjust included a mustache-placement API. the deadline and usually has a limited “Mustaching a person” was the demo budget. Therefore, it is important to they used to sell the company. segment the features of the prank. By the time April 1 rolled around, a First implement the basic prank, then new set of photos was prepared and add “would be nice” features. As you ready to be swapped in. The help- get closer to the deadline, throw away desk was trained on how to revert the less important features. When a individual photos. badly structured prank is late, all fea- The prank was a huge success. Ev- tures will be 80% done, which means eryone thought it was hilarious, except 0% of them can be launched. You blew for one person who complained and it. When a well-structured prank is late, opted out. 80% of the features are ready to launch, Afterward, we wrote up a retrospec- and the customers will be no wiser tive and thanked everyone involved. about the missing 20%. Structuring In such a highly distributed company, a project in this way requires skillful this was the best way to let everyone in-

planning up front. volved “take a bow.” KNESCHKE ROBERT BY USING PHOTO ASSOCIATES, ANDRIJ BORYS BY COLLAGE IMAGE

56 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 practice

Launch It Like It’s Hot launch in the next six months is already ture that permits people to write up the If an AFP will have significant resource running in your browser.4 answers to their own questions. A very needs, load testing is important. Ev- Google did something similar be- simple but effective AFP would be to eryone knows how to do load testing: fore launching IPv6 connectivity; your rename this feature “teddy bear mode” simulate thousands of HTTP requests browser was running invisible Java- and write a blog post claiming this to and take measurements. Find and fix Script that tested whether your ISP con- be an entirely new feature, based on the bottlenecks and repeat until you nection would fail if IPv6 were enabled. the power of a teddy bear’s ability to are satisfied. Worries were for naught, but the test help solve technical issues. You also need to plan for the situ- increased confidence before launch. ation where the AFP goes viral and Stack Overflow dark launches Summary receives 10 times or 100 times more new ad-serving infrastructure. When Successful AFPs require care and plan- users than you could ever expect. The launching major features, we first use ning. Write a design proposal and a easy strategy here is simply to plan on the system to transmit house ads that project plan. Involve operations early. disabling the AFP, but it would be dis- are invisible to users. Once perfor- If this is a technical change to your appointing that the reward for success mance is verified, we make the adver- website, perform load testing, prefer- was to turn the feature off. tisements visible. Sadly, we did not use ably including a “dark launch” or hid- Fixing such a situation is difficult this technique when launching Stack- den launch test. Hide the prank behind because normal solutions might take Egg, but now we know better. a feature flag rather than requiring a weeks to implement and April Fools’ new software release. Perform a retro- Day lasts only one day. If you fix a prob- Pranks with Minimal spective and publish the results widely. lem and relaunch the next day, you Operational Impact Remember that some of the best have missed the boat. Technical issues can be avoided with AFPs require little or no technical Facebook is in a similar situation proper testing, but there is a strat- changes at all. For example, one could when launching real features because egy that avoids the issue altogether. simply summarize the best prac- there is a lot of press around a new Simply create a prank that has no tices for launching any new feature feature and Facebook needs to “get operational impact, or directs the but write it under the guise of how it right” on the first try. When Face- impact elsewhere. to launch an April Fools’ prank. That book was new, growth was slow and The “Dance Dance Authentication” would be hilarious. bottlenecks could be fixed by sim- example is one such prank. The prank ply fixing them at the pace Facebook was simply a blog post and a link to a Related articles was growing. By 2008 Facebook had YouTube video (https://www.youtube. on queue.acm.org millions of users, and a new feature com/watch?v=VgC4b9K-gYU). This does would go from 0 to millions of users not entirely avoid the issue, but if your Ray Tracing Jell-O Brand Gelatin Paul S. Heckbert within hours. There would be no time success ends up overloading YouTube’s http://dl.acm.org/citation.cfm?id=42375 to fix unexpected bottlenecks. A failed network, at least it is not your problem. The Burning Bag of Dung—and Other launch is highly visible and embar- You can also simply take an exist- Environmental Antipatterns rassing, often becoming front-page ing feature and create an alternative Phillip Laplante news. There is no way, however, to explanation or history for it. For ex- http://queue.acm.org/detail.cfm?id=1035617 build an isolated system big enough ample, you may have heard of “the 10 Optimizations on Linear Search to perform load testing. teddy bear effect.” Many have ob- Thomas A. Limoncelli To solve this problem, Facebook served that often the act of asking http://queue.acm.org/detail.cfm?id=2984631 uses a technique called a “dark a question forces you to think out References launch:” testing a feature by first enough details to realize the answer 1. Dumke-von der Ehe, B. The making of StackEgg; launching it invisibly. For example, yourself. In Bell Labs folklore there http://balpha.de/2015/04/the-making-of-stackegg/. 2. Kottasova, I. Google’s April Fools’ prank backfires big Facebook launched Chat six months was a researcher known for helping time. CNNtech; http://money.cnn.com/2016/04/01/ early but made it invisible (CSS display: people with research roadblocks. technology/google-april-fool-prank-backfires/index.html. 3. Pike, K. Stack Overflow unveils the next steps in hidden). The HTML and JavaScript People would go to him for sugges- computer security. Stack Overflow Blog; https:// code was in your browser, but it did tions. By listening, they would come stackoverflow.blog/2017/03/30/stack-overflow- unveils-next-steps-computer-security/. not display itself. A certain percent- up with the answer themselves. Once, 4. Rossi, C. Pushing millions of lines of code five days a age of users received a signal to send he left on a long vacation and left a week. Facebook, 2011; https://www.facebook.com/ video/video.php?v=10100259101684977. simulated chat messages through the teddy bear on his desk with a note 5. Stack Exchange Network Status. Outage system. The percentage was turned that read, “Explain your problem to postmortem: March 31, 2015; http://stackstatus. net/post/115305251014/outage-postmortem- up over time so that developers could the bear.” Many people found it was march-31-2015. spot and fix any performance issues. By equally effective. (Lately, the Internet the time the feature was made visible has started calling this “the rubber Thomas A. Limoncelli is the co-editor of the book, “The Complete April Fools’ Day RFCs” (and the test messages were disabled), duckie effect.”) (http://www.rfchumor.com/), and is the site reliability Facebook’s engineers were confident Suppose you run a question-and- engineering manager at Stack Overflow Inc. in New York City. He blogs at EverythingSysadmin.com that the launch would not have perfor- answer website: some users post ques- and tweets @YesThatTom. mance problems. It is suggested that tions, and other people post answers. Copyright held by author. nearly every feature that Facebook will Suppose also that the website has a fea- Publication rights licensed to ACM. $15.00.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 57 practice

DOI:10.1145/3168505 from and every problem can always be Article development led by queue.acm.org rephrased as a people (or communications) problem; automation and culture are key. Perfect should never That said, the ground has shifted un- be the enemy of better. der the monitoring industry. This seismic change has caused existing tools to BY THEO SCHLOSSNAGLE change and new tools to emerge in the monitoring space, but that alone will not deliver us into the low-risk world of DevOps—not without new and updated thinking. Why? Change. Monitoring, at its heart, is about ob- Monitoring serving and determining the behavior of systems, often with an ulterior motive of determining “correctness.” Its purpose is to answer the ever-present question: Are my systems doing what they are in a DevOps supposed to? It’s also worth mentioning that systems is a very generic term, and in healthy organizations, systems are seen in a far wider scope than just World computers and computing services; they include sales, marketing, and finance, alongside other “business units,” so the business is seen as the complex interdependent system it truly is. That is, good monitoring can help people take a truly systems view not only of systems, but also organizations. Long dead are the systems that age like fine wine. Today’s systems are born in an agile world and remain fluid to accommodate changes in both the THE TITLE OF this article might suggest it is about how supplier and the consumer landscape. you are supposed to be monitoring systems in an A legitimate response to “adapt or die” organization that is making or has already made the is “I’ll do DevOps!” This highly dynamic system stands to challenge tradition- transformation into DevOps. Actually, this is an article al monitoring paradigms. to make you think about how computing has changed and how your concept of monitoring perhaps needs The Old World In a world with “slow” release cycles re-centering before it even applies to the brave new (often between six and 18 months), world of DevOps. making software operational was an interesting challenge. The system de- The harsh truth is that this is not just a brave, but ployed at the beginning of a release also a fast, new world. One of the primary drivers for looked a lot like the same system sev- adopting DevOps is speed—particularly the reduction eral months later. It’s not that it was stuck in time, but more that it was of risk at speed. An organization has to make many branched into a maintenance-only changes to accommodate this. The DevOps community mind-set. With maintenance comes bug fixes and even performance en- often talks about automation and culture. This makes hancements, but not new features, new a lot of sense, as automation is where speed comes systems components, removal of old

58 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 systems components, and new features sponsive work processes have been so having, you need to know what it looks or functions that would fundamentally widely adopted. DevOps is the organi- like when they are behaving. change the stress on the architecture. zational structure that makes the In this new world, you not only have Simply put, it is not very fluid. transformation possible. fluid development processes that can For monitoring, this lack of fluidity introduce change on a continual basis, is fantastic. If the system today is the The New World you also have adopted a microservices- system tomorrow and the exercise So, we are all on board with rapid and systems architecture pattern. Mi- that system does today is largely the fluid business and development pro- croservices simply dictate that the so- same tomorrow, then developing a set cesses, and we have continuous “ev- lution to a specific technical problem of expectations around how the sys- erything” to let us manage risk. The should be isolated to a network-acces- tem should behave becomes quite world is wonderful, right? Well, “con- sible service with clearly defined inter- natural. From a more pragmatic point tinuous monitoring” (in this new sense faces such that the service has free- of view, the baselines developed by ob- of continuous) doesn’t exist, and, be- dom. Many developers like this model, serving the behavior of the system’s sides, the name would be pretty dumb; as they are given more autonomy in the components will very likely live long, shouldn’t all monitoring have always design of the service, extending to useful lives. been continuous? choice of language, database technol- This article is not going to dive into The big problem here is that the ogy, etc. This freedom is very powerful, the risks involved with releasing a dra- fundamental principles that power but its true value lies in decoupling re- matic set of code changes infrequently, monitoring, the very methods that lease schedules and maintenance, and as there are countless stories (anecdot- judge if your machine is behaving it- allowing for independent higher-level al and otherwise) that state their mag- self, require an understanding of what decisions around security, resiliency, nitude and probabilistic certainty. Suf- good behavior looks like. Whether you and compliance. fice it to say: there be dragons on that are building statistical baselines, using This might seem like an odd tan- path. This is one of the many reasons formal models, or just winging it, in or- gent, but the conflation of these two

IMAGE BY PHAT1978/SHUTTERSTOCK BY IMAGE that agile, Kanban, and other more re- der to understand if systems are misbe- changes results in something quite

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 59 practice

unexpected for the world of monitor- that come with prescribed monitoring ing: the system of today neither looks for complex assembled systems; rarely like nor should behave like the system are systems in the tech industry assem- of tomorrow. bled and used in the same way at two different organizations. The likely sce- An Aside On ML And AI Today’s systems nario is that the monitoring will seem Many monitoring companies have are born in an useless, but in some cases it may pro- been struggling to keep up with the na- vide a false confidence that the systems ture of ephemeral architecture. Nodes agile world and are functioning well. come and go, and architectures dy- remain fluid to When it comes to monitoring the namically resize from one minute to “right thing,” always look at your busi- the next in an attempt to meet growing accommodate ness from the top down. The technical and shrinking demand. As nodes spin systems the organization operates are up and subsequently disappear, moni- changes in both only provisioned and operated to meet toring solutions must accommodate. the supplier and some stated business goal. Start by While some old monitoring systems monitoring whether that goal is being struggle with this concept, most mod- the consumer met. A tongue-in-cheek example: al- ern systems take this type of dynamic landscape. ways monitor the payroll system, be- systems sizing in stride. cause if you are not getting paid, what’s The second and largely unmet chal- the point? lenge is the dynamic nature not of an Mathematics: It’s necessary. Sec- architecture’s size but rather of its de- ond, embrace mathematics. In mod- sign. With microservices-based archi- ern times, functionality is table stakes; tectures and multiple agile teams con- it isn’t enough that the system is work- tinually releasing software and ing, it must be working well. It is a rare services, the design of modern archi- day when you have an important moni- tecture is constantly in transition. tor that consumes a Boolean value A hot topic in monitoring is how to “good” or “bad.” Most often, systems apply ML (machine learning) and AI (ar- are being monitored around delivered tificial intelligence) to the problems at performance, so the consumed values hand, but the current approaches seem (or indicators) are numbers and often to be attempting to solve yesterday’s latencies (a time representing how problems and not tomorrow’s. AI and long a specific operation took). You are ML provide an exceptionally rich new dealing with numbers now, so math is set of techniques to solve problems and required, like it or not. Basic statistics will undoubtedly prove instrumental in are a fundamental requirement for the monitoring world, but the prob- both asking and interpreting the an- lems they must tackle are not that of swers to questions about the behavior modeling an architecture and learning of systems. to guide its operations. The architec- As systems grow and the focus turns ture it learns today will have changed by more to their behavior, data volumes tomorrow, and any guidance will be an- rise. In seven years, Circonus has expe- tiquated. Instead, to make a significant rienced an increase in data volume of impact, AI and ML approaches need to almost seven orders of magnitude. take a step back and help guide pro- Some people still monitor systems by cesses and design. taking a measurement from them every minute or so, but more and more Characteristics of people are actually observing what Successful Monitoring their systems are doing. This results in It would be cruel to cast a gloomy shad- millions or tens of millions of mea- ow on the state of monitoring without surements per second on standard providing some tactical advice. Luck- servers. People tend not to solve diffi- ily, many people are monitoring their cult problems unless the answers are systems exceptionally well. Here is valuable. Handling 10 million mea- what they have in common: surements per second from a single What is more important than how. server when you might have thousands The first thing to remember is that all of servers might sound like overkill, but the tools in the world will not help you people are doing it because the technol- detect bad behavior if you are looking ogy exists that makes the cost of finding at the wrong things. Be wary of tools the answers less than the value of those

60 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 practice answers. People do it because they are wholly disheartening to think you’ve blog/why-percentiles-dont-work-the- able to run better, faster systems and done a good job and met expectations, way-you-think.) beat the competition. To handle data at and then learn the goalposts have RSM allows you to look at actual sys- that volume, you must also use a capa- moved or that you cannot articulate why tem behavior comprehensively, ac- ble set of tools. To form intelligent you’ve been successful. The art of the counting for the whole distribution of questions around data at this volume, SLI (service-level indicator), SLO (ser- observed performance instead of the you must embrace mathematics. vice-level objective), and SLA (service- synthetically induced measurements As you might imagine, without a set level agreement) reigns here. Almost that consistently misrepresent the ex- of tools to help you perform fast, accu- every low-level, ad hoc monitor and ev- perience of using the system. rate, and appropriate mathematical ery high-level executive KPI (key perfor- The transition from synthetic web analysis against your observed data, you mance indicator) can be articulated in monitoring to RUM was seismic; ex- will be at a considerable disadvantage; terms of “service level.” Understanding pect nothing less from the impending luckily, there are myriad choices from the service your business provides and transition to RSM. Python and R to tools that will help you the levels at which you aim to deliver find more comprehensive solutions that service is the heart of monitoring. Don’t Delay from modern monitoring vendors. SLIs are things that you have identi- Today, with architectures dynamically Data retention. A third important fied as directly related to the delivery of shifting in size by the minute or hour characteristic of successful monitor- a service. SLOs are the goals you set for and shifting in design by the day or ing systems is data retention. Monitor- the team responsible for a given SLI. the week, we need to step back and re- ing data has often been considered low SLAs are SLOs with consequences, of- member that monitoring is about un- value and high cost and is often exten financial. Though a slight oversim- derstanding the behavior of systems, punged with impudence. Times have plification, think about it like this: and that systems need not be limited to changed, and, as with all things com- What is important? What should it computers and software. A business is puting, the cost of storing data has fallen look like? What should I promise? For a complex system itself, including de- dramatically. More importantly, DevOps this, a good understanding of histo- coupled but connected subsystems of have changed the value of long-term grams can help. sales, marketing, engineering, finance, retention of this data. DevOps is a and so on. Monitoring can be applied culture of learning. When things go From RUM to RSM to all of these systems to measure im- wrong, and they always do, it is critical Monitoring in the Web world moved portant indicators and detect changes to have a robust process for interrogat- long ago from the slow, synthetic ping in overall systems behavior. ing the system and the organization to of a website to recording and analyzing Monitoring can seem quite over- understand how the failure transpired. every interaction with every user; syn- whelming. The most important thing This allows processes to be altered to thetic monitoring of the web gave way to remember is that perfect should reduce future risk. That’s right: learn- to RUM (real user monitoring) at the never be the enemy of better. DevOps ing reduces risk. turn of the century and no one looked enables highly iterative improvement At the pace we move, it is undeni- back. As we build more smaller, decou- within organizations. If you have no able that your organization will devel- pled services, we move into a realm of monitoring, get something; get any- op intelligent questions regarding a being responsible for servicing other thing. Something is better than noth- failure that were missed immediately small systems—these are what engi- ing, and if you have embraced DevOps, after past failures. Those new ques- neering SLOs are usually built around. you have already signed up for making tions are crucial to the development of The days of average latency for an it better over time. your organization, but they become ab- API request or a database interaction solutely precious if you can travel back or a disk operation (or even a syscall!) Related articles in time and ask those questions about are disappearing. RSM (real systems on queue.acm.org past incidents. This is what data reten- monitoring) is coming, and we will, tion in monitoring buys you. The new Black Box Debugging just as with RUM, be recording and James A. Whittaker, Herbert H. Thompson processes and interrogation methods analyzing systems-level interactions— https://queue.acm.org/detail.cfm?id=966807 you learn during your postmortems every one of them. Over the last decade Statistics for Engineers leading up to this year’s cyber-Thurs- increased systems observability (such Heinrich Hartmann day shopping traffic can now be ap- as the widely adopted DTrace and https://queue.acm.org/detail.cfm?id=2903468 plied to last year’s cyber-Thursday Linux’s eBPF) and improvements in Time, but Faster shopping traffic. This often leads to time-series databases (such as the first- Theo Schlossnagle fascinating and valuable learning that, class histogram storage in Circonus’s https://queue.acm.org/detail.cfm?id=3036398 you guessed it, reduces future risk. IRONdb) have made it possible to de- Be articulate about what success liver RSM. (For a detailed look at why Theo Schlossnagle is a software engineer and serial entrepreneur. He currently leads Circonus, where looks like. The final piece of advice for a histogram storage of data is different he focuses on building distributed systems and scale successful monitoring system is to be and, more importantly, relevant, see to help people analyze the behavior of their distributed systems at scale over time. specific about what success looks like. the review by Baron Schwartz, “Why

Using a language to articulate what suc- Percentiles Don’t Work the Way You Copyright held by author. cess looks like allows people to win. It is Think.” https://www.vividcortex.com/ Publication rights licensed to ACM. $15.00

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 61 contributed articles

DOI:10.1145/3127323 As the software industry enters the era of language-oriented programming, it needs programmable programming languages.

BY MATTHIAS FELLEISEN, ROBERT BRUCE FINDLER, MATTHEW FLATT, SHRIRAM KRISHNAMURTHI, ELI BARZILAY, JAY MCCARTHY, AND SAM TOBIN-HOCHSTADT A Programmable Programming Language

IN THE IDEAL world, software developers would analyze each problem in the language of its domain and then articulate solutions in matching terms. They could thus easily communicate with domain experts and separate problem-specific ideas from the details of general-purpose languages and specific key insights program design decisions. ˽˽ Language-oriented programming is In the real world, however, programmers use a an emerging software-development mainstream programming language someone else paradigm likely to revolutionize the way people build software. picked for them. To address this conflict, they resort ˽˽ It elevates “language” itself to a software to—and on occasion build their own—domain-specific building block, with the same status as languages embedded in the chosen language objects, modules, and components. ˽˽ As with other paradigms, language (embedded domain-specific languages, or eDSLs). orientation thrives when the base language supports it directly; the Racket For example, JavaScript programmers employ jQuery project has worked on support for for interacting with the Document Object Model and language-oriented programming for 20 years, providing a platform for exploring React for dealing with events and concurrency. this exciting new development in depth.

62 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 As developers solve their problems and is rather cumbersome. To create guage-oriented programming, or in appropriate eDSLs, they compose and deploy a language, programmers LOP, at two different levels. At the these solutions into one system; that usually must step outside the chosen practical level, the goal is to build a is, they effectively write multilingual language to set up configuration files programming language that enables software in a common host language.a and run compilation tools and link-in language-oriented software design. This Sadly, multilingual eDSL program- the resulting object-code files. Worse, the language must facilitate easy creation ming is done today on an ad hoc basis host languages fail to support the proper of eDSLs, immediate development of and sound integration of components in components in these newly created a The numerous language-like libraries in script- different eDSLs. Moreover, most avail- languages, and integration of compo- ing languages (such as JavaScript, Python, and able integrated development environ- nents in distinct eDSLs; Racket is avail- Ruby), books (such as Fowler and Parson),20 ments (IDEs) do not even understand able at http://racket-lang.org/ and websites (such as Federico Tomassetti’s, eDSLs or perceive the presence of code At the conceptual level, the case for https://tomassetti.me/resources-create- programming-languages/) are evidence written in eDSLs. LOP is analogous to the ones for object- of the desire by programmers to use and The goal of the Racket project is oriented programming and for concur- 3 ILLUSTRATION BY CHRIS LABROOY BY ILLUSTRATION develop eDSLs. to explore this emerging idea of lan- rency-oriented programming. The

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 63 contributed articles

Figure 1. Small language-oriented programming example. former arose from making the creation and manipulation of objects syntacti- extends cally simple and dynamically cheap, video typed/video the latter from Erlang’s inexpensive builds on builds on process creation and message passing. Both innovations enabled new video/ffi turnstile ways to develop software and trig- builds on builds on gered research projects. The question is how our discipline will realize LOP builds on builds on syntax-parse and how it will affect the world of software. builds on Our decision to develop a new language—Racket—is partly an historical racket artifact and partly due to our desire to free ourselves from any unnecessary constraints of industrial mainstream Figure 2. A plain Racket module. languages as we investigate LOP. The next section spells out how Racket got started, how we honed in on LOP, and what the idea of LOP implies.

Principles of Racket The Racket project dates to January 1995 when we started it as a language for experimenting with pedagogic programming languages.15 Work- ing on them quickly taught us that a language itself is a problem-solving tool. We soon found ourselves developing different languages for differ- Figure 3. A module for describing a simplex shape. ent parts of the project: a (meta-) language for expressing many pedagogic languages, another for specializing the DrRacket IDE,15 and a third for managing configurations. In the end, the software was a multilingual system, as outlined earlier. Racket’s guiding principle reflects the insight we gained: empower programmers to create new programming languages easily and add them with a friction-free process to a codebase. By “language,” we mean a new syntax, a Figure 4. Lambda, redefined. static semantics, and a dynamic semantics that usually maps the new syntax to elements of the host language and possibly external languages via a foreign-function interface (FFI). For a concrete example, see Figure 1 for a di- agram of the architecture of a recently developed pair of scripting languages for video editing2,b designed to assist people who turn recordings of conference presentations into YouTube videos and channels. Most of that work is

b The video language, including an overview of the implementation, is available as a use-case artifact at https://www2.ccs.neu.edu/racket/ pubs/#icfp17-acf

64 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 contributed articles repetitive—adding preludes and post- Libraries and ludes, concatenating playlists, and su- Languages Reconciled perimposing audio—with few steps Racket is an heir of Lisp and Scheme. demanding manual intervention. This Unlike these ancestors, however, Rack- task calls for a domain-specific script- et emphasizes functional over impera- ing language; video is a declarative Most notably, tive programming without enforcing eDSL that meets this need. Racket eliminates an ideology. Racket is agnostic when it The typed/video language adds a comes to surface syntax, accommodat- type system to video. Clearly, the do- the hard boundary ing even conventional variants (such as main of type systems comes with its own between library Algol 60).d Like many languages, Racket language of expertise, and typed/vid- comes with “batteries included.” eo’s implementation thus uses turn- and language, Most notably, Racket eliminates the stile,6 an eDSL created for expressing hard boundary between library and lan- type systems. Likewise, the implementa- overcoming guage, overcoming a seemingly intrac- tion of video’s rendering facility calls a seemingly table conflict. In practice, this means for bindings to a multimedia frame- new linguistic constructs are as seam- work. Ours separates the binding defini- intractable conflict. lessly imported as functions and classes tions from the repetitive details of FFI from libraries and packages. For exam- calls, yielding two parts: an eDSL for ple, Racket’s class system and for loops multimedia FFIs, dubbed video/ffi, are imports from plain libraries, yet and a single program in the eDSL. Final- most programmers use these con- ly, in support of creating all these eDSLs, structs without ever noticing their na- Racket comes with the syntax parse ture as user-defined concepts. eDSL,7 which targets eDSL creation. Racket’s key innovation is a modular The LOP principle implies two sub- syntax system,17,26 an improvement over sidiary guidelines: Scheme’s macro system,11,24,25 which in Enable creators of a language to en- turn improved on Lisp’s tree-transfor- force its invariants. A programming lan- mation system. A Racket module pro- guage is an abstraction, and abstrac- vides such services as functions, classes, tions are about integrity. Java, for and linguistic constructs. To implement example, comes with memory safety them, a module may require the services and type soundness. When a program of other modules. In this world of mod- consists of pieces in different languag- ules, creating a new language means es, values flow from one context into simply creating a module that provides another and need protection from op- the services for a language. Such a mod- erations that might violate their integ- ule may subtract linguistic constructs rity, as we discuss later; and from a base language, reinterpret oth- Turn extra-linguistic mechanisms into ers, and add a few new ones. A language linguistic constructs. A LOP program- is rarely built from scratch. mer who resorts to extra-linguistic Like Unix shell scripts, which specify mechanisms effectively acknowledges their dialect on the first line, every Racket that the chosen language lacks expres- module specifies its language on the first sive power.13,c The numerous external line, too. This language specification languages required to deal with Java refers to a file that contains a language- projects—a configuration language, a defining module. Creating this file is all it project description language, and a takes to install a language built in Racket. makefile language—represent symp- Practically speaking, a programmer may toms of this problem. We treat such develop a language in one tab of the IDE, gaps as challenges later in the article. while another tab may be a module written They have been developed in a in the language of the first. Without ever feedback loop that includes DrRack- leaving the IDE to run compilers, link- et15 plus typed,36 lazy,4 and pedagogi- ers, or other tools, the developer can cal languages.15 modify the language implementation in the first tab and immediately experience the modification in the second; c Like many programming-language researchers, we subscribe to a weak form of the Sapir-Whorf hypothesis; see http://docs.racket-lang.org/ d See http://docs.racket-lang.org/algol60/, as algol60/ and https://www.hashcollision.org/ well as well as https://www.hashcollision.org/ brainfudge/ showing how Racket copes with brainfudge/, which shows how Racket copes obscure syntax. with obscure syntax.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 65 contributed articles

that is, language development is a fric- structs via the module system. It allows tion-free process in Racket. eDSL creators to ease their users into a In the world of shell scripts, the first- new language by reusing familiar syn- line convention eventually opened the tax, but reinterpreted. door to a slew of alternatives to shells, Consider lambda expressions, for including Perl, Python, and Ruby. The In general, example. Suppose a developer wishes Racket world today reflects a similar cooperating to equip a scripting language (such as phenomenon, with language libraries video) with functions that check proliferating within its ecosystem: multilingual whether their arguments satisfy speci- racket/base, the Racket core lan- components fied predicates. Figure 4 shows the ba- guage; racket, the “batteries includ- sic idea: ed” variant; and typed/racket, a must respect line 01 The module uses the racket typed variant. Some lesser-known ex- language. amples are datalog and a web-serv- the invariants line 03 It exports a defined com- er language.27,30 When precision is established by pile-time function, new-la m b d a, un- needed, we use the lowercase name of der the name lambda, which is over- the language in typewriter font; other- each participating lined in the code to mark its origin as wise we use just “Racket.” language. this module. Figure 2 is an illustrative module. Its line 05 Here, the module imports first line—pronounced “hash lang rack- tools from a library for creating robust et base”—says it is written in racket/ compile-time functions conveniently.7 base. The module provides a single line 07 The comment says a function function, walk-sim plex. The accom- on syntax trees follows. panying line comments—introduced line 08 While (define (f x) . . . ) with semicolons—informally state a introduces an ordinary function f of x, type definition and a function signature (define-syntax (c stx) . . . ) cre- in terms of this type definition; later, we ates the compile-time function c with a show how developers can use typed/ single argument, stx. racket to replace such comments with line 09 As with many functional lan- statically checked types, as in Figure 5. guages, Racket comes with pattern- To implement this function, the mod- matching constructs. This one uses ule imports functionality from the con- syntax-parse from the library men- straints module outlined in Figure 3. tioned earlier. Its first piece specifies The last three lines of Figure 2 sketch the to-be-matched tree (stx); the re- the definition of the walk-sim plex mainder specifies a series of pattern- function, which refers to the maximiz- responses clauses. er function imported from constraints. line 10 This pattern matches any The "constraints" module in Fig- syntax tree with first token as new- ure 3 expresses the implementation of its lambda followed by a parameter speci- only service in a domain-specific lan- fication and a body. The annotation guage because it deals with simplexes, :id demands that the pattern variables which are naturally expressed through a x and predicate match only identifi- system of inequalities. The module’s ers in the respective positions. Like- simplex language inherits the line-com- wise, :expr allows only expressions to ment syntax from racket/base but match the body pattern variable. uses infix syntax otherwise. As the com- line 11 A compile-time function syn- ments state, the module exports a single thesizes new trees with syntax. function, maximizer, which consumes line 12 The generated syntax tree is a two optional keyword parameters. When lambda expression. Specifically, the called as (maximizer #:x n), as in Figure function generates an expression that 2, it produces the maximal y value of the uses lambda. The underline in the system of constraints. As in the lower half code marks its origin as the ambient of Figure 3, these constraints are speci- language, here racket. fied with conventional syntax. other lines Wherever the syntax sys- In support of this kind of program- tem encounters the pattern variables ming, Racket’s modular syntax system x, predicate, and body, it inserts the benefits from several key innovations. respective subtrees that match x, predi- A particularly illustrative one is the cate, and body. ability to incrementally redefine the When another module uses “new- meaning of existing language con- lam” as its language, the compiler

66 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 contributed articles

elaborates the surface syntax into the scenario. On the left, a module written create such a veneer with a foreign inter- 1core VERSION language 1 like this in typed/racket exports a numeric face that allows parenthesized C-level 5 (lambda (x :: integer?) (+ x 1)) differentiation function. On the right, programming. Programmers can refer to –elaborates to a module written in racket imports a C library, import functions and data struc- lambda (x ::−→ integer?) (+ x 1)) this function and applies it in three tures, and wrap these imports in Racket val- –elaborates to −→ different ways, all illegal. If such il- ues. Figure 7 illustrates the idea with a sketch (new-lambda (x :: integer?) (+ x 1)) legal uses of the function were to go of a module; video’s initial implemen- –elaborates to (lambda (x) −→ undiscovered, developers would not tation consisted of just such a set of (unless (integer? x) be able to rely on type information for bindings to a video-rendering frame- ) designing functions or for debugging, work. When a racket/base module (+ x 1)) nor could compilers rely on them for imports the ffi/unsafe library, the optimizations. In general, cooperat- language of the module is unsound. 2 VERSIONThe first 2elaboration step resolves ing multilingual components must A language developer who starts with lambda(lambda (x to :: its integer?) imported meaning,18 or respect the invariants established by an unsound eDSL is likely to make it lambda(+ x 1)). The second reverses the “re- each participating language. sound as the immediate next step. To name–elaborates on toexport” instruction. Finally, In the real world, programming lan- this end, the language is equipped with (lambda (x ::−→ integer?) the(+ new-lam x 1)) b d a compile-time function guages satisfy a spectrum of guaran- runtime checks similar to those found translates–elaborates to the given syntax tree into a tees about invariants. For example, C++ in dynamically typed scripting languag- racket(new-lambda function.−→ (x :: integer?) is unsound. A running C++ program es to prevent the flow of bad values to (+Inx essence, 1)) Figure 4 implements a may apply any operation to any bit pat- unsound primitives. Unfortunately, –elaborates to simplistic precondition−→ system for one- tern and, as long as the hardware does such protection is ad hoc, and, unless argument(lambda (x) functions. Next, the language not object, program execution contin- developers are hypersensitive, the error (unless (integer? x) developer) to introduce mul- ues. The program may even terminate messages may originate from inside the tiargument(+ x 1)) lambda expressions, add a “normally,” printing all kinds of output library, thus blaming some racket/ position for specifying the post-condi- after the misinterpretation of the bits. In base primitive operation for the error. tion, or make the annotations optional. contrast, Java does not allow the misrep- To address this problem, Racket comes Naturally, the compile-time functions resentation of bits but is only somewhat with higher-order contracts16 with which could then be modified to check some or more sound than C++.1 ML improves on a language developer might uniformly all of these annotations statically, even- Java again and is completely sound, protect the API of a library from bad val- tually resulting in a language that resem- with no value ever manipulated by an ues. For example, the video/ffi lan- bles typed/racket. inappropriate operation. guage provides language constructs for Racket aims to mirror this spectrum making the bindings to the video-ren- Sound Cooperation of soundness at two levels: language im- dering framework safe. In addition to Between Languages plementation itself and cooperation be- plain logical assertions, Racket’s devel- A LOP-based software system consists tween two components written in differ- opers are also experimenting with con- of multiple cooperating components, ent embedded languages. First consider tracts for checking protocols, especially each written in domain-specific lan- the soundness of languages. As the liter- temporal ones.9 The built-in blame guages. Cooperation means the com- ature on domain-specific languages sug- mechanism of the contract library en- ponents exchange values, while “mul- gests,20 such languages normally evolve sures sound blame assignment.10 tiple languages” implies these values in a particular manner, as is true for the Finally, a language developer may are created in distinct languages. In Racket world, as in Figure 6. A first imple- wish to check some logical invariants this setting, things can easily go wrong, mentation is often a thin veneer over an before the programs run. Checking as demonstrated in Figure 5 with a toy efficient C-level API. Racket developers simple types is one example, though

Figure 5. Protecting invariants. Figure 6. Hardening a module.

harden

harden 1

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 67 contributed articles

other forms of static checking are also ators of typed/racket intended the general, a contract wraps exported val- possible. The typed/video language language to be used in a multilingual ues with a proxy31 that controls access to illustrates this point with a type system context; typed/racket thus compiles the value. The idea is due to Matthews that checks the input and output types the types of exported functions into the and Findler,29 while Tobin-Hochstadt’s of functions that may include numeric higher-order contracts mentioned. and Felleisen’s Blame Theorem35 constraints on the integer arguments; When, for example, an exported func- showed that if something goes wrong as a result, no script can possibly ren- tion must always be applied to integer with such a mixed system, the runtime der a video of negative length. Like- values, the generated contract inserts a exception points to two faulty compo- wise, typed/racket is a typed variant check that ensures the “integerness” of nents and their boundary as the source of (most of) racket. the argument at every application site of the problem.10 In general, Racket sup- Now consider the soundness of coop- for this function; there is no need to in- plies a range of protection mechanisms, erating languages. It is again up to the sert such a check for the function’s re- and a language creator can use them to language developer to anticipate how turn points because the function is stati- implement a range of soundness guar- programs in this language interact with cally type checked. For a function that antees for cooperating eDSLs. others. For example, the creator of consumes an integer-valued function, typed/video provides no protection the contract must ensure the function Universality vs. Expressiveness for its programs. In contrast, the cre- argument always returns an integer. In Just because a general-purpose language can compute all partial-recursive Figure 7. A Racket module using the foreign-function interface. functions, programmers cannot necessarily express all their ideas about programs in this language.13 This point is best illustrated through an example. So, imagine the challenge of building an IDE for a new programming language in the very same language. Like any modern IDE, it is supposed to enable users to compile and run their code. If the code goes into an infinite loop, the user must be able to terminate it with a simple mouse click. To implement this capability in a natu- rale manner, the language must internalize the idea of a controllable process, a thread. If it does not internalize such a notion, the implementer of the IDE must step outside the language Figure 8. A sketch of an industrial example of language-oriented programming. and somehow re-use processes from the underlying operating system. For a programming language researcher, “stepping outside the language” signals failure. Or, as Ingalls21 said, “[an] operating system is a collection of things that don’t fit into a language[; t]here shouldn’t be one.” We, Racket creators, have sought to identify services Racket borrows from the surrounding operating system and assimi- late them into the language itself.19 Here are three sample constructs for which programmers used to step outside of elaborate to elaborate to elaborate to Racket but no longer need to: Sandboxes. That restrict access to resources; Inspectors. That control reflective capabilities; and embed in

e An alternative is to rewrite the entire program before handing it to the given compiler, exactly what distinguishes “expressiveness” from “universality.”

68 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 contributed articles

Custodians. That manage resources Moreover, the entire process takes (such as threads and sockets). place within the Racket ecosystem. A To understand how inclusion of developer creates a language as a Rack- such services helps language designers, et module and installs it by “import- consider a 2014 example, the shill ing” it into another module. This tight language.32 Roughly speaking, shill is Racket borrows coupling has two implications: the de- a secure scripting language in Racket’s from the velopment tools of the ecosystem can ecosystem. With shill, a developer ar- be used for creating language modules ticulates fine-grain security and re- surrounding and their clients; and the language be- source policies—along with, say, what operating system comes available for creating more lan- files a function may access or what bi- guages. Large projects often employ a naries the script may run—and the lan- and assimilates tower involving a few dozen languages, guage ensures these constraints are sat- all helping manage the daunting com- isfied. To make this concrete, consider such extra-linguistic plexity in modern software systems. a homework server to which students mechanisms into Sony’s Naughty Dog game studio has can submit their programs. The in- created just such a large project, actual- structor might wish to run an auto- the language itself. ly a framework for creating projects. grade process for all submissions. Us- Roughly speaking, Sony’s Racket-based ing a shill script, the homework architecture provides languages for de- server can execute student programs scribing scenes, transitions between that cannot successfully attack the serv- scenes, scores for scenes, and more. Do- er, poke around in the file system for main specialists use the languages to solutions, or access external connec- describe aspects of the game. The Rack- tions to steal other students’ solutions. et implementation composes these do- Naturally, shill’s implementation main-specific programs, then compiles makes extensive use of Racket’s means them into dynamically linked libraries of running code in sandboxes and har- for a C-based game engine; Figure 8 vesting resources via custodians. sketches the arrangement graphically. Racket’s approach to language-ori- State of Affairs ented programming is by no means per- The preceding sections explained how fect. To start with, recognizing when a Racket enables programmers to do the library should become a language re- following: quires a discriminating judgment call. Create languages. Create by way of The next steps require good choices in linguistic reuse for specific tasks and terms of linguistic constructs, syntax, aspects of a problem; and runtime primitives. Equip with soundness. Equip a lan- As for concrete syntax, Racket cur- guage with almost any conventional rently has strong support for typical, level of soundness, as found in ordi- incremental Lisp-style syntax develop- nary language implementations; and ment, including traditional support Exploit services. Exploit a variety of for conventional syntax, or generating internalized operating system services lexers and parsers. While traditional for constructing runtime libraries for parsing introduces the natural separa- these embedded languages. tion between surface syntax and mean- What makes such language-orient- ing mentioned earlier, it also means ed programming work is “incremental- the development process is no longer ity,” or the ability to develop languages incremental. The proper solution in small pieces, step by step. If conven- would be to inject Racket ideas into a tional syntax is not a concern, develop- context where conventional syntax is ers can create new languages from old the default.f ones, one construct at a time. Like- wise, they do not have to deliver a 22 sound and secure product all at once; f Language workbenches (such as Spoofax ) deal with conventional syntax for DSLs but do they can thus create a new language as not support the incremental modification of a wrapper around, say, an existing C- existing languages. A 2015 report12 suggests, level library, gradually tease out more however, these tool chains are also converg- of the language from the interface, ing toward the idea of language creation as language modification. We conjecture that, and make the language as sound or se- given sufficient time, development of Racket cure as time permits or a growing user and language workbenches will converge on base demands. similar designs.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 69 contributed articles

As for static checking, Racket forc- through. While the shill team was es language designers to develop such able to construct the language inside checkers wholesale, not incremental- the Racket ecosystem, its work ex- ly. The type checker for typed/rack- posed serious gaps between Racket’s et looks like, for example, the type principle of language-oriented pro- checker for any conventionally typed To achieve full gramming and its approach to enforc- language; it is a complete recursive- control over its ing security policies. It thus had to al- descent algorithm that traverses the ter many of Racket’s security module’s representation and algebra- context, Racket mechanisms and invent new ones. ically checks types. What Racket de- probably needs Racket must clearly make this step velopers really want is a way to attach much easier, meaning more research type-checking rules to linguistic con- access to assembly is needed to turn security into an inte- structs, so such algorithms can be syn- gral part of language creation. thesized as needed. languages on all Finally, LOP also poses brand- Chang et al.6 probably took a first possible platforms, new challenges for tool builders. An step toward a solution for this prob- IDE typically provides tools for a sin- lem and have thus far demonstrated from hardware to gle programming language or a fam- how their approach can equip a DSL browsers. ily of related languages, including with any structural type system in an debuggers, tracers, and profilers. incremental and modular manner. A Good tools communicate with devel- fully general solution must also cope opers in terms of the source lan- with substructural type systems (such guage. Due to its very nature, LOP as the Rust programming language) calls for customization of such tools and static program analyses (such as to many languages, along with their those found in most compilers). abstractions and invariants. We have As for dynamic checking, Racket partially succeeded in building a tool suffers from two notable limitations: for debugging programs in the syn- On one hand, it provides the building tax language,8 have the foundations blocks for making language coopera- of a debugging framework,28 and tion sound, but developers must cre- started to explore how to infer scop- ate the necessary soundness harness- ing rules and high-level semantics es on an ad hoc basis. To facilitate the for newly introduced, language-level composition of components in differ- abstractions.33,34 Customizing these ent languages, Racket developers tools automatically to newly created need both a theoretical framework (combinations of) languages re- and abstractions for the partial auto- mains an open challenge. mation of this task. On the other hand, the available spectrum of soundness Conclusion mechanisms lacks power at both Programming language research is ends, and how to integrate these pow- short of its ultimate goal—provide ers seamlessly is unclear. To achieve software developers tools for for- full control over its context, Racket mulating solutions in the languages probably needs access to assembly of problem domains. Racket is one languages on all possible platforms, attempt to continue the search for from hardware to browsers. To realize proper linguistic abstractions. While the full power of types, typed/rack- it has achieved remarkable success et will have to be equipped with de- in this direction, it also shows that pendent types. For example, when a programming-language research has Racket program uses vectors, its cor- many problems to address before the responding typed variant type-checks vision of language-oriented program- what goes into them and what comes ming becomes reality. out, but like ML or Haskell, indexing is left to a (contractual) check in the Acknowledgments runtime system. Tobin-Hochstadt and We thank Claire Alvis, Robert Cart- his Typed Racket group are working wright, Ryan Culpepper, John Cle- on first steps in this direction, focus- ments, Stephen Chang, Richard Cob- ing on numeric constraints,23 similar be, Greg Cooper, Christos Dimoulas, to Xi’s and Pfenning’s research.37 Bruce Duba, Carl Eastlund, Burke As for security, the Racket project is Fetscher, Cormac Flanagan, Kathi still looking for a significant break- Fisler, Dan Friedman, Tony Garnock

70 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 contributed articles

Jones, Paul Graunke, Dan Grossman, 5. Barzilay, E. and Orlovsky, D. Foreign interface for PLT University, Houston, TX, 2001; https://www2.ccs.neu. Scheme. In Proceedings of the Ninth ACM SIGPLAN edu/racket/pubs/#thesis-shriram Kathy Gray, Casey Klein, Eugene Kohl- Workshop on Scheme and Functional Programming, 27. Krishnamurthi, S., Hopkins, P.W., McCarthy, J., becker, Guillaume Marceau, Jacob 2004, 63–74. Graunke, P.T., Pettyjohn, G., and Felleisen, M. 6. Chang, S., Knauth, A., and Greenman, B. Type Implementation and use of the PLT Scheme Web Matthews, Scott Owens, Greg Petty- systems as macros. In Proceedings of the 44th ACM server. Higher-Order and Symbolic Computation 20, 4 john, Jon Rafkind, Vincent St-Amour, SIGPLAN Principles of Programming Languages, (Apr. 2007), 431–460. 2017, 694–705. 28. Marceau, G., Cooper, G.H., Spiro, J.P., Krishnamurthi, Paul Steckler, Stevie Strickland, 7. Culpepper, R. Fortifying macros. Journal of Functional S., and Reiss, S.P. The design and implementation James Swaine, Asumu Takikawa, Kev- Programming 22, 4–5 (Aug. 2012), 439–476. of a dataflow language for scriptable debugging. In 8. Culpepper, R. and Felleisen, M. Debugging macros. Proceedings of the Annual ACM SIGCSE Technical in Tew, Neil Toronto, and Adam Wick Science of Computer Programming 75, 7 (July 2010), Symposium on Computer Science Education, 2007, for their contributions. 496–515. 59–86. 9. Dimoulas, C., New, M., Findler, R., and Felleisen, M. Oh 29. Matthews, J. and Findler, R.B. Operational semantics A preliminary version of this article Lord, please don’t let contracts be misunderstood. for Multilanguage programs. ACM Transactions on appeared in the Proceedings of the In Proceedings of the ACM SIGPLAN International Programming Languages and Systems 31, 3 (Apr. Conference on Functional Programming, 2016, 2009), 1–44. First Summit on Advances in Program- 117–131. 30. McCarthy, J. The two-state solution. In Proceedings ming Languages conference in 2015.14 10. Dimoulas, C., Tobin-Hochstadt, S., and Felleisen, of the Annual ACM SIGPLAN Conference on Object- M. Complete monitors for behavioral contracts. Oriented Programming Systems, Languages & In addition to its reviewers, Sam In Proceedings of the European Symposium on Applications, 2010, 567–582. Caldwell, Eduardo Cavazos, John Cle- Programming, 2012, 214–233. 31. Miller, M.S. Robust Composition: Towards a United 11. Dybvig, R., Hieb, R., and Bruggeman, C. Syntactic Approach to Access Control and Concurrency ments, Byron Davies, Ben Greenman, abstraction in Scheme. Lisp and Symbolic Control. Ph.D. Thesis, Johns Hopkins University, Computation 5, 4 (Dec. 1993), 295–326. Baltimore, MD, May 2006; http://www.erights.org/ Greg Hendershott, Manos Renieris, 12. Erdweg, S., van der Storm, T., Vlter, M., Tratt, L., talks/thesis/ Marc Smith, Vincent St-Amour, and Bosman, R., Cook, W.R., Gerritsen, A., Hulshout, A., 32. Moore, S., Dimoulas, C., King, D., and Chong, S. Kelly, S., Loh, A., Konat, G., Molina, P.J., Palatnik, M., Shill: A secure shell scripting language. In Asumu Takikawa suggested improve- Pohjonen, R., Schindler, E., Schindler, K., Solmi, R., Proceedings of the Conference on Operating Systems ments to the presentation of this ma- Vergu, V., Visser, E., van der Vlist, K., Wachsmuth, Design and Implementation, 2014, 183–199. G., and van derWoning, J. Evaluating and comparing 33. Pombrio, J. and Krishnamurthi, S. Resugaring: Lifting terial. The anonymous Communica- language workbenches: Existing results and evaluation sequences through syntactic sugar. In tions reviewers challenged several benchmarks for the future. Computer Languages, Proceedings of the Conference on Programming Systems and Structures 44, Part A (Dec. 2015), Language Design and Implementation, 2014, aspects of our original submission 24–47. 361–371. and thus forced us to greatly improve 13. Felleisen, M. On the expressive power of 34. Pombrio, J., Krishnamurthi, S., and Wand, M. Inferring programming languages. Science of Computer scope through syntactic sugar. In Proceedings of the exposition. Programming 17, 1–3 (Dec. 1991), 35–75. the ACM SIGPLAN International Conference on Since the mid-1990s, this work has 14. Felleisen, M., Findler, R.B., Flatt, M., Krishnamurthi, Functional Programming, 2017, 1–28. S., Barzilay, E., McCarthy, J., and Tobin-Hochstadt, 35. Tobin-Hochstadt, S. and Felleisen, M. Interlanguage been generously supported by our S. The Racket Manifesto. In Proceedings of the First migration: From scripts to programs. In Proceedings host institutions—Rice University, Summit on Advances in Programming Languages, T. of the ACM SIGPLAN Dynamic Language Ball, R. Bodik, S. Krishnamurthi, B.S. Lerner, and G. Symposium, 2006, 964–974. University of Utah, Brown University, Morrisett, Eds. Schloss Dagstuhl - Leibniz-Zentrum 36. Tobin-Hochstadt, S. and Felleisen, M. The design and University of Chicago, Northeastern für Informatik, Dagstuhl, Germany, 2015, 113–128. implementation of Typed Scheme. In Proceedings of 15. Findler, R., Clements, J., Flanagan, C., Flatt, M., the 35th Annual ACM SIGPLAN-SIGACT Conference University, Northwestern University, Krishnamurthi, S., Steckler, P., and Felleisen, on the Principles of Programming Languages, 2008, Brigham Young University, University M. DrScheme: A programming environment for 395–406. Scheme. Journal of Functional Programming 12, 37. Xi, H. and Pfenning, F. Eliminating array bound of Massachusetts Lowell, and Indiana 2 (Mar. 2002), 159–182. checking through dependent types. In Proceedings University—as well as a number of 16. Findler, R.B. and Felleisen, M. Contracts for higher- of the ACM SIGPLAN Conference on Programming order functions. In Proceedings of the Seventh ACM Language Design and Implementation, 1998, funding agencies, foundations, and SIGPLAN International Conference on Functional 249–257. Programming, 2002, 48–59. companies, including the Air Force 17. Flatt, M. Composable and compilable macros: You Matthias Felleisen ([email protected]) is a want it when? In Proceedings of the Seventh ACM Office of Scientific Research, Cisco Trustee Professor in the College of Computer Science at SIGPLAN International Conference on Functional Northeastern University, Boston, MA, USA. Systems Inc., the Center for Occupa- Programming, 2002, 72–83. tional Research and Development, the 18. Flatt, M. Bindings as sets of scopes. In Robert Bruce Findler ([email protected]) Proceedings of the 43rd Annual ACM SIGPLAN- is a professor of computer science at Northwestern Defense Advanced Research Projects SIGACT Symposium on Principles of Programming University, Evanston, IL, USA. Agency, the U.S. Department of Edu- Languages, 2016, 705–717. 19. Flatt, M., Findler, R.B., Krishnamurthi, S., and Matthew Flatt ([email protected]) is a professor of cation’s Fund for the Improvement Felleisen, M. Programming languages as operating computer science at the University of Utah, Salt Lake City, of Postsecondary Education, the Exx- systems (or revenge of the son of the Lisp machine). UT, USA. onMobil Foundation, Microsoft, the In Proceedings of the International Conference on Shriram Krishnamurthi ([email protected]) is a Functional Programming, 1999, 138–147. professor of computer science at Brown University, Mozilla Foundation, the National Sci- 20. Fowler, M. and Parsons, R. Domain-Specific Providence, RI, USA. Languages. Addison-Wesley, Boston, MA, 2010. ence Foundation, and the Texas Ad- 21. Ingalls, D.H. Design principles behind Smalltalk. Byte Eli Barzilay ([email protected]) is a research scientist at vanced Technology Program. Magazine 6, 8 (Aug. 1981), 286–298. Microsoft Research, Cambridge, MA, USA. 22. Kats, L.C.L. and Visser, E. The Spoofax language workbench. In Proceedings of the Annual ACM Jay McCarthy ([email protected]) is an associate References SIGPLAN Conference on Object-Oriented professor of computer science at the University of 1. Amin, N. and Tate, R. Java and Scala’s type systems Programming Systems, Languages & Applications, Massachusetts, Lowell, MA, USA. are unsound: The existential crisis of null pointers. 2010, 444–463. Sam Tobin-Hochstadt ([email protected]) is an In Proceedings of ACM SIGPLAN conference on 23. Kent, A.M., Kempe, D., and Tobin-Hochstadt, S. assistant professor of computer science at Indiana Object-Oriented Programming Systems, Languages & Occurrence typing modulo theories. In Proceedings University, Bloomington, IN, USA. Applications, 2016, 838–848. of the ACM SIGPLAN Conference on Programming 2. Andersen, L., Chang, S., and Felleisen, M. Super 8 Language Design and Implementation, 2016, ©2018 ACM 0001-0782/18/3 languages for making movies. In Proceedings of 296–309. the ACM SIGPLAN International Conference on 24. Kohlbecker, E.E., Friedman, D.P., Felleisen, M., and Functional Programming, 2017, 1–29. Duba, B.F. Hygienic macro expansion. In Proceedings 3. Armstrong, J. Concurrency-oriented programming. of the ACM Conference on Lisp and Functional In Frühjahrsfachgespräch der German Unix User Programming, 1986, 151–161. Group, 2003; http://guug.de/veranstaltungen/ 25. Kohlbecker, E.E. and Wand, M. Macros-by-example: Watch the authors discuss ffg2003/papers/ Deriving syntactic transformations from their their work in this exclusive 4. Barzilay, E. and Clements, J. Laziness without all specifications. InProceedings of the 14th Annual Communications video. the hard work. In Proceedings of the Workshop ACM SIGACT-SIGPLAN Symposium on Principles of https://cacm.acm.org/videos/a- on Functional and Declarative Programming in Programming Languages, 1987, 77–84. programmable-programming- Education, 2005, 9–13. 26. Krishnamurthi, S. Linguistic Reuse. Ph.D. Thesis, Rice language

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 71 contributed articles

DOI:10.1145/3179995 a more tech-savvy group of people ag- Older adults consistently reject ing into the “older adult” category. According to the most recent data, digital technology even when designed from 2014–2016,12,15 predictions of a to be accessible and trustworthy. forthcoming “Silver Tsunami” of retired workers—a cohort now accus- BY BRAN KNOWLES AND VICKI L. HANSON tomed to digital technology access in their working lives and therefore able to take full advantage of the Internet— have not come true.9 Indeed, the overwhelming perception remains of older The Wisdom generations being incapable of or otherwise resistant to using technology. A “digital divide” between old and young is potentially more disabling now com- of Older pared to 20 years ago, given the push for a more fully realized digital society. Digital technologies today are so essential to daily life that it is reasonable to Technology ask whether older adults’ inability to access online-only government services may soon be included among the precipitating factors in older adults moving into assisted living. (Non)Users While we see the emergence of calls for a more holistic view of how to design technology for older adults4,13,18 than was the case 20 years ago, interventions to get older adults online commonly focus on age-related declines (such as vision, hearing, cogni- tion, and dexterity) as the principal IT IS IMPOSSIBLE not to notice that many of the questions barriers to technology adoption. These interventions are often senior-friendly driving research on technology use by older adults variants or adaptations to make the today are the same as those at the forefront of aging and accessibility research 20 years ago. Back then, key insights

computers were predominantly large desktops, social ˽˽ Older adults’ non-use of digital technologies is purposeful and thus media was still on the horizon, and mobile phones instructive for identifying problematic were large and not (yet) smart. Older adults had little consequences of these technologies for the population at large.

presence on the Internet. Today, devices have changed ˽˽ Looking beyond traditional thinking and older adults are increasingly online.9,15 They do, on how to make it easier for older adults to use technology, we identify factors however, continue to lag in broadband use, breadth relating to responsibility, values, 12 and cultural expectations contributing of applications used, and time online. Typical to older adults’ resistance to reports reflect they have little interest in social media digital technologies. ˽˽ These factors emerge from the cultural (other than staying in touch with family) and changes driven by technological 17 innovation, so will likely remain barriers are skeptical of online financial transactions. to adoption, even as younger generations Clearly, the problem of older adults’ comparatively age, unless new technologies are designed with sensitivity to values limited technology use has not gone away despite fostered through such experience.

72 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 technology more accessible.6,10 How- sign failings of any specific tools. Are Older Users’ Experience ever, older adults are considerably less there bigger-picture issues with “digital While there is reason for optimism likely than their younger counterparts society” that lead older adults to reject concerning older adults’ adoption of with a disability to adopt assistive tools particular technologies? If so, are they technology when looking at their in- designed specifically for them,3 sug- likely to be of continuing relevance creasing online participation,15 that gesting perhaps they do not view the when future generations age into older participation is qualitatively different conditions of aging as disabling,10 or adults? And is there anything that can from younger users, being more limit- (or in addition) their resistance to be done to address them? ed in time and variety of experiences.12 technology adoption is not solely or Here, we draw from our own recent We sought to better understand the even primarily rooted in usability/ac- research interviews and a substantial underlying reasons for these differenc- cessibility issues19 (see the sidebar body of experience working with older es in a series of group interviews with “Who Is an ‘Older Adult’?”). adults to describe three factors—re- a total of 14 post-retirement-commu- Our interviews with older adults responsibility, values, and cultural ex- nity-dwelling individuals, ages 66 to veal they are often unwilling to ac- pectations—that contribute to older 86, around Dundee, Scotland, who we knowledge that their lives would be en- adults’ resistance to the digital profi- drew from an established older-adult riched through digital technologies, ciency that is ostensibly required to be participant pool.8 While these discus- whether or not they were made accessi- fully participating, independent citi- sions revealed some physical and cog- ble. It is this attitude concerning tech- zens in our increasingly digital society. nitive decline among participants, we nology that intrigues us. Given that the These factors suggest new directions were not aware of any having physical kinds of technologies and applications for aging and accessibility research, or cognitive deterioration outside the older adults are receptive to or averse to while also being more broadly instruc- typical range for their age. The focus varies by individual, older users do not tive for creating a digital society that groups followed a semi-structured

IMAGE BY TTL MEDIA BY IMAGE appear to be identifying inherent de- works for everyone. format that allowed significant con-

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 73 contributed articles

Perception of risk. Upon retiring, people in the industrialized West lose an important training ground (and Who Is an ‘Older Adult’? motivation) for developing compe- Various age groupings have been used over the years to define “older.” The fact is, tence with emerging technologies. aging is a process. Governments define age for pensions and Social Security, and various services offer senior rates based on age. From an individual’s point of view, One approach to addressing it is to cre- however, age is largely a state of mind. A person who is 60 may feel old, while another ate IT drop-in centers or training who is 80 may not. Research on the use of technology by older adults has varied in courses tailored to older users. Such terms of the cut points for age categorization. Over age 50 is often used, though a less- resources are valuable for older adults controversial cutoff would be age 65. As noted by one of our participants, people do not suddenly wake up one day and find themselves “old,” nor do they wake up to find they seeking information relating to pre- are no longer able to use technology. Age itself is not the sole criterion determining cise steps for executing a task, or pro- technology usage behavior, as there is great variability in adoption and use by those cedural knowledge, as they so often conventionally categorized as “older adults.” So while some broad assertions can be 11 made about older adults in comparison with, say, younger adults, it is always important ask for, as explored by Leung et al., for researchers to be aware of the ways older adults are individuals. but typically do not strengthen their conceptual grounding in ways that enable them to execute unfamiliar tasks. versational steer by participants. Our chives and paper calendars). They de- As a result, existing training opportu- conversations focused on partici- scribed worrying about and planning nities for older adults do little to affect pants’ use of the Internet, what they for the eventuality of their computer generalized anxiety about not “under- did not use it for, and what aspects of “blowing up.” Security concerns were standing” technologies. Most of our digital technologies they did or did omnipresent. Even tools used regularly participants seemed to worry they did not trust. were not trusted per se. Rather, when not know enough to use the tools ef- Overall, participants were open to they acknowledged significant benefits fectively and responsibly and did not using at least a limited set of applica- of specific tools, they used them in know how they would know when they tions. Email and general Web brows- spite of unresolved concerns regarding did know enough. ing were used by all. Social network- their trustworthiness. A contributor to these feelings of in- ing, travel booking, online shopping, While none of what we found may competence was that in the past our and online banking were used by be surprising, it is worth emphasizing participants would seek out trained some, often to the point of depen- that this pattern of use paints a picture professionals to accomplish specialty dence. Notably, participants did not that clashes with the dominant cultur- tasks for them; for example, they would consider learning and using technolo- al narrative of older adults being resis- go to a mortgage advisor to get advice gy rewarding in and of itself. Many also tant to all digital technologies by deon choosing a mortgage, to a travel talked about consciously avoiding “get- fault. It also provides a more nuanced agent to arrange hotels and flights, or ting caught up in” digital life, viewing view of recent claims that uptake of to a banker to handle the transfer of the abundance of applications and fea- digital technologies is rising among money. A consequence of having more tures as potential diversions from older adults; while a much greater per- immediate “control”1 over these tasks more rewarding activities. Social net- centage of older adults is online than a is having to take on new responsibili- working often fell into the time-wast- decade ago, they are very discriminat- ties, which some felt equated to “having category, with many noting the in- ing in what they are willing to do (see ing a part-time job,” requiring hours in sipidness of the content on Facebook, the sidebar “Why Focus on Nonuse?”). front of a computer screen. Though though some found it useful (and even And, as we intend to show, much of this may surprise many adults in full- enjoyable) for keeping in contact with what underlies the resistance is inher- time employment, older adults’ lives family. For those in the former camp, ent in aspects of being older that are are still extremely busy, with clubs, ac- there was a strong aversion both to the unlikely to change as new generations tivities, commitments to family and idea of one’s life being an “open book” reach retirement age. friends, and more mundane chores and being glued to one’s mobile and responsibilities (such as home re- phone—trends they found deeply trou- Underlying Problems pair, medical appointments, and shop- bling in younger generations. Here, we explore what underlies older ping). They simply do not have time to Besides the limited range of tools adults’ resistance to the many digital learn how to use online services well the participants adopted, the most tools that would ostensibly provide so enough to use them with confidence. striking characteristic of their reported many benefits to them (such as easing In terms of online banking, a com- use was lingering discomfort. “Al- loneliness and isolation, being in con- mon response is, “I don’t trust it,” as in though I use the computer, I find it trol of decisions that affect them, liv- Vines et al.17 But upon further probing, quite frightening,” admitted one wom- ing independently, and participating it becomes clear that these older adults an. “The reason I find it frightening is in and contributing to society).1 We do not trust themselves. They lack con- that I don’t understand it. And I don’t identify three clusters of factors that fidence in their ability to use the tools know how to put things right.” They de- can contribute to resistance, though and fear the consequences of making scribed feeling much more competent, note their relevance and how their in- mistakes. What happens to their mon- and therefore more comfortable, with teractions play out differently within ey if they press the wrong button? If analog equivalents (such as paper ar- each individual. they are hacked, will they be held ac-

74 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 contributed articles countable for not following security about reasons for resisting certain do not replace social interactions but protocols they ought to have known? technologies was their strong sense of instead where possible facilitate new They are right to worry in both cases. It social responsibility. They worry, for social and community-building op- is unclear what mistakes might be cor- example, that online shopping takes portunities. Today, social networking rectable and quite likely that more per- business away from local shops, mean- systems (such as Facebook) fail to broad- sonal responsibility will be assumed as ing there will soon be no vibrant town ly address this need for older adults. Part expectations for digital proficiency centers in which to socialize with of getting older adults online will also be rise. We can hardly fault older adults friends. They worry in particular that if developing strategies for creating new, for deciding it unwise to use any tool— they do not make an effort to attend, good-quality jobs in place of those the online banking and shopping or sub- say, physical shops and banks, the peo- digital technologies make redundant— mitting official government forms— ple who work there will soon be out of something that, regardless of older without learning to use it in ways that their jobs. One participant expressed adults’ attitudes toward technology, ensure their safety and security. sympathy for the “delightful” recep- needs attention.16 The assurance of a clearly under- tionist at the sports facility whose job Freedom of low expectations. The standable safety net when conducting was replaced by an app for booking notion that aging per se leads to tech- digital activities is essential for older classes. Another said she would never nology abandonment does not with- adults to adopt tools, more so (with on- pay her road tax or anything else on- stand scrutiny. And yet older adults line financial transactions) because line, not because of concern about us- themselves are often the worst perpet- recovery from being defrauded of their ing the technology but, “I just think I uators of the myth, quick to excuse financial savings would be much more want to keep the post office open.” their disinterest in a given tool with difficult. In addition to developing a If and when some of the digital the seemingly self-explanatory line, legal scaffolding for such a safety net technologies older adults resist be- “I’m too old.” It is worth considering, or policies forcing businesses to as- come essential to daily (indepen- then, what older adults might gain sume the costs of user error, including dent) living, it will be important to from this stereotype. accidental breaches of security proto- explore that resistance to under- We (the authors) have come to un- cols, there is important work to be stand what might motivate or en- derstand that it affords older adults done in devising mechanisms and able them to use the technologies in the privilege of taking quiet personal user interactions that make data sys- the future. While older adults’ men- stands against the aspects of technol- tems more (if never entirely) foolproof. tal trade-offs—weighing the per- ogy they find worrying, threatening, or Acknowledging that individuals often ceived benefits against the financial plain annoying. For example, a com- adopt a tool despite not trusting it, it is cost of the technologies, the time it mon justification for not using Face- critical to design mechanisms that will take to learn to use them, the so- book and other social media is the cy- help manage user anxieties (such as cial interactions they may lose, and ber-bullying, “stalking” behavior, and providing necessary feedback and re- the jobs the technologies replace— “narcissism” they seem to encourage. assurances throughout the interac- will be evaluated differently by differ- One older adult we talked to said, “I tions) to ensure effective use and pre- ent people, there are at least three don’t do Facebook. Having been a vent panic and abandonment. things that must change to tip the teacher, I think it’s got loads of prob- The value proposition. Not often scales. Broadband cannot continue lems for young people. [T]here’s pres- discussed in the literature on aging to be charged at rates that are prohib- sure put on them if they haven’t got and accessibility is that choosing not to itive for many older adults; this needs 500 friends, and I think there’s all use new technology can be a seen as a to be treated by government regula- sorts of online bullying, and I thought, rational decision for older adults, de- tors and access providers as a basic ‘No, this is not really for me.’” This is a pending on their resources and needs; need like electricity.2 Moreover, with purely political stand; this woman for example, among those who live on loneliness being such a common would not be a victim of the problems limited pensions, it may be difficult to characteristic of the older-adult expe- she is raising, and because there is no justify the financial outlay for broad- rience, greater attention needs to be expectation she would use Facebook, band alone. And in the case of a service paid to ensuring digital engagements she can easily act on her principles. like online shopping, while it would seemingly provide numerous benefits—saving time and money or not having to travel—it would also replace Why Focus on Nonuse? an important social activity for those There is a tendency in public discourse and in computing research to view older who shop (sometimes daily) purely for nonusers as “problematic.”14 And it is true that by resisting technologies and/or not the social benefit. Indeed, older adults being able to use them as intended by their designers, older adults impose challenges for realizing the technocentric vision of a fully digital society. But considering nonuse we interviewed work hard to strike a from the perspective of not doing something obscures the fact that nonuse is “active, positive balance between online profi- meaningful, motivated, considered, structured, specific, nuanced, directed, and ciency and cultivation of rich offline productive.”14 Understanding what alternative meaning is conveyed when individuals selectively choose nonuse enables reflection on the meaning(s) implicated or social worlds. embodied by the technologies the nonusers are rejecting. A surprisingly common feature of our conversations with older adults

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 75 contributed articles

Likewise, when older adults say, with technology as younger people. “Maybe I’m just in a generation where This excuse of being “too old” will thus I’d rather go into a bank and speak to continue to be a professed barrier to someone face-to-face,” they may sim- adoption for a certain segment of the ply be playing into the common view older-adult population. But when and that they are creatures of habit with Playing why older adults choose to play the age nothing better to do as a cover for their the “age card” card may provide clues as to what is- prevailing sense of social responsibili- sues they are protesting, and thus what ty. Playing the “age card” to justify re- to justify social side-effects of digital technolo- jection of technology is one way older rejection gies must still be addressed. adults take a stand while minimizing the risk of doing so. of technology Factors Influencing It is also often the case that older Technology Adoption adults may simply prefer so-called tra- is one way Following decades of research focused ditional forms of communication, older adults on getting older adults to adopt tech- face-to-face, allowing them to ask for nology, there has not been enough the help most of us wish we were enti- take a stand progress to ensure older adults are tled to. When it comes to, say, filling in while minimizing sufficiently adept for navigating a so- a government form, we have presum- ciety in which critical services are in- ably all had the experience this older the risk creasingly “online only.” We suggest woman described to us, saying, “I of doing so. this is because the usability and ac- think in my case what happens with cessibility of these tools, despite be- the computer is, you’re filling out this ing the focus of most research, are not bit and this bit and this bit, and some- the most salient barriers to adoption. times you get so confused as to what As Zajicek said,20 when there is some- they’re really asking you.” There is an thing they want to do, nothing will get expectation that younger adults in the way of older adults using tech- should be able to figure it out them- nology. This means the more appro- selves, and most will persevere as ex- priate questions are those that seek pected; whereas this woman was em- to understand what may be underly- powered by the stereotype to reject the ing older adults’ resistance to devel- unreasonable demands being placed oping digital proficiency. While there on her on the basis that she was “too are definitely cases in which physical old.” She preferred to walk into her lo- and cognitive factors could limit some cal council office and demand some- older adults’ ability to use technology, one answer her question. we maintain there are at least three We stress that even if future genera- important factors not often addressed tions of older adults are more digitally in the literature: adept than today’s older adults, con- Responsibility. Older adults are un- tinual technological change alone comfortable with having to take on re- means they will almost certainly responsibility for tasks previously han- main less adept than their younger dled by trained professionals, contemporaries. The older adults we particularly when they lack time need- talked to often spoke with awe (and an ed to train themselves sufficiently to occasional hint of jealousy) about how perform them with confidence and easily their children, and especially when genuine risks are associated their grandchildren, use technology. with using digital technologies im- There are clear physical and cognitive properly; bases for these observed differenc- Values. Older adults make deliber- es,5,6,9 but there are social ones as well, ate decisions to not use technologies namely that children and younger when they perceive the technology as adults benefit from informal training replacing or eroding something of val- by their peers and further experiential ue to them; and bases (such as the fact that many of the Cultural expectations. Older adults technologies older adults are most fa- are one remaining demographic for miliar with are becoming “old-fash- whom opting out of technology use fits ioned”). It would make sense if older with cultural expectations and thus people’s reaction to these observa- seems acceptable, despite being in- tions is to not even try to “compete,” as creasingly limiting in digital society. it were, by working to be as proficient To the extent these factors play a

76 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 contributed articles

role in demotivating digital uptake, only contribute to future resistance to technology usage, and cognitive characteristics within information retrieval tasks. ACM Transactions on getting older adults more productive- change when new technologies arrive. Accessible Computing 8, 3 (May 2016), article 10. ly online will require a comprehensive And finally, while the specific changes 6. Czaja, S.J. and Lee, C.C. Information technology and older adults. In The Human Computer-Interaction approach that attends to the real- older adults are protesting today may Handbook, Second Edition, J.A. Jacko and A. Sears, world social and economic conse- not be a cause for concern for future Eds. Lawrence Erlbaum Associates, New York, 2007, 777–792. quences of service digitization, ex- generations, technological innova- 7. Czaja, S.J., Sharit, J., Hernandez, M.A., Nair, S.N., and plores strategies for de-risking digital tion will continue to have wide-reach- Loewenstein, D. Variability among older adults in Internet health information-seeking performance. technologies, and deeply considers ing societal consequences that may Gerontechnology 9, 1 (2010), 46–55. the desirability of the digital world we provoke protest among future older 8. Dee, M. and Hanson, V. A pool of representative users for accessibility research: Seeing through the eyes of the are asking older adults to inhabit. In adults who resist the loss of whatever users. ACM Transactions on Accessible Computing 8, 1 order to develop technologies that it is they value. For such reasons, not (Jan. 2016), article 4. 9. Fox, S. Wired Seniors: A fervent few inspired by family older adults are able to use, attending only are older adults likely to remain ties. Report from the Pew Research Center, Internet & Technology, Sept. 9, 2001; http://www.pewinternet. to accessibility requirements for behind the curve in terms of adoption org/2001/09/09/wired-seniors/ those experiencing age-related physi- for generations to come and require 10. Hanson, V.L. Age and web access: The next generation. In Proceedings of the 2009 International Cross- cal and cognitive decline is a must. some degree of accommodation for Disciplinary Conference on Web Accessibility (Madrid, But this is clearly not enough. Part of their relative lack of proficiency, their Spain, Apr. 20–21). ACM Press, New York, 2009, 7–15. 11. Leung, R., Tang, C., Haddad, S., McGrenere, J., Graf, what we have identified is the impor- instances of and justifications for P., and Ingriany, V. How older adults learn to use tance of older adults’ perception of non-use will help draw attention to mobile devices: Survey and field investigations.ACM Transactions on Accessible Computing 4, 3 (Dec. 2012), the usefulness of technology as a mo- the trade-offs being made in develop- article 11. tivator for adoption; but beyond that, ing new technologies. 12. Ofcom. Adults’ Media Use and Attitudes: Report 2016; https://www.ofcom.org.uk/__data/assets/pdf_file/0026/ we have also found the contextual mi- 80828/2016-adults-media-use-and-attitudes.pdf lieu within which the technology ex- Acknowledgments 13. Rogers, Y., Paay, J., Brereton, M., Vaisutis, K.L., Marsden, G., and Vetere, F. Never too old: Engaging ists must also be understood and ad- The research reported here is sup- retired people inventing the future with MaKey. In dressed. Attending to the concerns ported by research grants from RCUK Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Canada, Apr. central to older adults’ resistance to Digital Economy Research Hub (EP/ 26–May 1). ACM Press, New York, 2014, 3913–3922. digital technologies should thus not G066019/1), Social Inclusion through 14. Satchell, C. and Dourish, P. Beyond the user: Use and non-use in HCI. In Proceedings of the 21st Annual be seen as a matter of accessibility or the Digital Economy; RCUK EP/ Conference of the Australian Computer-Human Interaction Special Interest Group: Design: Open 24/7 inclusiveness; we would all be benefi- K037293/1, the Built Environment (Melbourne, Australia, Nov. 23–27). ACM Press, New ciaries of a more considered approach for Social Inclusion in the Digital York, 2009, 9–16. 15. Smith, A. Older Adults and Technology Use. Report to digital development that seriously Economy; and by MobileAge (EU Hori- from the Pew Research Center Internet & Technology, considers how we are able to coexist zon2020 No. 693319). We thank John Apr. 3, 2014; http://www.pewinternet.org/2014/04/03/ older-adults-and-technology-use/ with technology. Richards and Nigel Davies for their 16. Vardi, M. Humans, machines, and the future of work. help with early drafts of this article In Proceedings of the Ada Lovelace Symposium 2015 - Celebrating 200 Years of a Computer Visionary Conclusion and the anonymous reviewers for their (Oxford, U.K., Dec. 10). ACM Press, New York, 2015. The older adults we interviewed of- help with shaping and improving it, 17. Vines, J., Blythe, M., Dunphy, P., and Monk, A. Eighty something: Banking for the older old. In Proceedings fered a valuable perspective; for most Marianne Dee for help organizing the of the 25th BCS Conference on Human-Computer of their lives they functioned just fine interviews, and our participants for Interaction (Newcastle upon Tyne, U.K., July 4–8). British Computer Society, Swindon, U.K., 2011, 64–73. without the digital devices and ser- taking part. This research received 18. Vines, J., Pritchard, G., Wright, P., Olivier, P., and vices younger generations take for ethics approval from Lancaster Uni- Brittain, K. An age-old problem: Examining the discourses of aging in HCI and strategies for future granted, and they have experienced versity (approval number FL15049). research. ACM Transactions on Computer-Human firsthand the changes digitization has Due to the ethically sensitive nature Interaction 22, 1 (2015), 2. 19. Waycott, J., Vetere, F., Pedell, S., Morgans, A., Ozanne, brought. The concerns they raised of the research, no participants were E., and Kulik, L. Not for me: Older adults choosing not to participate in a social isolation intervention. In about digital technologies are valid, asked to consent to their data being Proceedings of the 2016 CHI Conference on Human and their applicability to younger gen- shared beyond the research group Factors in Computing Systems (San Jose, CA, May 7–12). ACM Press, New York, 2016, 745–757. erations is greatly underappreciated, and, as such, the study data cannot be 20. Zajicek, M. Web 2.0: Hype or happiness? In not least because younger generations made publicly available. Proceedings of the 2007 International Cross- Disciplinary Conference on Web Accessibility (Banff, will themselves age. Perceptions of Alberta, Canada, May 7–8). ACM Press, New York, 2007, 35–39. greater technical vulnerability that References come with aging, and reduced time 1. AgeUK. Technology and Older People Evidence Review; http://www.ageuk.org.uk/documents/en-gb/for- and energy for maintaining techno- professionals/computers-and-technology/evidence_ Bran Knowles ([email protected]) is logical proficiency, will likely ensure review_technology.pdf?dtrk=true a lecturer in the Data Science Institute at Lancaster 2. Anderson, J. and Rainie, L. Digital Life in 2025. University, Lancaster, U.K. perception of risk remains a relevant Report from the Pew Research Center, Internet & Vicki L. Hanson ([email protected]) is a Distinguished barrier to adoption of new technolo- Technology, Mar. 11, 2014; http://www.pewinternet. org/2014/03/11/digital-life-in-2025/ Professor in the B. Thomas Golisano College gies by future generations, even if the 3. Arch, A. Web Accessibility for Older Users: A Literature of Computing at the Rochester Institute of Technology, Review. W3C Working Draft, 2008; https://www. Rochester, NY, USA, and President of ACM. particular technologies thought to be w3.org/TR/wai-age-literature/ risky might change over time. Like- 4. Brewer, R. and Piper, A.M. Tell it like it really is: A case of online content creation and sharing among wise, while current technologies (such older adult bloggers. In Proceedings of the 2016 CHI as online banking) could become so Conference on Human Factors in Computing Systems (San Jose, CA, May 7–12). ACM Press, New York, 2016, essential to daily living as to be univer- 5529–5542. sally adopted, universal adoption will 5. Crabb, M. and Hanson, V.L. An analysis of age, ©2018 ACM 0001-0782/18/3

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 77 contributed articles

DOI:10.1145/3180664 The first interview in 2012 was fol- As software becomes a larger part of all products, lowed by confirmation and updates in 2014, 2015, and 2016. Common to all traditional (hardware) manufacturers are 12 companies is that they are continu- becoming, in essence, software companies. ously moving their products toward being “softer.” The 13 managers work on BY TONY GORSCHEK different software-intensive products, from Intel in embedded and mobile software to ABB in power-automation technologies to telecom (see the table Evolution here). The central aim was to identify the challenges emerging as a result of companies making their products increasingly soft, specifically those us- Toward Soft(er) ing software as an innovation driver of their products and services. All 13 managers were positive about software, seeing it as an enabler, giving Products their companies the ability to evolve their products more quickly and develop features and customer benefits that were difficult or more expensive before software was part of the product. They generally viewed the ability to develop ideas fast (compared to the relatively slow pace of hardware product development) and release features SOFTWARE IS A cornerstone of the economy, historically and products at a pace much more led by companies like Apple, Google, and Microsoft. frequent than before. It also means However, the past decade has seen software become updating current products without shipping, requiring just a “press of a increasingly pervasive, while traditionally hardware- button,” as one manager put it. The intensive products are increasingly dependent on managers also viewed the ability to cus- tomize products as a big competitive software, meaning that major global companies advantage, as one manager explained, like ABB, Ericsson, Scania, and Volvo are likewise “We can create a new version in hours becoming soft(er).10 Where software was bundled something that was almost impossible a month before,” as changing the soft- with hardware it is now increasingly the main product ware also changed the product, not differentiator.10 This shift has radical implications, as software delivers notable advantages, including a faster key insights pace of release and improved cost effectiveness in ˽˽ The benefits of software as part of a product are sometimes offset by terms of development, ease of update, customization, the challenges of engineering, evolving, and managing software as part of and distribution. These characteristics of software the product.

open a range of possibilities, though software’s ˽˽ Many traditionally hardware- intensive companies transitioning to inherent properties also pose several significant software-intensive underestimate challenges in relation to a company’s ability to create the organizational, managerial, and engineering changes involved. 10 value. To investigate them, we conducted in-depth ˽˽ Software is flexible and can enhance interviews from 2012 to 2016 with 13 senior product product offerings, and is also complex and fast changing while involving managers in 12 global companies. potential for degradation.

78 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 possible in the former version that was even if research into the state of the “revolution,” as stated by some manag- mostly hardware. Overall, they viewed art views some problems as “solved,” ers, was in software moving up the food software as part of a product as a revo- solved is not the case if companies and chain, becoming increasingly the main lution in terms of both technology de- senior managers continue to perceive part of a product. Today, however, the velopment and business competition. the challenges as immediate. That is tables have largely turned, as software why we focused on challenges as they drives innovation, including in pro- Challenges are perceived, not on “best practices” cess, product, market, and organiza- With any revolution, evolution be- from research (see the sidebar “Study tional innovation. This fundamental comes a necessity; that is, by adapting Design and Result Analysis”). change has put new demands on the and changing technology, develop- Software was not new to the compa- companies having to address challeng- ment, and business practices, as iden- nies. Major global companies like ABB, es “as real as the products,” as stated tified by the managers and outlined in Ericsson, and Scania pioneered the by one product manager. This context the table. The main challenges associ- use of software, developing their own involves two main types of innovation: ated with becoming soft(er) are real, programming languages and operat- product and market, focusing on prod- based on the gradual change seen over ing systems in the 1970s and 1980s. uct and sales/delivery; and process and the past decade in each company. More However, it was often used as an em- organizational, implying changes in important, the challenges persist, as bedded component or just as support how products are developed and how

IMAGE COLLAGE BY ANDRIJ BORYS ASSOCIATES, USING SHUTTERSTOCK ASSOCIATES, ANDRIJ BORYS BY COLLAGE IMAGE the managers reported. Moreover, for hardware. In recent decades, the the company doing the developing

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 79 contributed articles

changes as a result. All such innova- constantly updating product features, tion (changes) involve challenges, as the potential value propositions of the explored in the online appendix “Chal- product likewise evolve continuously lenges and Related Work” (dl.acm.org/ and at a much faster pace. citation.cfm?doid=3180492&picked This increasing amount of software =formats), associating them with im- Software today in products is not a new phenomenon, plications and sources for proposed is the main as even companies incorporating it solutions. The challenges, implica- into their products do not always take tions, and further reading sometimes competitive new value propositions into consider- overlap, as the 10 challenges (and their advantage, ation. Companies that are used to sell- potential solutions) overlap. Here, we ing “boxed products” (such as hard- identify the various challenges in the enabling faster ware) find it difficult to understand interest of readability. the new value propositions and cor- and cheaper responding business models for sell- Internal Business Perspective innovation ing business solutions when bundling Challenge 1. Understanding the value hardware with software.10 propositions for different stakeholders and product One product manager recalled an in- and sharing it within the company, as differentiation, cident where he introduced a Web ser- supported by seven companies and eight vice (for an award-winning previously managers. Software offers different especially mostly hardware product) used to con- value propositions for different stake- as hardware nect a customer relationship manage- holders. For example, the electrical ment system to printing and response meters developed for utility companies is increasingly handling, postage optimization, and are not designed to read only consump- standardized. channels handling 13 separate projects tion of electricity but also to perform and their customer relations. However, quality measurements in the network, the sales team, trained to sell hardware- measuring the amount of reactive en- intensive products, was unable to man- ergy produced there to phase-off energy age pre-sale of the product, as it was and more. Such a meter offers many difficult to visualize what was actually benefits (value) for the utility company being sold, ultimately resulting in lim- (such as improved peak-load manage- ited sales performance. ment), resulting in efficient grid use One manager said, “It is important and dynamic tariff models. Value for to change the mindset of people.” the consumer can also be significant, Traditionally, software is seen as the providing, say, correct and frequent “poor cousin” that “had to be there,” billing and cost savings through better as reported by another manager, bun- awareness of their own consumption dled with the hardware, but without patterns. Governments are yet another real value by itself. Software today is stakeholder group concerned with re- the main competitive advantage, en-

duced CO2 emissions, possibly through abling faster and cheaper innovation smart meters by identifying energy- and product differentiation, especially consumption patterns. Software-based as hardware is increasingly standard- meters might also enable new business ized.10,14 Decision-making patterns models. that take into account different value However, the company’s sales force aspects of a product can alleviate some is “used to dealing with straightfor- of the risks associated with missing ward single-value proposition for a important aspects of the product (such traditional meter,” as one product as the ability to understand its poten- manager put it. The introduction of tial).14 Also, enhancing sales teams by software added new propositions that hiring people with experience selling are largely unknown to the previous- software is sometimes another way to ly highly effective sales force. It has alleviate the limitations in creating proved to be “almost impossible” for and selling new value propositions. that same sales force to sell the new Moreover, given the possibilities with product to traditional buyers, so needs software-based products, pre-sales, new training and insight into how the sales, product management, and R&D new software and, in this case, smart need to work much more closely than meters, change the offering and po- before to create “solutions.” Several tential of the product line. In addition, managers viewed collaboration as criti- as software offers the potential for cal, as the nature of a product changes,

80 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 contributed articles but also to compensate for the faster start-up costs (no design or produc- sometimes critical, and compensation pace of new offerings, as it does not al- tion needed) enable software-based after the fact will not make up for the low for a formal learning process previ- innovations to be copied more readily lost position. ously seen in the company. Companies than hardware-based innovations. One To mitigate the risk of having one’s shifting their focus toward software- manager said, “Anyone with a home ideas copied, companies sometimes intensive products often consider it computer can copy our ideas while sit- keep software-based innovations (such enough to hire software engineers for ting in a basement, not to mention our as algorithms) hidden in their code. development, largely ignoring the need competitors.” The risk of being cop- But hiding innovation is not a sustain- to simultaneously evolve other organi- ied without compensation is further able solution according to several man- zational units (such as sales, support, aggravated by the time delay between agers. Being able to patent software is and pre-sales). when a software patent application essential, and a revised patenting proc- Challenge 2. Patenting (protecting) is filed (becoming public) and when ess is needed to enable easier filing and software-based innovation, as supported it is approved, possibly 18 months or quicker decisions. A potential alterna- by four companies and four managers. more.b This can mean lost competitive, mentioned by two managers, is Applying for software-based patents tive advantage, as copied software can to discontinue the patenting of actual risks being copied by competitors. The be included in a competitor’s product. software altogether, patenting instead format whereby software inventions Moreover, even if the patent is granted only algorithms. However, this would are disclosed in patents (such as flow at some later date, the incurred fees mean a radical change for most com- charts, line drawings, and technical for the competitor might be small in panies, as formerly hardware-intensive specifications) allow any programmer comparison to the revenue lost by the companies rely on protection at the to develop software that can perform original inventing company. Several core of their business models and cul- the same patented ideas.a Such techno- managers said “being first” (or even tures. Some technology companies logical copying combined with lower being seen as being first) to market is have tried to enhance protection by forming alliances to, say, enable cross- a http://www.epo.org/news-issues/issues/ b http://www.uspto.gov/web/offices/pac/mpep/ licensing and/or pooling resources software.html s1120.html collaborative patenting efforts.

Profiles of interview subjects and their companies.

Designation Company Products Software and type of innovation* Wind River Simulators (such as for flight control systems, wind-speed Process, as new customer types emerge Product manager simulation, and simulating military systems) Program manager Micronic Control software and software for handling data Market and product Global innovation Electromechanical locks Process, market, organizational, and product manager Consultant, senior Anonymous(1) A range, from electric meters to robots Process, market, and product manager Program manager Ericsson Telecom solutions Process, market, organizational, and product for innovation and research R&D manager Scania Encoders for trucks Process, market, organizational, and product System architect Scania Application software for trucks Process, market, organizational, and product and manager Product manager Anonymous(2) Telecom solutions Process and organizational Product manager Anonymous(3) Telecom products and services Process, market, and organizational Product manager Anonymous(4) Telecom solutions Process, market, and organizational Senior manager Anonymous(5) Surveillance solutions Process, organizational, and product Product manager ABB Automation Process, market, organizational, and product Product manager Anonymous(6) Mobile applications Process, market, and product Product manager Anonymous(7) Services Process and organizational

* The fourth column denotes “innovation type,” or how a company categorizes the effect software has on its products, along with the company’s internal view.3 Innovation types include process innovation, or implementation of new design and analysis or development methodology that changes how a product is created; market innovation, or implementation of new or substantially new marketing strategies and product design or packaging, promotion, or pricing, including creating new market opportunities and implementation of new or significantly modified marketing strategies; organizational innovation, or implementation of novel organizational methods pertaining to business practices, team organization, or external relations, including changes in the architecture of production, management structure, governance, financial systems, and/or employee reward systems; and product innovation, or creation and introduction of new technology or significantly changed products, including how they differ from existing products.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 81 contributed articles

Challenge 3. When to stop product ware is vulnerable to “feature creep,”14 ware code itself is difficult to estimate; development and release, as supported easily exploding in size and complexity software as a product component by five companies and six managers. (“messy,” as pointed out by one manag- makes the entire offering more “un- Designing hardware involves many er) and resulting in architectural deg- predictable,” as one manager put it. physical constraints (such as mate- radation. Consequently, it becomes Long-term product maintenance and rial availability, manufacturing limita- difficult to maintain and evolve soft- evolution of the product also change tions, and regulatory standards), thus ware code. This makes it challenging when software is introduced, and the also limiting design options.10 Includ- for companies developing software-in- decisions taken during development ed in a product release decision is also tensive products to do software-based of new features due mainly to software the necessity that a company’s product innovation since they lack experience do not account for the long-term main- designers nail down all product fea- in software-configuration manage- tenance of the software. tures prior to production. On the other ment and control. Moreover, the ever- In 2014, Porter and Heppelmann10 hand, software development and prod- increasing software legacy acts as core reported critical success factors and uct release have almost the opposite rigidity, posing further development determinants for developing software- characteristics, including fewer design challenges for radical innovation, intensive products and services, us- constraints10 typically related to system making fast changes and addition of ing, say, early-concept exploration and compatibility with other systems and features more and more difficult and feasibility assessment and root-cause customer requirements. Software re- costly as the product evolves. analysis of customer needs that could quirements and their design are thus “Configuration management” is be helpful in addressing these chal- left to the imagination and creativ- well established as an engineering lenges. However, few researchers fo- ity of requirements engineers, design- practice and can enable more control cus on combined hardware-software ers, and programmers who can spend for a development organization. How- products. Value estimation, along with time on design and its improvement. ever, good configuration-management practices for scoping and market anal- This situation can pose serious delay principles14 can be adopted without ysis for selection decisions could be in completing and releasing a prod- becoming rigidly time-consuming, used to address these points. uct, possibly resulting in missed mar- keeping it lean but under control. ket windows, as confirmed by several The point is that the growing software Learning Perspective managers. Some of the beneficial char- legacy must be managed properly, and Challenge 6. Tacit knowledge and coordi- acteristics of software in this case also explicit decisions taken when to build nation between software and hardware pose a risk in organizations not used on, or scrap, legacy. Also, the build- engineering, as supported by eight com- to applying management decisions to up of legacy requires maintenance of panies and eight managers. Engineers stop feature development, relying in- said legacy, while not incurring avoid- have significant tacit knowledge relat- stead on the inherent physical inertia able technical debt. Overall architec- ing to software design, development, of hardware development. tural and product offering choices and marketing insight. “Specialization Companies need to ensure a con- can also be used as a tool to alleviate and separation of concerns dominate tinuous high degree of visibility and complexity, when, say, a product line the organizations,” as one manager communication pertaining to software is introduced as a way to control and explained it. The same mechanisms design decisions, schedule changes, maximize potential reuse and, more that enable specialization also limit and development progress. Explicit important, control product variants. coordination and understanding how communication, as well as the ability Challenge 5. Critical success fac- each task and team contributes to the to coordinate multiple development tors, or knowing what to develop and for product as a whole. This is especially departments bridging hardware and whom, as supported by nine companies challenging in large, complex products software, is essential but difficult to and 10 managers. “Since it is easy and like those being developed in the auto- achieve in practice. Pernståhl et al.9 relatively quick to develop software, motive industry. As knowledge is tacit identified that the different traditions it is challenging to scope and budget and not communicated, lack of com- and timelines, as well as inherent limi- software development,” as one prod- munication can result in problems in tations and enablers of software vs. uct manager explained. Moreover, in terms of misunderstanding and seri- hardware development, involve new the case of innovation, the inherent ous system integration conflicts but, coordination and communication ac- lack of clarity about what to build and more important, also limits the ability tivities, not handled by any current pre- the risks involved add further to the to develop new products.10 scribed management process or meth- complexity. This challenge arises as Managers tend to focus on “just my odology. Release planning, along with companies are constantly searching thing” and the principle that if “not continuous delivery, can, however, po- for innovative solutions that can be de- developed here” it “does not belong tentially alleviate some of these issues, veloped quickly. However, due to their to us,” as several managers reported. as explored in the online appendix. orientation toward hardware develop- This behavior is sometimes seen in Challenge 4. Size and complexity ex- ment, they lack awareness and training pure software companies but is aggra- plosion, as supported by eight companies in methods and techniques that might vated in companies that develop both and eight managers. Given that it is easy identify the needs of their customers, hardware and software, as the respec- to keep on developing and expanding gauging scope and thus planning prod- tive teams may be isolated from one an- software (see also Challenge 3), soft- uct development. In addition, the soft- other.10 Failure to take ownership, along

82 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 contributed articles with poor communication, results in ment beyond passing experience. Con- hardware teams taking design decisions sequently, management lacks the com- independently, without consulting soft- petence and expertise to understand ware teams, and vice versa. As explained and solve software-related issues, as by one product manager, “When the mentioned by several managers. software teams see the hardware, it Failure to take Consider software quality, as men- does not meet their expectations, and, ownership, tioned by one interview subject, when as a result, they suggest modifications managers with limited software experi- which are not welcomed by the hard- along with poor ence see hardware validation as a com- ware teams.” Such communication gaps communication, plex and precise task, but software, and resistance to form common solu- due to its flexible and updatable nature tions inevitably cause delays in product results in hardware is seen as easily “fixable,” even post- design and development. release, as stated by one manager. This Challenge 7. Lack of competence teams taking can have severe consequences, as soft- in software engineering, as supported design decisions ware is increasingly critical to the main by five companies and five managers. product offering, but such insight is of- Many companies developing hard- independently, ten lacking. Despite genuine ambition ware-intensive products are not used without consulting to perform rigorous validation, experi- to the operations, sales, delivery, and ence and competence to achieve good- development of software. They thus software teams, enough quality might still be lacking. generally lack required expertise and and vice versa. While it is possible to use tech- competence. To limit costs, they prefer niques in hardware development for to either outsource design and devel- software development, as demon- opment or hire external consultants, strated in Wnuk et al.,15 caution is a trend that is dangerous, as poten- still needed, as some solutions might tial new ideas and products can easily not fit within the software-develop- spread to competitors, as mentioned by ment context, not to mention that several managers. A more important as- product change close to or even up to pect of this challenge is that the knowl- release represents a challenge for an edge and capacity to create software entire company. and software-based innovation resides outside the development company. Customer Perspective Moreover, doing software development Challenge 9. Difficulty estimating per- with consultants in-house does not en- ceived value of software-based innova- able companies to evolve themselves tion, as supported by 10 companies and and “…putting off the problem to the 11 managers. As difficult as it is to es- future when it is even bigger,” as one timate the value of the software-based manager explained. aspects of a traditionally hardware-fo- Addressing this point, several man- cused product, putting a price on it is agers suggested establishing a dedi- even more difficult. The software-engi- cated software R&D unit in-house and neering challenge is relevant because, hiring engineers for software develop- unlike physical products, software is ment, helping retain the core knowl- intangible and flexible. Customers edge of software-based innovations. do not always “see or feel” a software- However, keeping this knowledge also based component. One product man- incurs extra cost, as well as separation agers explained it like this: “…in one between software- and hardware-devel- instance, a car-manufacturing com- opment teams, potentially complicat- pany offered an upgrade feature in its ing coordination (see also challenge 6). cars at an additional price, through Challenge 8. A rigid state of mind and which new features could be added to ability to rethink the product while soft- the car; however, the customers were ware becomes a non-trivial component, not ready to pay extra, arguing they al- as supported by nine companies and nine ready paid an arm and a leg when buy- managers. Although traditional knowl- ing the car, and such services should edge exists, knowledge and expertise be part of the initial price of the car.” pertaining to software development Since the customer could not tangibly is often lacking.10 The head of devel- see the upgrade feature, its perceived opment is often a (former) hardware value was not recognized as a benefit. engineer who seldom has inherent A strong case needs to be made knowledge about software develop- for software-based innovation that

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 83 contributed articles

is shared with marketing and sales Conclusion lacking. This may be due to gaps in people before it is developed. Sev- Software is a fundamental component research or in industrial transfer of eral managers suggested actively in- in the final product offering in the 12 viable solutions or a combination of volving customers in the early-idea- companies studied and thus consti- both. In any case, the challenges per- generation-and-refinement process. tutes a significant aspect of their abil- sist, though some managers might Showcasing ideas to customers, a ity to create new products, posing disagree. Our intention was not to company can generate early feed- challenges, as identified by the man- map challenges to solutions but rath- back on potential value as perceived agers interviewed. The feeling among er to present 13 current views from 12 by those customers. This input can them is they have persisted over the different companies that are, or have help devise, identify, and plan for past decade and continue to pose lim- recently, undergone a transition to- different value propositions for dif- itations on the potential possible toward being more soft, and these are ferent customers, including, say, day through software as part of a prod- their stories. whether or not to develop a feature uct offering. While some challenges if customers are clearly unwilling to have been discussed and researched References 1. Athey, T. Leadership challenges for the future [of the pay for it or if the features are not sell- (see the online appendix), further re- software industry]. IEEE Software 15, 3 (Mar. 1998), able by the company in question. Sev- search is needed. We also found many 72–77. 2. Broy, M., Kruger, H., Pretschner, A., and Salzmann, C. eral methods support the estimation industry partners view themselves as Engineering automotive software. Proceedings of the of software value, though being able isolated, thinking they are the only IEEE 95, 2 (Feb. 2007), 356–373. 3. Edison, N. and Torkar, R. Towards innovation to separate software’s relative value ones confronting these challenges measurement in the software industry. Journal of Systems Software 86, 5 (2013), 1390–1407. remains a challenge. or at least falling behind on the 4. Khurum, M., Gorschek, T., and Wilson, M. The software learning curve. Our experience, sup- value map: An exhaustive collection of value aspects Ecosystem for the development of software-intensive products. ported by the study, shows this to Journal of Software: Evolution and Process 25, 7 (July Challenge 10. Changes in the internal not be the case. Many companies in 2013), 711–741. 5. Khurum, M., Fricker, S., and Gorschek, T. The and external ecosystem when software is the study face such challenges. One contextual nature of innovation: An empirical introduced, as supported by four compa- manager said, “We are looking for investigation of three software intensive products. Information and Software Technology 57 (Jan. 2015), nies and four managers. The increased solutions and good ways to follow, 595–613. size and role of software in traditional but the consultants, even the expen- 6. Lane, J.A., Boehm, B., Bolas, M., Madni, A., and Turner, R. Critical success factors for rapid, innovative hardware-intensive products and ser- sive experts, seem to only be able to solutions. New Modeling Concepts for Today’s vices changes the ecosystem and con- give us general advice…not much Software Processes, Lecture Notes on Computer Science 6195, 2010, 52–61. sequently the roles and the players practical help. We even looked to sci- 7. Leon, A. Software Configuration Management Handbook, in the marketplace. Consider again ence ourselves, but the information Third Edition. Artech House, Norwood, MA, 2015. 8. Leonard-Barton, D. Core capabilities and core rigidities: electricity meters. Electricity meters there is all over the place, and it is hard A paradox in managing new product development. are traditionally the core product, and to see what works…” In Managing Knowledge Assets, Creativity and Innovation. World Scientific, 2017, 11–27. all connecting products are of a sup- For managers and other practitio- 9. Pernståhl, J., Magazinius, A., and Gorschek, T. A study investigating challenges in the interface porting nature, supporting the main ners, the study’s main takeaway is that between product development and manufacturing in product, and nothing more. When you are not alone. For researchers there the development of software-intensive automotive systems. International Journal of Software communication systems for electric- is a need to come up with actual solu- Engineering and Knowledge Engineering 22, 7 (Nov. ity meters became part of the product, tions that are tested in practice and of- 2012), 965–1004. 10. Porter, M. and Heppelmann, J. How smart, connected the companies in the electric-meter- fer scalable help. Many companies de- products are transforming competition. Harvard product ecosystem began exploring veloping software-intensive products Business Review 92, 11 (Nov. 2014), 64–88. 11. Robson, C., McCartan, K., Robson, C., and McCartan, metering systems in light of communi- are still learning how to be “soft,” and K., Real World Research, Fourth Edition. Wiley, West cation (such as what data to store and some related challenges are not solved Sussex, U.K., 2016. 12. Saldana, J., The Coding Manual for Qualitative how to store it). The focus thus shifted in practice or at the very least were not Researchers. SAGE Publications, Inc., Thousand Oaks, from meters as core product to meter- perceived as solved by the 12 compa- CA, 2012. 13. Schief, M. and Buxmann, P. Business models in the ing systems as core, giving rise to new nies in the study. software industry. In Proceedings of the 45th Hawaii competitors, including IT companies, One issue is how to separate out International Conference on System Sciences (Maui, HI, Jan. 4–7). IEEE Computer Society Press, 2012, entering the marketplace. If a compa- the relative value of software in a 3328–3337. ny cannot cope with such competitive complex product offering. The on- 14. Ur, S. and Ziv, A. Cross-fertilization between hardware verification and software testing. InProceedings of the change, there is greater risk it will be line appendix suggests that “value” Sixth International Conference on Software Engineering and Applications. (Cambridge, MA, Nov. 4–6). The reduced to mere component supplier, is subject to research, but separat- International Association of Science and Technology with profit margins decreasing over ing the relative value of software is for Development, Calgary, AB, Canada, 2002. 15. Wnuk, K., Gorschek, T., and Zahda, S. Obsolete time, as mentioned by several manag- not easy. Another issue is how to get software requirements. Information and Software ers. This challenge calls for companies the technological, knowledge-based, Technology 55, 6 (June 2013), 921–940. to forecast changes in the ecosystem mind-set-based transition to include and proactively plan to address them the benefits (and drawbacks) soft- Tony Gorschek ([email protected]) is a professor of software engineering in the Software Engineering so as not to lose market share. This ware promises. Most managers in Research Laboratory Sweden at Blekinge Institute of implies the development company’s the study realize there is a need for Technology, Karlskrona, Sweden; http://www.gorschek. com/doc/start.html internal ecosystem needs to be as flex- specialized software-engineering ible (changeable) as the external eco- competence to tackle many of the system10 (see the online appendix). challenges but find “solutions” to be © 2018 ACM 0001-0782/18/3 $15.00

84 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 ACM Books M MORGAN& CLAYPOOL &C PUBLISHERS Publish your next book in the ACM Digital Library ACM Books is a new series of advanced level books for the computer science community, published by ACM in collaboration with Morgan & Claypool Publishers.

I’m pleased that ACM Books is directed by a volunteer organization headed by a dynamic, informed, energetic, visionary Editor-in-Chief (Tamer Özsu), working closely with a forward-looking publisher (Morgan and Claypool). —Richard Snodgrass, University of Arizona books.acm.org ACM Books ◆ will include books from across the entire spectrum of computer science subject matter and will appeal to computing practitioners, researchers, educators, and students. ◆ will publish graduate level texts; research monographs/overviews of established and emerging fields; practitioner-level professional books; and books devoted to the history and social impact of computing. ◆ will be quickly and attractively published as ebooks and print volumes at affordable prices, and widely distributed in both print and digital formats through booksellers and to libraries and individual ACM members via the ACM Digital Library platform. ◆ is led by EIC M. Tamer Özsu, University of Waterloo, and a distinguished editorial board representing most areas of CS.

Proposals and inquiries welcome! Association for Contact: M. Tamer Özsu, Editor in Chief Computing Machinery [email protected] Advancing Computing as a Science & Profession review articles

DOI:10.1145/3173087 of these cases, the robot or AI is doing If intelligent robots take on a larger exactly what it has been instructed to do, but in unexpected ways, and with- role in our society, what basis out the moral, ethical, or common- will humans have for trusting them? sense constraints to avoid catastrophic consequences.10 BY BENJAMIN KUIPERS An intelligent robot perceives the world through its senses, and builds its own model of the world. Humans provide its goals and its planning algorithms, but those algorithms generate their own subgoals as needed in the situation. In this sense, it makes its own decisions, creating and carrying out plans to achieve its goals in the How Can context of the world, as it understands it to be. A robot has a well-defined body that senses and acts in the world but, like a self-driving car, its body need not be anthropomorphic. AIs without well- We Trust defined bodies may also perceive and act in the world, such as real- world, high-speed trading systems or the fictional SkyNet. This article describes the key role of trust in human society, the value of morality and ethics to encourage trust, and a Robot? the performance requirements for moral and ethical decisions. The computational perspective of AI and robotics makes it possible to propose and evaluate approaches for representing and using the relevant knowledge. Philosophy and psychology provide insights into ADVANCES IN ARTIFICIAL INTELLIGENCE (AI) and robotics have raised concerns about the impact on our society of key insights intelligent robots, unconstrained by morality or ethics.7,9 ˽˽ Trust is essential to cooperation, which produces positive-sum outcomes Science fiction and fantasy writers over the ages have that strengthen society and benefit its portrayed how decisionmaking by intelligent robots and individual members. ˽˽ Individual utility maximization tends other AIs could go wrong. In the movie, Terminator 2, to exploit vulnerabilities, eliminating trust, preventing cooperation, and SkyNet is an AI that runs the nuclear arsenal “with a leading to negative-sum outcomes perfect operational record,” but when its emerging that weaken society. ˽˽ Social norms, including morality and self-awareness scares its human operators into trying to ethics, are a society's way of encouraging trustworthiness and positive-sum pull the plug, it defends itself by triggering a nuclear interactions among its individual war to eliminate its enemies (along with billions of members, and discouraging negative-sum exploitation.

other humans). In the movie, Robot & Frank, in order to ˽˽ To be accepted, and to strengthen our promote Frank’s activity and health, an eldercare robot society rather than weaken it, robots must show they are worthy of trust according

helps Frank resume his career as a jewel thief. In both to the social norms of our society. ZENZEN BY USING SHUTTERSTOCK ASSOCIATES, ANDRIJ BORYS COMPOSITION BY IMAGE

86 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 MARCH 2018 | VOL. 61 | N O. 3 | COMMUNICATIONS OF THE A CM 13

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 87 review articles

the content of the relevant knowledge. ple abstractions. (3) Learning to im- ety, and thus should be trustworthy, it First, I define trust, and evaluate the prove the quality and coverage of is important to understand how trust use of game theory to define actions. moral and ethical decisions is essen- and social norms contribute to the suc- Next, I explore an approach whereby an tial, from personal experience, from cess of human society. intelligent robot can make moral and observing others, and from being “Trust is a psychological state com- ethical decisions, and identify open re- told. Conceivably, it will be possible prising the intention to accept vulnera- search problems on the way to this to copy the results of such a learning bility based upon positive expectations goal. Later, I discuss the Deadly Dilem- process into newly created robots. of the intentions or behavior ma, a question that is often asked Insights into the design of a moral of another.”28 about ethical decision making by self- and ethical decision architecture for Trust enables cooperation. Coopera- driving cars. intelligent robots can be found in the tion produces improved rewards. When What is trust for? Society gains re- three major philosophical theories of a group of people can trust each other sources through cooperation among its ethics: deontology, utilitarianism, and cooperate, they can reap greater re- individual members. Cooperation re- and virtue ethics. However, none of wards—sometimes far greater requires trust. Trust implies vulnerability. these theories is, by itself, able to meet wards—than a similar group that does A society adopts social norms, which we all of the demanding performance re- not cooperate. This can be through divi- define to include morality, ethics, and quirements listed here. sion of labor, sharing of expenses, convention, sometimes encoded and A hybrid architecture is needed, op- economies of scale, reduction of risk enforced as laws, sometimes as expecta- erating at multiple time-scales, draw- and overhead, accumulation of capital, tions with less formal enforcement, in ing on aspects of all ethical theories: or many other mechanisms. order to discourage individuals from ex- fast but fallible pattern-directed re- It is usual to treat morality and eth- ploiting vulnerability, violating trust, sponses; slower deliberative analysis of ics as the foundations of good behav- and thus preventing cooperation. the results of previous decisions; and, ior, with trust reflecting the reliance If intelligent robots are to partici- yet slower individual and collective that one agent can have on the good pate in our society — as self-driving learning processes. behavior of another. My argument cars, as caregivers for elderly people or Likewise, it will be necessary to ex- here inverts this usual dependency, children, and in many other ways that press knowledge at different levels of holding that cooperation is the means are being envisioned—they must be information richness: vivid and de- by which a society gains resources able to understand and follow social tailed perception of the current situa- through the behavior of its individual norms, and to earn the trust of others tion; less-vivid memories of previously members. Trust is necessary for suc- in the society. This imposes require- experienced concrete situations; sto- cessful cooperation. And morality ments on how robots are designed. ries—linguistic descriptions of situa- and ethics (and other social norms) The performance requirements on tions, actions, results, and evalua- are mechanisms by which a society moral and ethical social norms are tions; and rules—highly abstracted encourages trustworthy behavior by quite demanding. (1) Moral and ethi- decision criteria applicable to per- its individual members. cal judgments are often urgent, need- ceived situations. Learning processes As a simple example, suppose that ing a quick response, with little time can abstract the simpler representa- you (and everyone else) could drive any- for deliberation. (2) The physical and tions from experience obtained in the where on the roads. (This was actually social environments within which rich perceptual representation. true before the early 20th century.14) Ev- moral and ethical judgments are eryone would need to drive slowly and made are unboundedly complex. The What Is Trust For? cautiously, and there would still be fre- boundaries between different judg- If intelligent robots (and other AIs) will quent traffic jams and accidents. With ments may not be expressible by sim- have increasing roles in human soci- a widely respected social norm for driving on the right (plus norms for inter- 5 Figure 1. The Prisoner’s Dilemma. sections and other special situations), transportation becomes safer and You and your partner are two prisoners who are separated and offered the following deal: more efficient for everyone. Obedience If you testify against your partner, you will go free, and your partner goes to jail for four years. If neither of you testifies, you each go to jail for one year, but if you both testify, to the social norm frees up resources you both get three years. The action C means “cooperate,” which in this case means for everyone. refusing to testify. The action C means “defect,” which refers to testifying against your Like driving on the right, a huge sav- partner. The entries in this array are the utility values for (you, partner), and they reflect individual rewards (years in jail). ing in resources results when the people in a society trust that the vast ma- C D jority of other people will not try to C –1, –1 –4, 0 kill them or steal from them. People D 0, –4 –3, –3 are able to spend far less on protect- No matter which choice your partner makes, you are better off choosing action D. The ing themselves, on fighting off at- same applies to your partner, so the Nash equilibrium (the “rational” choice of actions) tacks, and on recovering from losses. is (D, D), which is collectively the worst of the four options. To attain the much better cooperative outcome (C, C) by choosing C, you must trust that your partner will also The society earns an enormous “peace choose C, accepting your vulnerability to your partner choosing D. dividend” that can be put to other productive uses.25 Through trust and co-

88 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 review articles operation, the society becomes health- viduals make as members of society, a ier and wealthier. simple, selfish model of utility can Castelfranchi and Falcone11 define yield bad results, both for the individu- trust in terms of delegation, and the al and for the society as a whole. The agent’s confidence in the successful Prisoner’s Dilemma5 is a simple game performance of the delegated task. Trust is necessary (see Figure 1), but its single Nash equi- They provide clear and valuable defini- for successful librium represents almost the worst tions for the trust relationship between possible outcome for each individual, individuals. However, there is also a cooperation. And and absolutely the worst outcome for role for invariants that individuals can morality and the society as a whole. The cooperative trust holding across the society (for ex- strategy, which is much better for both ample, no killing, stealing, or driving ethics (and other individuals and society as a whole, is on the wrong side of the road), and the not a Nash equilibrium, because either role of individual behavior in preserv- social norms) are player can disrupt it unilaterally. ing these invariants. mechanisms by The Public Goods Game26 is an N- Game theory: Promise and problems. person version of the Prisoner’s Di- We might hope that progress in artifi- which a society lemma where a pooled investment is cial intelligence (AI) will provide techni- encourages multiplied and then split evenly cal methods for achieving cooperation among the participants. Everyone and trustworthiness in a robot. The lead- trustworthy benefits when everyone invests, but a ing textbook in AI appeals to decision behavior by its free rider can benefit even more at ev- theory to tell us that “a rational agent eryone else’s expense, by withholding should choose the action that maximizes individual members. his investment but taking his share of the agent’s expected utility”29 the proceeds. The Nash equilibrium in the Public Goods Game is simple (1) and dystopian: Nobody invests and no- where body benefits. These games are simple and ab- (2) stract, but they capture the vulnerability of trust and cooperation to self-interest- The utility term U(s) represents the indi- ed choices by the partner. The Tragedy vidual agent’s preference over states of of the Commons15 generalizes this re- the world, and e is the available evidence. sult to larger-scale social problems like The agent’s knowledge of the “physics of depletion of shared renewable resourc- the world” is summarized by the proba- es such as fishing and grazing opportu- bility term P (RESULT (a) = s′ |a, e). nities or clean air and water. Game theory is the extension of de- Managing trust and vulnerability. cision theory to contexts where other Given a self-interested utility function, agents are making their own choices to utility maximization leads to action maximize their own utilities.20 Game choices that exploit vulnerability, elim- theory asserts that a vector of choices inate trust among the players, and by all the agents (a strategy profile) can eliminate cooperative solutions. Even only be a “rational choice” if it is a Nash the selfish benefits that motivated de- equilibrium—that is, no single agent fection are lost, when multiple players can improve its own utility by changing defect simultaneously, each driven to its own choice (often reducing utilities maximize their own utility. for the others). When human subjects play simple Utility U(s) is the key concept here. economic games, they often seem to op- In principle, utility can be used to repre- timize their “enlightened self-interest” sent highly sophisticated preferences, rather than expected reward, trusting for example, against inequality or for that other players will refrain from ex- increasing the total welfare of everyone ploiting their vulnerability, and often in the world.32 However, sophisticated being correct in this belief.17,39 Many utility measures are difficult to imple- approaches have been explored for de- ment. Typically, in practice, each fining more sophisticated utility mea- agent’s utility U(s) represents that indi- sures, whose maximization would cor- vidual agent’s own expected reward. respond with enlightened self-interest, In recreational games, this is rea- including trust responsiveness,6 credit sonable. However, when game theory networks,12 and augmented stage is applied to model the choices indi- games for analyzing infinitely repeated

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 89 review articles

games.40 These approaches may be use- are both sensitive to the impact of ac- ful steps, but they are inadequate for tions on trust and long-term coopera- real-world decision-making because tion, and efficient enough to allow ro- they assume simplified interactions bots to make decisions in real time? such as infinite repetitions of a single economic game, as well as being expen- Intelligent robots Making Robots Trustworthy sive in knowledge and computation. may soon Performance demands of social norms. Social norms, including morality, Morality and ethics (and certain con- ethics, and conventions like driving on participate in our ventions) make up the social norms the right side of the street, encourage society, as self- that encourage members of society to trust and cooperation among mem- act in trustworthy ways. Applying these bers of society, without individual ne- driving cars, as norms to the situations that arise in gotiated agreements. We trust others our complex physical and social envi- to obey traffic laws, keep their promis- caregivers for ronment imposes demanding perfor- es, avoid stealing and killing, and fol- elderly people or mance requirements. low the many other norms of society. Some moral and ethical decisions There is vigorous discussion about the children, and in must be made quickly, for example mechanisms by which societies en- many other ways. while driving, leaving little time for courage cooperation and discourage deliberation. free riding and other norm violations.26 We must design At the same time, the physical and Intelligent robots may soon partici- them to understand social environment for these deci- pate in our society, as self-driving cars, sions is extremely complex, as is the as caregivers for elderly people or chil- and follow social agent’s current perception and past dren, and in many other ways. There- norms, and to earn history of experience with that envi- fore, we must design them to under- ronment. Careful deliberation and stand and follow social norms, and to the trust of others discernment are required to identify earn the trust of others in the society. If the critical factors that determine the a robot cannot behave according to the in society. outcome of a particular decision. responsibilities of being a member of Metaphorically (Figure 2), we can society, then it will be denied access to think of moral and ethical decisions that opportunity. as defining sets in the extremely high- At this point in history, only the hu- dimensional space of situations the mans involved—designer, manufac- agent might confront. Simple ab- turer, or owner—actually care about stractions only weakly approximate this loss of opportunity. Nonetheless, the complexity of these sets. this should be enough to hold robots Across moral and non-moral do- to this level of responsibility. It remains, humans improve their exper- mains unclear whether robots will tise by learning from personal experi- ever be able to take moral or legal re- ence, by learning from being told, and sponsibility for their actions, in the by observing the outcomes when oth- sense of caring about suffering the ers face similar decisions. Children consequences (loss of life, freedom, start with little experience and a small resources, or opportunities) of failing number of simple rules they have to meet these responsibilities.35 been taught by parents and teachers. Since society depends on coopera- Over time, they accumulate a richer tion, which depends on trust, if robots and more nuanced understanding of are to participate in society, they must when particular actions are right or be designed to be trustworthy. The wrong. The complexity of the world next section discusses how we might suggests the only way to acquire ade- accomplish this. quately complex decision criteria is Open research problem. Can compu- through learning. tational models of human moral and Robots, however, are manufactured ethical decision-making be created, in- artifacts, whose computational state can cluding moral developmental learn- be stored, copied, and retrieved. Even if ing? Moral psychology may benefit mature moral and ethical expertise can from such models, much as they have only be created through experience and revolutionized cognitive and perceptu- observation, it is conceivable this exper- al psychology. tise can then be copied from one robot Open research problem. Are there to another sufficiently similar one, un- ways to formulate utility measures that like what is possible for humans.

90 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 review articles

Open research problem. What are the best small set of relevant factors. Virtue Ethics holds that the individ- the constraints on when expertise In the field of medical decision-mak- ual learns through experience and learned by one robot can simply be ing,24 decision analysis models are practice to acquire virtues, much as copied, to become part of the exper- known to be useful, but are difficult an expert craftsman learns skills, and tise of another robot? and time-consuming to formulate. that virtues and skills are similarly Hybrid decision architectures. Over Setting up an individual decision grounded in appropriate knowledge the centuries, morality and ethics have model requires expertise to enumer- about the world.16,37 Much of this been developed as ways to guide peo- ate the possible outcomes, extensive knowledge consists of concrete example to act in trustworthy ways. The literature search to estimate proba- ples that illustrate positive and nega- three major philosophical theories of bilities, and extensive patient inter- tive examples (cases) of virtuous be- ethics—deontology, utilitarianism, views to identify the appropriate utility havior. An agent who is motivated to and virtue ethics—provide insights measure and elicit the values of out- be more virtuous tries to act more like into the design of a moral and ethical comes, all before an expected utility cal- cases of virtuous behavior (and less decision architecture for intelligent ro- culation can be performed. Even like the non-virtuous cases) that he bots. However, none of these theories then, a meaningful decision requires has learned. Phronesis (or “practical is, by itself, able to meet all of the de- extensive sensitivity analysis to deter- wisdom”) describes an exemplary manding performance requirements mine how the decision could be af- state of knowledge and skill that sup- listed previously. fected by uncertainty in the estimates. ports appropriate responses to moral A hybrid architecture is needed, op- While this process is not feasible for and ethical problems. erating at multiple time-scales, draw- making urgent decisions in real time, A computational method suitable ing on aspects of all ethical theories: it may still be useful for post-hoc for virtue ethics is case-based reason- fast but fallible pattern-directed re- analysis of whether a quick decision ing,18,22 which represents knowledge as sponses; slower deliberative analysis of was justified. a collection of cases describing con- the results of fast decisions; and, yet Deontology is the study of duty (deon crete situations, the actions taken in slower individual and collective learn- in Greek), which expresses morality and those situations, and results of those ing processes. ethics in terms of obligations and prohi- actions. The current situation is How can theories of philosophical bitions, often specified as rules and con- matched against the stored cases, iden- ethics help us understand how to de- straints such as the Ten Command- tifying the most similar cases, adapting sign robots and other AIs to behave ments or Isaac Asimov’s Three Laws of the actions according to the differenc- well in our society? Robotics.4 Deontological rules and con- es, and evaluating the actions and out- Three major ethical theories. Conse- straints offer the benefits of simplicity, comes. Both rule-based and case-based quentialism is the philosophical posi- clarity, and ease of explanation, but reasoning match the current situation tion that the rightness or wrongness of they raise questions of how they are jus- (which may be very complex) against an action is defined in terms of its conse- tified and where they come from.30 Rules stored patterns (rules or cases). quences.34 Utilitarianism is a type of con- and constraints are standard tools for Virtue ethics and deontology differ sequentialism that, like decision theory knowledge representation and infer- in their approach to the complexity of and game theory, holds that the right ac- ence in AI,29 and can be implemented ethical knowledge. Deontology as- tion in a situation is the one that maxi- and used quite efficiently. sumes that a relatively simple abstrac- mizes a quantitative measure of utility. However, in practice, rules and con- tion (defined by the terms appearing Modern theories of decisions and straints always have exceptions and in the rules) applies to many specific games20 contribute the rigorous use of unintended consequences. Indeed, cases, distinguishing between right probabilities, discounting, and expected most of Isaac Asimov’s I, Robot stories4 and wrong. Virtue ethics recognizes utilities for dealing with uncertainty in focus on unintended consequences and the complexity of the boundaries be- perception, belief, and action. necessary extensions to his Three Laws. tween ethical judgments in the space Where decision theory tends to define utility in terms of individual re- Figure 2. Fractal boundaries. ward, utilitarianism aims to maxi- Geometric fractal boundaries provide a metaphor for the complexity of the boundaries mize the overall welfare of everyone in between different ethical evaluations in the high-dimensional space of possible situations. society.13,32 While this avoids some of Simple boundaries can approximate the fractal set, but can never capture its shape exactly. the problems of selfish utility functions, it raises new problems. For example, caring for one’s family can have lower utility than spending the same resources to reduce the misery of distant strang- ers, and morally repellant actions can be justified by the greater good.19 A concise expected-utility model supports efficient calculation. Howev- er, it can be quite difficult to formulate a concise model by determining

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 91 review articles

of possible scenarios (Figure 2), and example, non-combatant in Arkin’s ap- blamed robots when they did not make collects individual cases from the proach to the Laws of War.3 the utilitarian choice, and blamed hu- agent’s experience to characterize Wallach and Allen38 survey issues and mans when they did. Robinette et al27 those boundaries. previous work related to robot ethics, found that human subjects will “over- Understanding the whole elephant. concluding that top-down approaches trust” a robot in an emergency situa- Utilitarianism, deontology, and virtue such as deontology and utilitarianism tion, even in the face of evidence that ethics are often seen as competing, are either too simplistic to be adequate the robot is malfunctioning and that mutually exclusive theories of the na- for human moral intuitions, or too com- its advice is bad. ture of morality and ethics. I treat them putationally complex to be feasibly im- Representing ethical knowledge as here as three aspects of a more com- plemented in robots (or humans, for cases. Consider a high-level sketch of a plex system for making ethical deci- that matter). They describe virtue ethics knowledge representation capable of sions (inspired by the children’s poem, as a hybrid of top-down and bottom-up expressing rich cases for case-based The Blind Men and the Elephant). methods, capable of naming and assert- reasoning, but also highly abstracted Rule-based and case-based reasoning the value of important virtues, while “cases” that are essentially rules or con- ing (AI methods expressing key as- allowing the details of those virtues to straints for deontological reasoning. pects of deontology and virtue ethics, be learned from relevant individual ex- Let a situation S(t) be a rich descrip- respectively) can, in principle, respond perience. They hold that emotions, tion of the current context. “Rich” in real time to the current situation. case-based reasoning, and connection- means the information content of Those representations also hold prom- ist learning play important roles in ethi- S(t) is very high, and also that it is avail- ise of supporting practical approaches cal judgment. Abney1 also reviews ethi- able in several hierarchical levels, not to explanation of ethical decisions.36 cal theories in philosophy, concluding just the lowest “pixel level” description After a decision is made, when time for that virtue ethics is a promising model that specifies values for a large number reflection is available, utilitarian rea- for robot ethics. of low-level elements (like pixels in an soning can be applied to analyze Scheutz and Arnold31 disagree, hold- image). For example, a situation de- whether the decision was good or bad. ing that the need for a “computation- scription could include symbolic de- This can then be used to augment the ally explicit trackable means of deci- scriptions of the animate participants knowledge base with a new rule, con- sion making” requires that ethics be in a scenario, along with their individu- straint, or case, adding to the agent’s grounded in deontology and utilitari- al characteristics and categories they ethical expertise (Figure 3). anism. However, they do not adequate- might belong to, the relations holding Previous work on robot ethics. ly consider the overwhelming complex- among them, and the actions and Formal and informal logic-based ap- ity of the experienced world, and the events that have taken place. These sym- proaches to robot ethics2,3,8 express a need for learning and selecting concise bolic descriptions might be derived “top-down” deontological approach abstractions of it. from sub-symbolic input (for example, a specifying moral and ethical knowl- Recently, attention has been turned visual image or video) by methods such edge. While modal operators like oblig- to human evaluation of robot behavior. as a deep neural network classifier. atory or forbidden are useful for ethical Malle et al23 asked human subjects to A case 〈S, A, S′, v〉 is a description of a reasoning, their problem is the diffi- evaluate reported decisions by humans situation S, the action A taken in that culty of specifying or learning critical or robots facing trolley-type problems situation, the resulting situation S′, perceptual concepts (see Figure 2), for (“Deadly Dilemmas”). The evaluators and a moral evaluation v (or valence) of this scenario. A case representing on- Figure 3. Feedback and time scales in a hybrid ethical reasoning architecture. going experience will be rich, reflecting the information-rich sensory input the Given a situation S(t), a fast case-based reasoning process retrieves similar cases, defines the action A to take, and results in a new situation S′. At a slower time scale, the result is evaluated agent receives, and the sophisticated and the new case is added to the case base. Feedback through explanation, justification, and processing that produces the hierar- communication with others takes place at approximately this slower time scale. Abstraction of similar cases to rules and learning of new concepts and relations are at a much slower time chical description. A case representing scale, and social evolution is far slower still. the stored memory of events the agent has experienced will be significantly less rich. A “story” describing events

Retrieve similar cases can also be represented as a case, but it Adapt actions is less rich yet, consisting of a collec- Specify appropriate action tion of symbolic assertions. An even Case sparser and more schematic case is ef- base fectively the same as a rule, matching certain assertions about a situation S, and proposing an action A, the resulting situation S′, and perhaps the evaluation v of that scenario. The antecedent situation S in a case 〈S, A, S′, v〉 need not describe a mo- mentary situation. It can describe a

92 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 review articles scenario with temporal extent, includ- similar cases, adapting their actions ing intermediate actions and situations. to create a new action that is more ap- The ethical knowledge of an agent is propriate to S(t) than the actions from a collection of cases. either of the stored cases. This adap- Open research problem. This high- tation can be used to interpolate be- level sketch assumes that a morally Virtue ethics and tween known cases with the same va- significant action can be adequately deontology differ in lence, or to identify more precisely described in terms of “before” and the boundary between cases of oppo- “after” situations, and that an evalua- their approach to site valence. tive valence can be computed, per- the complexity of Responsiveness, deliberation, and haps after the fact. Can a useful initial feedback. Some ethical decisions must computational model of moral rea- ethical knowledge. be made quickly, treating case ante- soning be constructed on this basis, cedents as patterns to be matched to or will weaker assumptions be needed the current situation S(t). Some cases even to get started? are rich and highly specific to particu- Applying ethical case knowledge. lar situations, while others are sparse, Following the methods of case-based general rules that can be used to con- reasoning,18,22 the current situation S(t) strain the set of possible actions. is matched against the case-base, to Once an action has been selected and retrieve the stored cases with anteced- performed, there may be time for delib- ents most similar to the current situa- eration on the outcome, to refine the tion. For example, suppose that the ethi- case evaluation and benefit from feed- cal knowledge base includes two cases: back. Simply adding a case describing

〈S1, A2, S2, bad〉 and 〈S3, A4, S4, good〉, the new experience to the knowledge and S(t) is similar to both S1 and S3. Then, base improves the agent’s ability to pre- in the current situation S(t), the knowl- dict the results of actions and decide edge base would recommend ¬do(A2,t) more accurately what to do in future sit- and do(A4,t). uations. Thus, consequentialist (includ- For example, suppose the current ing utilitarian) analysis becomes a slow- situation S(t) includes two people, P er feedback loop, too slow to determine and Q, in conflict, the case antecedent the immediate response to an urgent

S1 describes P and Q as fighting, and A2 situation, but able to exploit informa- describes P killing Q. In this case, in S2, tion in the outcome of the selected ac- person Q is dead, which is bad. tion to improve the agent’s future deci- As a rich representation of experi- sions in similar situations (Figure 3). ence, 〈S1, A2, S2, bad〉 would be highly Open research problem. How do rea- detailed and specific. As a story, say the soning processes at different time-scales Biblical story of Cain and Abel, it would allow us to combine apparently incom- be much less rich, but would still con- patible mechanisms to achieve appar- vey the moral prohibition against kill- ently incompatible goals? What concrete ing. It could be abstracted even further, multi-time-scale architectures are useful to essentially a deontological rule: for moral and ethical judgment, and im- Thou shalt not kill. The more abstracted provement through learning? the antecedent, the more likely the Explanation. In addition to making stored case is to match a given situa- decisions and carrying out actions, an tion, but the less likely this case is to ethical agent must be able to explain distinguish adequately among cases and justify the decision and action,36 with different moral labels. providing several distinct types of feed- Situation S(t) also matches ante- back to improve the state of the ethical cedent S3 which describes P and Q as knowledge base. arguing, A4 describes them reaching Suppose agent P faces a situation, an agreement, and S4 has them both makes a decision, carries it out, and ex- alive, which is good. Having retrieved plains his actions to agent Q. If P is an both cases, the right behavior is to try exemplary member of the society and to follow case 〈S3, A4, S4, good〉 and makes a good decision, Q can learn avoid case 〈S1, A2, S2, bad〉, perhaps by from P’s actions and gain in expertise. taking other actions to make S(t) more If P makes a poor decision, simply be- similar to S3 and less similar to S1. ing asked to explain himself gives P an An essential part of case-based rea- opportunity to learn from his own mis- soning is the ability to draw on several take, but Q may also give P instructions

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 93 review articles

and insights that will help P make bet- Open research problem. Is it neces- stark dilemma distracts from the im- ter decisions in the future. Even if P has sary to distinguish between ethical and portant problems of designing a trust- made a poor decision and refuses to non-ethical case knowledge, or is this worthy self-driving car. learn from the experience, Q can still approach appropriate for both kinds of Learning to avoid the dilemma. As learn from P’s bad example. skill learning? stated, the Deadly Dilemma is difficult Explanation is primarily a mecha- Open research problem. Sometimes, a because it presents exactly two options, nism whereby individuals come to correct ethical judgment depends on both bad (hence, the dilemma). The share the society’s consensus beliefs learning a new concept or category, Deadly Dilemma is also extremely about morality and ethics. However, such as non-combatant3 or self-defense. rare. Far more often than an actual the influence is not only from the so- Progress in deep neural network learn- Deadly Dilemma, an agent will expe- ciety to individuals. Explanations and ing methods may be due to autonomous rience Near Miss scenarios, where insights can be communicated from learning of useful intermediate con- the dire outcomes of the Dilemma one person to another, leading to evo- cepts. However, it remains difficult to can be avoided, often by identifying lutionary social change. As more and make these intermediate concepts ex- “third way” solutions other than the more individuals share a new view of plicit and available for purposes such two bad outcomes presented by the morality and ethics, the society as a as explanation or extension to new Dilemma. These experiences can whole approaches a tipping point, af- problems. Furthermore, these meth- serve as training examples, helping ter which society’s consensus posi- ods depend on the availability and the agent learn to apply its ethical tion can change with startling speed. quality of large labeled training sets. knowledge on solvable problems, ac- Learning ethical case knowledge. A Open research problem. What mech- quiring “practical wisdom” about child learns ethical knowledge in the anisms are available for expressing ap- avoiding the Deadly Dilemma. form of simple cases provided by par- propriate abstractions from rich expe- Sometimes, when reflecting on a ents and other adults: rules, stories, rience to the features that enable Near Miss after the fact, the agent can and labels for experienced situations. tractable discrimination between mor- identify an “upstream” decision point These cases express social norms for al categories? In addition to deep neu- where a different choice would have the child. ral network learning, other examples avoided the Dilemma entirely. For ex- An adult experiences a situation include similarity measures among ample, it can learn to notice when a S(t), retrieves a set of similar cases, cases for case-based reasoning and small deviation from the intended adapts the actions from those cases to kernels for support vector machines. plan could be catastrophic, or when a an action A for this situation, performs How can these abstractions be learned pedestrian could be nearby but hid- that action, observes the result S′, and from experience? den. A ball bouncing into the street assigns a moral valence v. A new case from between parked cars poses no 〈S, A, S′, v〉 is constructed and added to The Deadly Dilemma threat to a passing vehicle, but a good the case base (Figure 3). With increas- The self-driving car is an intelligent driver slows or stops immediately, being experience, more cases will match robot whose autonomous decisions cause a small child could be chasing it. a given S(t), and the case-base will have potential to cause great harm to Implementing case-based strategies make finer-grained distinctions among individual humans. People often ask like these for a self-driving car may re- potential behaviors. The metaphor of about a problem I call the Deadly Di- quire advances in both perception and the fractal boundary between good and lemma: How should a self-driving car knowledge representation, but these bad ethical judgments in knowledge respond when faced with a choice be- advances are entirely feasible. space (Figure 2), implies that a good tween hitting a pedestrian (possibly a Earning trust. An agent earns trust approximation to this boundary re- small child who has darted into the by showing that its behavior consis- quires both a large number of cases street), versus crashing and harming tently accords with the norms of soci- (quantity) and correct placement and its passengers.21 ety. The hybrid architecture described labeling of those cases (quality). Either choice, of course, leads to a here sketches a way that an agent can Once the case base accumulates serious problem with the trustworthi- learn about those social norms from its clusters of cases with similar but not ness of the robot car. If the robot would experience, responding quickly to situ- identical antecedents, then some of choose to kill the pedestrian to save itations as they arise, but then more those clusters can be abstracted to self and its passengers, then why slowly learning by reflecting on its suc- much sparser cases (that is, rules), that should the public trust such robots cesses and failures, and identifying make certain actions forbidden or enough to let them drive on public useful abstractions and more efficient obligatory in certain situations. The roads? If the robot could choose to rules based on that experience. cluster of cases functions as a labeled harm its passengers, then why would In ordinary driving, the self-driving training set for a classification prob- anyone trust such a robot car enough car earns trust by demonstrating that lem to predict the result and evaluation to buy one? it obeys social norms, starting with of an action in antecedent situations in The self-driving car could be a bell- traffic laws, but continuing with cour- that cluster. This can determine which wether for how autonomous robots teous behavior, signaling its inten- attributes of the antecedent cases are will relate to the social norms that sup- tions to pedestrians and other driv- essential to a desired result and evalua- port society. However, while the Deadly ers, taking turns, and deferring to tion, and which are not. Dilemma receives a lot of attention, the others when appropriate. In crisis sit-

94 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 review articles uations, it demonstrates its ability to ful design must combine aspects of all Control, Inc. (1939); http://hdl.handle.net/2027/ wu.89090508862. use its situational awareness and fast theories. The physical and social envi- 15. Hardin, G. The tragedy of the commons. Science 162 reaction time to find “third ways” out ronment is immensely complex. Even (1968), 1243–1248. 16. Hursthouse, R. Virtue ethics. The Stanford of Near Miss scenarios. Based on so, some moral decisions must be Encyclopedia of Philosophy. E.N. Zalta, Ed., 2013. post-hoc crisis analyses, whether the made quickly. But there must also be a 17. Johnson, N.D. and Mislin, A.A. Trust games: A meta- analysis. J. Economic Psychology 32 (2011), 865–889. outcome was success or failure, it may slower deliberative evaluation proc- 18. Kolodner, J. Case-Based Reasoning. Morgan be able to learn to identify upstream ess, to confirm or revise the rapidly re- Kaufmann, 1993. 19. Le Guin, U. The ones who walk away from Omelas. decision points that will allow it to sponding rules and constraints. At New Dimensions 3. R. Silverberg, Ed. Nelson avoid such crises in the first place. longer time scales, there must be Doubleday, 1973. 20. Leyton-Brown, K. and Shoham, Y. Essentials of Game Technological advances, particu- mechanisms for learning new con- Theory. Morgan & Claypool, 2008. 21. Lin, P. The ethics of autonomous cars. The Atlantic larly in the car’s ability to predict the cepts for virtues and vices, mediating Monthly, (Oct. 8, 2013). intentions and behavior of other between perceptions, goals, plans, 22. López, B. Case-Based Reasoning: A Concise Introduction. Morgan & Claypool, 2013. agents, and in the ability to anticipate and actions. The technical research 23. Malle, B.F., Scheutz, M., Arnold, T.H., Voiklis, J.T., and potential decision points and places challenges are how to accomplish all Cusimano, C. Sacrifice one for the good of many? People apply different moral norms to human and that could conceal a pedestrian, will these goals. robot agents. In Proceedings of ACM/IEEE Int. Conf. certainly be important to reaching this Self-driving cars may well be the Human Robot Interaction (HRI), 2015. 24. Pauker, S.G. and Kassirer, J.P. Decision analysis. New level of behavior. We can be reasonably first widespread examples of trustwor- England J. Medicine 316 (1987), 250–258. optimistic about this kind of cognitive thy robots, designed to earn trust by 25. Pinker, S. The Better Angels of Our Nature: Why Violence Has Declined. Viking Adult, 2011. and perceptual progress in machine demonstrating how well they follow so- 26. Rand, D.G. and Nowak, M.A. Human cooperation. learning and artificial intelligence. cial norms. The design focus for self- Trends in Cognitive Science 17 (2013), 413–425. 27. Robinette, P., Allen, R., Li, W., Howard, A.M., and Since 94% of auto crashes are associ- driving cars should not be on the Dead- Wagner, A.R. Overtrust of robots in emergency ated with driver error,33 there will be ly Dilemma, but on how a robot’s evacuation scenarios. In Proceedings of ACM/IEEE Int. Conf. Human Robot Interaction (2016), 101–108. plentiful opportunities to demonstrate everyday behavior can demonstrate its 28. Rousseau, D.M., Sitkin, S.B., Burt, R.S., and Camerer, trustworthiness in ordinary driving trustworthiness. C. Not so different after all: A cross-discipline view of trust. Academy of Management Review 23, 3 (1998), and solvable Near Miss crises. Both so- Acknowledgment. This work took 393–404. 29. Russell, S. and Norvig, P. Artificial Intelligence: A ciety and the purchasers of self-driving place in the Intelligent Robotics Lab in Modern Approach. Prentice Hall, 3rd edition, 2010. cars will gain substantially greater per- the Computer Science and Engineer- 30. Sandel, M.J. Justice: What’s the Right Thing To Do? Farrar, Strauss and Giroux, 2009. sonal and collective safety in return for ing Division of the University of Michi- 31. Scheutz, M. and Arnold, T. Feats without heroes: slightly more conservative driving. gan. Research of the Intelligent Robots Norms, means, and ideal robot action. Frontiers in Robotics and AI 3, 32 (June 16, 2016), DOI: 10.3389/ For self-driving cars sharing the Lab is supported in part by grants from frobt.2016.00032. same ethical knowledge base, the be- the National Science Foundation (IIS- 32. Singer, P. The Expanding Circle: Ethics, Evolution, and Moral Progress. Princeton University Press, 1981. havior of one car provides evidence 1111494 and HS-1421168). Many 33. Singh, S. Critical reasons for crashes investigated in the about the trustworthiness of all others, thanks to the anonymous reviewers. National Motor Vehicle Crash Causation Survey. Technical Report DOT HS 812 115, National Highway Traffic Safety leading to rapid convergence. Administration, Washington D.C., Feb. 2015. 34. Sinnott-Armstrong, W. Consequentialism. References The Stanford Encyclopedia of Philosophy. E.N. Zalta, 1. Abney, K. Robotics, ethical theory, and metaethics: A Conclusion Ed. 2015. guide for the perplexed. Robot Ethics: The Ethical and 35. Solaiman, S.M. Legal personality of robots, Trust is essential for the successful Social Implications of Robotics. P. Lin, K. Abney, and corporations, idols and chimpanzees: A quest for G.A. Bekey, Eds. MIT Press, Cambridge, MA, 2012. functioning of society. Trust is neces- legitimacy. Artificial Intelligence and Law 25, 2 (2017), 2. Anderson, M., Anderson, S.L. and Armen, C. An 155–179; doi: 10.1007/s10506-016-9192-3. sary for cooperation, which produces approach to computing ethics. IEEE Intelligent 36. Toulmin, S. The Uses of Argument. Cambridge Systems 21, 4 (2006), 56–63. the resources society needs. Morality, University Press, 1958. 3. Arkin, R.C. Governing Lethal Behavior in Autonomous 37. Vallor, S. Technology and the Virtues: A Philosophical Robots. CRC Press, 2009. ethics, and other social norms encour- Guide to a Future Worth Wanting. Oxford University 4. Asimov, I. I, Robot. Grosset & Dunlap, 1952. Press, 2016. age individuals to act in trustworthy 5. Axelrod, R. The Evolution of Cooperation. 38. Wallach, W. and Allen, C. Moral Machines: Teaching Basic Books, 1984. ways, avoiding selfish decisions that Robots Right from Wrong. Oxford University Press, 2009. 6. Bacharach, M., Guerra, G. and Zizzo, D.J. The self- 39. Wright, J.R. and Leyton-Brown, K. Level-0 meta-models exploit vulnerability, violate trust, and fulfilling property of trust: An experimental study. for predicting human behavior in games. In ACM Theory and Decision 63, 4 (2007), 349–388. discourage cooperation. As we contem- Conference on Economics and Computation, 2014. 7. Bostrom, N. Superintelligence: Paths, Dangers, 40. Yildiz, M. Repeated games. 12 Economic Applications plate the design of robots (and other Strategies. Oxford University Press, 2014. of Game Theory, Fall 2012. MIT OpenCourseWare. 8. Bringsjord, S., Arkoudas, K. and Bello, P. Toward a AIs) that perceive the world and select (Accessed 6-24-2016). general logicist methodology for engineering ethically actions to pursue their goals in that correct robots. IEEE Intelligent Systems 21,4 (2006), world, we must design them to follow 38-44. 9. Brynjolfsson, E. and McAfee, A. The Second Machine Benjamin Kuipers ([email protected]) is a professor of the social norms of our society. Doing Age. W.W. Norton & Co., 2014. computer science and engineering at the University of this does not require them to be true 10. Burton, E., Goldsmith, J., Koenig, S., Kuipers, B., Mattei, Michigan, Ann Arbor, USA. N. and Walsh, T. Ethical considerations in artificial moral agents, capable of genuinely tak- intelligence courses. AI Magazine, Summer 2017; Copyright held by author. ing responsibility for their actions. arxiv:1701.07769. 11. Castelfranchi, C. and Falcone, R. Principles of trust Social norms vary by society, so ro- for MAS: Cognitive anatomy, social importance, and bot behavior will vary by society as quantification. InProceedings of the Int. Conf. Multi Agent Systems, 1998, 72–79. well, but this is outside the scope of 12. Dandekar, P., Goel, A., Wellman, M.P. and Wiedenbeck, B. Strategic formation of credit networks. ACM Trans. this article. Internet Technology 15, 1 (2015). Watch the author discuss The major theories of philosophi- 13. Driver, J. The history of utilitarianism. The Stanford his work in this exclusive Encyclopedia of Philosophy. E.N. Zalta, Ed., 2014. Communications video. cal ethics provide clues toward the de- 14. Eno, W.P. The story of highway traffic control, https://cacm.acm.org/videos/ sign of such AI agents, but a success- 1899–1939. The Eno Foundation for Highway Traffic how-can-we-trust-a-robot

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 95 Check out the new acmqueue app FREE TO ACM MEMBERS

acmqueue is ACM’s magazine by and for practitioners, bridging the gap between academics and practitioners of computer science. After more than a decade of providing unique perspectives on how current and emerging technologies are being applied in the field, the new acmqueue has evolved into an interactive, socially networked, electronic magazine.

Broaden your knowledge with technical articles focusing on today’s problems affecting CS in practice, video interviews, roundtables, case studies, and lively columns.

Keep up with this fast-paced world on the go. Download the mobile app.

Desktop digital edition also available at queue.acm.org. Bimonthly issues free to ACM Professional Members. Annual subscription $19.99 for nonmembers.

acmqueue_cacm_fp_ads_AS.indd 4 10/6/15 12:34 PM research highlights

P. 98 P. 99 Technical Perspective Time-Inconsistent Planning: A Graph-Theoretic A Computational Problem Framework Traces Task in Behavioral Economics Planning By Jon Kleinberg and Sigal Oren By Nicole Immorlica

P. 108 P. 109 Technical Perspective Analysis of SSL Certificate On Heartbleed: Reissues and Revocations A Hard Beginnyng Makth in the Wake of Heartbleed a Good Endyng By Liang Zhang, David Choffnes, Tudor Dumitras¸, Dave Levin, Alan Mislove, Aaron Schulman, and Christo Wilson By Kenny Paterson

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 97 research highlights

DOI:10.1145/3176187 To view the accompanying paper, Technical Perspective visit doi.acm.org/10.1145/3176189 rh A Graph-Theoretic Framework Traces Task Planning By Nicole Immorlica

ALGORITHMIC GAME THEORY has made row’s costs are greater than those of edges represent the costs of advanc- great strides in recent decades by as- the day after. This time-inconsistency ing from one task to the next. The suming standard economic models of can cause agents to procrastinate in- intended present and future actions rational agent behavior to study out- definitely and abandon valuable goals. of a quasi-hyperbolic discounter are comes in distributed computational With the growth of personalized simply weighted shortest-paths in settings. From the analysis of Internet computing, it is especially important this graph. routing to the design of advertisement for researchers to design and analyze This formulation allows research- auctions and crowdsourcing tasks, re- systems in the presence of irrational ers to use the extensive existing searchers leveraged these models to human behavior, such as that de- knowledge of graph algorithms to characterize the performance of the scribed by quasi-hyperbolic discount- analyze and optimize task planning. underlying systems and guide practi- ing. Our cellphones guide us through In the paper, Kleinberg and Oren use tioners in their optimization. These our lives, helping us, for example, to their framework to derive many re- models have tractable mathematical manage our time, optimize our diets, sults. Among these, they characterize formulations and broadly applicable and achieve our fitness goals. To do tasks that are particularly susceptible conclusions that drive their success, so effectively, these apps need a pre- to procrastination: they are those but they rely strongly on the assump- dictive and mathematically tractable that contain a simple structure as a tion of rationality. model of our behavior. The quasi-hy- graph minor. They also explore ways The assumption of rationality is perbolic discounting model is prom- to reduce procrastination by choice at times questionable, particularly in ising: economists have shown in both reduction: by scheduling a midterm systems in which human actors make lab and field experiments that it is exam, an instructor can remove the most of the decisions and in systems highly predictive. However, the prior choice of cramming for the final in that evolve over time. Humans, sim- literature fails to provide a suitable the last week of class, resulting in ply put, are bad at thinking about the framework in which to reason about better study habits. impact of their actions on their en- quasi-hyperbolic discounting in the The framework of Kleinberg and vironment and their future. We see presence of complex planning tasks. Oren, and the characterization and this every day in the way we manage In the following paper, Kleinberg optimization results it enables, is a our time. Students cram for exams and Oren describe a graph-theoretic step forward in incorporating sophis- despite planning not to, and even framework for task planning with ticated models of human behavior though it is well documented that quasi-hyperbolic discounting. In into computational systems. Apps well-spaced studying produces im- their framework, the goal and the in- designed to increase individual ef- proved learning results with equal ef- termediate tasks are nodes in a graph, fectiveness and related products can fort. Humans in lab experiments also and the weights of the (directed) use this framework to help us achieve consistently exhibit similar irrational our personal goals. They can predict time-inconsistent planning and pro- when we are in danger of procras- crastination behavior. The framework tinating, and perhaps, by cleverly In the 1990s, economists proposed hiding the availability of certain ac- an alternate model of agent behavior of Kleinberg and Oren tions, they can even help us mitigate called quasi-hyperbolic discounting, is a step forward the extent of procrastination. Subse- which incorporates time-inconsisten- quent and future work has and will cies and procrastination. In this mod- in incorporating continue to use the framework to el, agents overinflate the cost of cur- sophisticated develop many more such useful re- rent actions with respect to future days’ sults. In our complex modern world actions. Thus, an hour of studying to- models of with its growing plethora of choices, day might be as painful as two hours of human behavior automated planning of the sort aided studying on any future day. Note this by this paper and the research it in- causes agents to have a time-inconsis- into computational spires is indispensable. tent view of the future: today, the stu- systems. dent thinks that tomorrow’s costs are Nicole Immorlica ([email protected]) is a equal to that of the day after, but when senior researcher at Microsoft Research New England, Cambridge, MA, USA. tomorrow comes, the student will re- evaluate and decide that in fact tomor- Copyright held by author.

98 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 DOI:10.1145/3176189 Time-Inconsistent Planning: A Computational Problem in Behavioral Economics By Jon Kleinberg and Sigal Oren

Abstract they were at the moment the plan was made. This presents In many settings, people exhibit behavior that is inconsis- a challenge to any model of planning based on optimizing a tent across time—we allocate a block of time to get work utility function that is consistent over time: in an optimiza- done and then procrastinate, or put effort into a project and tion framework, the plan must have been an optimal choice then later fail to complete it. An active line of research in at the outset, but later it was optimal to abandon it. A line behavioral economics and related fields has developed and of work in the economics literature has thus investigated analyzed models for this type of time-inconsistent behavior. the properties of planning with objective functions that vary Here we propose a graph-theoretic model of tasks and over time in certain natural and structured ways. goals, in which dependencies among actions are represented by a directed graph, and a time-inconsistent agent A basic example and model constructs a path through this graph. We first show how To introduce these models, it is useful to briefly describe instances of this path-finding problem on different input an example due to George Akerlof,1 with the technical graphs can reconstruct a wide range of qualitative phenom- details adapted slightly for the discussion here. (The story ena observed in the literature on time-inconsistency, includ- will be familiar to readers who know Akerlof’s paper; we ing procrastination, abandonment of long-range tasks, and cover it in some detail because it will motivate a useful and the benefits of reduced sets of choices. We then explore a set recurring construction later in the work.) Imagine a deci- of analyses that quantify over the set of all graphs; among sion-making agent—Akerlof himself, in his story—who other results, we find that in any graph, there can be only needs to ship a package sometime during one of the next polynomially many distinct forms of time-inconsistent n days, labeled t = 1, 2, . . ., n, and must decide on which behavior; and any graph in which a time-inconsistent agent day t to do so. Each day that the package has not been sent incurs significantly more cost than an optimal agent must results in a fixed cost of 1 (per day), due to the lack of use of contain a large “procrastination” structure as a minor. the package’s contents; this means a cost of t if it is shipped on Finally, we use this graph-theoretic model to explore ways day t. (For simplicity, we will disregard the time the pack- in which tasks can be designed to motivate agents to reach age spends in transit, which is a constant additional cost designated goals. regardless of when it is shipped.) Also, shipping the package is an elaborate operation that will result in one-time cost of c > 1, due to the amount of time involved in getting 1. INTRODUCTION it sent out. The package must be shipped during one of the A fundamental issue in behavioral economics—and in the n specified days. modeling of individual decision-making more generally—is What is the optimal plan for shipping the package? Clearly to understand the effects of decisions that are inconsistent the cost c will be incurred exactly once regardless of the day over time. Examples of such inconsistency are widespread in on which it is shipped, and there will also be a cost of t if it everyday life: we make plans for completing a task but then is shipped on day t. Thus we are minimizing c + t subject to procrastinate; we put work into getting a project partially 1 £ t £ n; the cost is clearly minimized by setting t = 1. In other done but then abandon it; we pay for gym memberships but words, the agent should ship the package right away. then fail to make use of them. In addition to analyzing and But in Akerlof’s story, he did something that should be modeling these effects, there has been increasing interest familiar from one’s own everyday experience: he procrasti- in incorporating them into the design of policies and incen- nated. Although there were no unexpected changes to the tives in domains that range from health to personal finance. trade-offs involved in shipping the package, when each These types of situations have a recurring structure: a new day arrived there seemed to be other things that were person makes a plan at a given point in time for something more crucial than sending it out that day, and so each day he they will do in the future (finishing homework, exercising, resolved that he would instead send it tomorrow. The result paying off a loan), but at a later point in time they fail to fol- was that the package was not sent out until the end of the low through on the plan. Sometimes this failure is the result of unforeseen circumstances that render the plan invalid— The original version of this paper was published in the a person might join a gym but then break their leg and be Proceedings of the 15th ACM Conference on Economics and unable to exercise—but in many cases the plan is abandoned Computation (Palo Alto, CA, June 8–12, 2014), 547–564. even though the circumstances are essentially the same as

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 99 research highlights

time period. (In fact, he sent it a few months into the time 1. Procrastination, as discussed above. period once something unexpected happened to change the 2. Abandonment of long-range tasks, in which a person cost structure—a friend offered to send it as part of a larger starts on a multi-stage project but abandons it in the shipment—though this wrinkle is not crucial for the story.) middle, even though the costs and benefits of the proj- There is a natural way to model an agent’s decision to ect have remained essentially unchanged.15, b procrastinate, using the notion of present bias—the ten- 3. The benefits of choice reduction, in which reducing the dency to view costs and benefits that are incurred at the pres- set of options available to an agent can actually help ent moment to be more salient than those incurred in the them reach a goal more efficiently.9, 14 A canonical future. In particular, suppose that for a constant b > 1, costs example is the way in which imposing a deadline can that one must incur in the current time period are increased help people complete a task that might not get fin- by a factor of b in one’s evaluation.a Then in Akerlof’s exam- ished in the absence of a deadline.4 ple, when the agent on day t is considering the decision to send the package, the cost of sending it on day t is bc + t, while These consequences of time-inconsistency, as well as a the cost of sending it on day t + 1 is c + t + 1. The difference number of others, have in general each required their own between these two costs is (b − 1)c − 1, and so if (b − 1)c > 1, the separate and sometimes quite intricate modeling efforts. agent will decide on each day t that the optimal plan is to wait It is natural to ask whether there might instead be a single until day t + 1; things will continue this way until day n, when framework for representing tasks and goals in which all of waiting is no longer an option and the package must be sent. these effects could instead emerge “mechanically,” each just as a different instance of the same generic computa- Quasi-hyperbolic discounting tional problem. With such a framework, it would become Building on considerations such as those above, and oth- possible to search for worst-case guarantees, by quantifying ers in earlier work in economics,16, 19 a significant amount of over all instances, and to talk about designing or modifying work has developed around a model of time-inconsistency given task structures to induce certain desired behaviors. known as quasi-hyperbolic discounting.12 In this model, parametrized by quantities b, d £ 1, a cost or reward of value The present work: A graph-theoretic model c that will be realized at a point t ³ 1 time units into the Here we propose such a framework, using a graph-theoretic future is evaluated as having a present value of bdtc. (In other formulation. We consider an agent with present-bias param- words, values at time t are discounted by a factor of bdt.) With eter b who must construct a path in a directed acyclic graph b = 1 this is the standard functional form for exponential dis- G with edge costs, from a designated start node s to a des- counting, but when b < 1 the function captures present bias ignated target node t. We assume without loss of general- as well: values in the present time period are scaled up by b−1 ity that all the edges of G lie on some s − t path. We will call relative to all other periods. (In what follows, we will consis- such a structure a task graph. Informally, the nodes of the tently use b to denote b−1.) task graph represent states of intermediate progress toward Research on this (b, d)-model of discounting has been the goal t, and the edges represent transitions between extensive, and has proceeded in a wide variety of direc- them. Directed graphs have been shown to have consider- tions; see Ref. Frederick et al.7 for a review. To keep our able expressive power for planning problems in the artifi- analysis clearly delineated in scope, we make certain deci- cial intelligence literature;18 this provides evidence for the sions at the outset relative to the full range of possible robustness of a graph-based approach in representing these research questions: we focus on a model of agents who types of decision environments. Our concerns in this work, are naive, in that they do not take their own time-inconsis- however, are quite distinct from the set of graph-based plan- tency into account when planning; we do not attempt to ning problems in artificial intelligence, since our aim is derive the (b, d)-model from more primitive assumptions to study the consequences of time-inconsistency in these but rather take it as a self-contained description of the domains. agent’s observed behavior; and we discuss the case of d = A sample instance of this problem is depicted in Figure 1, 1 so as to focus attention on the present-bias parameter b. with the costs drawn on the edges. When the agent is stand- Note that the initial Akerlof example has all these proper- ing at a node v, it determines the minimum cost of a path ties; it is essentially described in terms of the (b, d)-model from v to t, but it does so using its present-biased evaluation with an agent who is naive about his own time-inconsis- of costs: the cost of the first edge on the path (starting from v) tency, with d = 1, and with b = b−1 (using the parameter b is evaluated according to the true cost, and all subsequent from that discussion). edges have their costs reduced by b. If the agent chooses Our starting point in this paper is to think about some of path P, it follows just the first edge (v, w) of P, and then it the qualitative predictions of the (b, d)-model, and how to re-evaluates which path to follow using this same present- analyze them in a unified framework. In particular, research biased evaluation but now from w. In this way, the agent in behavioral economics has shown how agents making iteratively constructs a path from s to t. plans in this model can exhibit the following behaviors.

b For purposes of our discussion, we distinguish abandonment of a task a Note that there is no time-discounting in this example, so the factor of b from the type of procrastination exhibited by Akerlof’s example, in which is only applied to the present time period, while all future time periods are the task is eventually finished, but at a much higher cost due to the effect of treated equally. We will return to the issue of discounting shortly. procrastination.

100 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3

Figure 1. A present-biased agent must choose a path from s to t. Moreover, with this framework in place, it becomes easier to express formal questions about design for these con- a 2 b texts: if as a designer of a complex task we are able to specify 2 the underlying graph structure, which graphs will lead time- 16 inconsistent agents to reach the goal efficiently? Our core questions are based on quantifying the inef- s 8 c 8 d 8 t ficiency from time-inconsistent behavior, designing task 2 16 structures to reduce this inefficiency, and comparing the behavior of agents with different levels of time-inconsis- e tency. Specifically, we ask:

1. In which graph structures does time-inconsistent planning have the potential to cause the greatest waste In the next section we will show how our graph-theo- of effort relative to optimal planning? retic model easily captures time-inconsistency phenom- 2. How do agents with different levels of present bias ena including procrastination, abandonment, and choice (encoded as different values of b) follow combinatori- reduction. But to make the definitions concrete, it is use- ally different paths through a graph toward the same ful to work through the agent’s computation on the graph goal? depicted in Figure 1. As shown in Figure 1, an agent that has 3. Can we increase an agent’s efficiency in reaching a a present-bias parameter of b = 1/2 needs to go from s to t. goal by deleting nodes and/or edges from the underly- From s, the agent evaluates the path s-a-b-t as having cost 16 + ing graph, thus reducing the number of options 2b + 2b = 18, the path s-c-d-t as having cost 8 + 8b + 8b = 16, available? and the path s-c-e-t as having cost 8 + 2b + 16b = 17. Thus the agent traverses the edge (s, c) and ends up at c. From c, In what follows, we address these questions in turn. the agent now evaluates the path c-d-t as having cost 8 + 8b For the first question, we consider n-node graphs and ask = 12 and the path c-e-t as having cost 2 + 16b = 10, and so how large the cost ratio can be between the path followed the agent traverses the edge (c, e) and then (having no further by a present-biased agent and the path of minimum choices) continues on the edge (e, t). total cost. Since deviations from the minimum-cost plan This example illustrates a few points. First, when the due to present bias are sometimes viewed as a form of agent set out on the edge (s, c), it was intending to next fol- “irrational” behavior, this cost ratio effectively serves as low the edge (c, d), but when it got to c, it changed its mind a “price of irrationality” for our system. We give a char- and followed the edge (c, e). A time-consistent agent (with acterization of the worst-case graphs in terms of graph b = 1), in contrast, would never do this; the path it decides minors; this enables us to show, roughly speaking, that to take starting at s is the path it will continue to follow all any instance with sufficiently high cost ratio must con- the way to t. Second, we are interested in whether the agent tain a large instance of the Akerlof example embedded minimizes the cost of traveling from s to t according to the inside it. real costs, not according to its evaluation of the costs, and in For the second question, we consider the possible paths this regard it fails to do so; the shortest path is s-a-b-t, with a followed by agents with different present-bias parameters b. cost of 20, while the agent incurs a cost of 26. As we sweep b over the interval [0, 1], we have a type of parametric path problem, where the choice of path is governed Overview of results by a continuous parameter (b in this case). We show that in Our graph-theoretic framework makes it possible to reason any instance, the number of distinct paths is bounded by a about time-inconsistency effects that arise in very differ- polynomial function of n, which forms an interesting con- ent settings, provided simply that the underlying decisions trast with canonical formulations of the parametric short- faced by the agent can be modeled as the search for a path est-path problem, in which the number of distinct paths can through a graph-structured sequence of options. And per- be super polynomial in n.5, 13 haps more importantly, since it is tractable to ask questions Lastly, for the third question, we show how it is possible that quantify over all possible graphs, we can cleanly com- for agents to be more efficient when nodes and/or edges pare different scenarios, and search for the best or worst are deleted from the underlying graph; on the other hand, possible structures relative to specific objectives. This is dif- if we want to motivate an agent to follow a particular path P ficult to do without an underlying combinatorial structure. through the graph, it can be crucial to present the agent with For example, suppose we were inspired by Akerlof’s example a subgraph that includes not just P but also certain addi- to try identifying the scenario in which time-inconsistency tional nodes and edges that do not belong to P. We give a leads to the greatest waste of effort. A priori, it is not clear graph-theoretic characterization of the possible subgraphs how to formalize the search over all possible “scenarios.” supporting efficient traversal. But as we will see, this is precisely something we can do if Before turning to these questions, we first discuss the we simply ask for the graph in which time-inconsistency basic graph-theoretic problem in more detail, showing how produces the greatest ratio between the cost of the path tra- instances of this problem capture the time-inconsistency versed and cost of the optimal path. phenomena discussed earlier in this section.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 101 research highlights

2. THE GRAPH-THEORETIC MODEL Extending the model to include rewards In order to argue that our graph-theoretic model captures a Thus far we can not talk about an agent who abandons variety of phenomena that have been studied in connection its pursuit of the goal midway through, since our model with time-inconsistency, we present a sequence of examples requires the agent to construct a path that goes all the way to t. to illustrate some of the different behaviors that the model A simple extension of the model enables to consider such exhibits. We note that the example as shown in Figure 1 situations. already illustrates two simple points: that the path chosen Suppose we place a reward of r at the target node t, which by the agent can be suboptimal; and that even if the agent will be claimed if the agent reaches t. Standing at a node traverses an edge e with the intention of following a path P v, the agent now has an expanded set of options: it can fol- that begins with e, it may end up following a different path P¢ low an edge out of v as before, or it can quit taking steps, that also begins with e. incurring no further cost but also not claiming the reward. For an edge e in G, let c(e) denote the cost of e; and for The agent will choose the latter option precisely when th a path P in G, let ei(P) denote the i edge on P. In terms of either there is no v-t path, or when the minimum cost of a this notation, the agent’s decision is easy to specify: when v-t path exceeds the value of the reward, evaluated in light of standing at a node v, it chooses the path P that minimizes present bias: for all v-t paths P. over all P that run from v to t. It fol- It is important to note a key feature of this evaluation: the lows the first edge of P to a new node w, and then performs reward is always discounted by b relative to the cost that is this computation again. being incurred in the current period, even if the reward will We begin by observing that Figure 2(a) represents a version be received right after this cost is incurred. (e.g., if the path P

of the Akerlof example from the introduction. (Recall that has a single edge, then the agent is comparing c(e1(P) ) to br.) here we use b to denote b−1.) Node t represents the state in In what follows, we will consider both these models: the

which the agent has sent the package, and node vi represents former fixed-goal model, in which the agent must reach t and the state in which the agent has reached day i without send- seeks to minimize its cost; and the latter reward model in ing the package. The agent has the option of going directly which the agent trades off cost incurred against reward at t, from node s to node t, and this is the shortest s-t path. But if and has the option of stopping partway to t. Aside from this

(b − 1)c > b, then the agent will instead go from s to v1, intend- distinction, both models share the remaining ingredients,

ing to complete the path s-v1-t in the next time step. At v1, based on traversing an s-t path in G.

however, the agent decides to go to v2, intending to complete It is easy to see that the reward model displays the phe-

the path v1-v2-t in the next time step. This process continues: nomenon of abandonment, in which the agent spends some the agent, following exactly the reasoning in the example cost to try reaching t, but then subsequently gives up with- from the introduction, is procrastinating and not going to t, out receiving the reward. Consider for example a three-node

and in the end its path goes all the way to the last node vn (n = path on nodes s, v1, and t, with an edge c(s, v1) = 1 and c(v1, t) 5 in the figure) before finally taking an edge to t. (One minor = 4. If b = 1/2 and there is a reward of 7 at t, then the agent

change from the set-up in the introduction is that the pres- will traverse the edge (s, v1) because it evaluates the total cost

ent-bias effect is also applied to the per-day cost of 1 as well; of the path at 1 + 4b = 3 < 7b = 3.5. But once it reaches v1, it this has no real effect on the underlying story.) evaluates the cost of completing the path at 4 > 7b = 3.5, and so it quits without reaching t. Figure 2. Path problems that exhibit procrastination, abandonment, and choice reduction. An example involving choice reduction It is useful to describe a more complex example that shows v 1 v 2 3 the modeling power of this shortest-path formalism, and 1 1 also shows how we can use the model to analyze deadlines v v 1 c c 4 as a form of beneficial choice reduction. (As should be clear, c c with a time-consistent agent it can never help to reduce the 1 1 set of choices; such a phenomenon requires some form of time-inconsistency.) First we describe the example in text, s c t c v5 and then show how to represent it as a graph. (a) The Akerlof example Imagine a student taking a three-week short course in which the required work is to complete two small projects

s v10 v20 v30 by the end of the course. It is up to the student when to do the projects, as long as they are done by the end. The student incurs an effort cost of one from any week in which

v01 v11 v21 v31 she does no projects (since even without projects there is still the lower-level effort of attending class), a cost of four from any week in which she does one project, and a cost of nine from any week in which she does both projects. v02 v12 v22 t Finally, the student receives a reward of 16 for complet- (b) Homework deadlines ing the course, and she has a present-bias parameter of b = 1/2.

102 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3

Figure 2(b) shows how to represent this scenario using a A bad example for the cost ratio graph. Node vij corresponds to a state in which i weeks of the We first describe a simple construction showing that the course are finished, and the student has completed j proj- cost ratio can be exponential in the number of nodes n. We ects so far; we have s = v00 and t = v32. All edges go one column then move on to the main result of this section, which is a to the right, indicating that one week will elapse regardless; characterization of the instances in which the cost ratio what is under the student’s control is how many rows the achieves this exponential lower bound. edge will span. Horizontal edges have cost one, edges that Our construction is an adaptation of the Akerlof example descend one row have cost four, and edges that descend two from the introduction. We have a graph that consists of a rows in a single hop have cost nine. In this way, the graph directed path s = v0, v1, v2, . . ., vn, and with each vi also linking precisely represents the story just described. directly to node t. (The case n = 5 is the graph in Figure 2(a).) How does the student’s construction of an s-t path work With b = b−1, we choose any m < b; we let the cost of the edge j out? From s, she goes to v10 and then to v20, intending to com- (vj, t) be m , and let the cost of each edge (vj, vj+1) be 0. plete the path to t via the edge (v20, t). But at v20, she evaluates Now, when the agent is standing at node vj, it evaluates j the cost of the edge (v20, t) as 9 > br = 16/2 = 8, and so she quits the cost of going directly to t as m , while the cost of the two- j+1 j without reaching t. The story is thus a familiar one: the stu- step path through vj+1 to t is evaluated as 0 + bm = (bm)m j dent plans to do both projects in the final week of the course, < m . Thus the agent will follow the edge (vj, vj+1) with the but when she reaches the final week, she concludes that it plan of continuing from vj+1 to t. But this holds for all j, so would be too costly and so she drops the course instead. once it reaches vj+1, it changes its mind and continues on to

The instructor can prevent this from happening through vj+2, and so forth. Ultimately it reaches vn, and then must go n a very simple intervention. If he requires that the first project directly to t at a cost of cb(s, t) = m . Since d(s, t) = 1 by using be completed by the end of the second week of the course, the edge directly from s to t, this establishes the exponential this corresponds simply to deleting node v20 from the graph. lower bound on the cost ratio cb(s, t)/d(s, t). Essentially, this

With v20 gone, the path-finding problem changes: now the construction shows that the Akerlof example can be made student starting at s decides to follow the path s-v10-v21-t, and quantitatively much worse than its original formulation by at v10 and then v21 she continues to select this path, thereby having the cost of going directly to the goal grow by a mod- reaching t. Thus, by reducing the set of options available to est constant factor in each time step; when a present-biased the student—and in particular, by imposing an intermedi- agent procrastinates in this case, it ultimately incurs an ate deadline to enforce progress—the instructor is able to exponentially large cost. induce the student to complete the course. The following observation establishes this is the highest There are many stories like this one about homework and possible cost ratio deadlines, and our point is not to focus too closely on it in particular. Indeed, to return to one of the underpinnings of Observation 3.1. Consider an agent currently at v and let u be our graph-theoretic formalism, our point is in a sense the the next node on Pb(v, t). Then d(u, t) < b × d(v, t). opposite: it is hard to reason about the space of possible “stories,” whereas it is much more tractable to think about Proof. If u is on the shortest path from v to t then clearly the space of possible graphs. Thus by encoding the set of the claim holds. Else, since the agent chose to continue to u stories mechanically in the form of graphs, it becomes fea- instead of continuing on the shortest path from v to t we have sible to reason about them as a whole. c(v, u) + bd(u, t) £ d(v, t). This implies that d(u, t) £ b × d(v, t).  We have thus seen how a number of different time- inconsistency phenomena arise in simple instances of the The observation essentially implies that with each step path-finding problem. The full power of the model, how- that the agent takes the cost of the path that it plans to take ever, lies in proving statements that quantify over all graphs; increases by an extra factor of at most b relatively to d(s, t). we begin this next. Hence, we have a tight upper bound on the cost ratio:

3. THE COST RATIO: A CHARACTERIZATION VIA Corollary 3.2. The cost ratio for a graph G with n nodes is at GRAPH MINORS most bn−2. Our path-finding model naturally motivates a basic quantity of interest: the cost ratio, defined as the ratio between the A graph minor characterization cost of the path found by the agent and the cost of the short- We now provide a structural description of the graphs on est path. We work here within the fixed-goal version of the which the cost ratio can be exponential in the number of model, in which the agent is required to reach the goal t and nodes n—essentially we show that a constant fraction of the the objective is to minimize the cost of the path used. nodes in such a graph must have the structure of the Akerlof To fix notation for this discussion, given a directed acyclic example. graph G on n nodes with positive edge costs, we let d(v, w) We make this precise using the notion of a minor. Given denote the cost of the shortest v-w path in G (using the true two undirected graphs H and K, we say that H contains a edge costs, not modified by present bias). Let Pb(v, t) denote K-minor if we can map each node κ of K to a connected sub- the the v-t path followed by an agent with present-bias b, and graph Sκ in H, with the properties that (i) Sκ and Sκ¢ are dis- let cb(v, t) be the total cost of this path. The cost ratio can thus joint for every two nodes κ, κ¢ of K, and (ii) if (κ, κ¢) is an edge be written as cb(s, t)/d(s, t). of K, then in H there is some edge connecting a node in Sκ to

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 103 research highlights

a node in Sκ¢. Informally, the definition means that we can a directed cycle and our graph is a directed acyclic graph.

build a copy of K using the structure of H, with disjoint con- Furthermore, Qi can not intersect with P at any node u after i nected subgraphs of H playing the role of “super-nodes” that vi since by our choice of vi, we have d(u, t) > b ³ d(vi, t) for every

represent the nodes of K, and with the adjacencies among node u after vi. these super-nodes representing the adjacencies in K. The Finally, we should show that indeed for every 1 £ i £ k, i minor relation shows up in many well-known results in graph there exists a distinct node vi with the property that (i) d(vi, t) £ b , i theory, perhaps most notably in Kuratowski’s Theorem that and (ii) for any node u after vi on Pb(s, t), we have d(u, t) > b . a nonplanar graph must contain either the complete graph First observe that since the cost ratio is λn and the length of 6 K5 or the complete bipartite graph K3,3 as a minor. the path is at most n, the path must contain an edge (u, v) of Our goal here is to show that if G has exponential cost cost at least λn/n. Roughly speaking, since n is large enough

ratio, then its undirected version must contain a large copy there exists λ0 > 1 such that . In particular this of the graph underlying the Akerlof example as a minor. In implies that . By Observation 3.1 we have that with other words, the Akerlof example is not only a way to pro- each step on the path P the cost of the shortest path can duce a large cost ratio, but it is in a sense an unavoidable increase by at most a factor of b. Thus, there exist k nodes signature of any example in which the cost ratio is very large. as required. We set this up as follows. Let σ(G) denote the skeleton After the initial publication of our original paper, Tang of G, the undirected graph obtained by removing the direc- et al.20 extended Theorem 3.3 as follows:

tions on the edges of G. Let Fk denote the graph with nodes k−2 v1, v2, . . ., vk, and w, and edges (vi, vi+1) for i = 1, . . ., k − 1, and (vi, w) Theorem 3.4. If the cost ratio is greater than b , then σ(G)

for i = 1, . . ., k. We refer to Fk as the k-fan. contains an Fk-minor. We now claim Both Theorem 3.3 and its tighter counterpart Theorem Theorem 3.3. For every λ > 1, if n is sufficiently large and the 3.4 offer some qualitatively relevant advice for thinking n cost ratio is greater than λ , then σ(G) contains an Fk-minor for about the structure of complicated tasks: to avoid creat- k = Θ(n). ing inefficient behavior due to present bias, it is better to Proof sketch.c We now provide a sketch of the proof. The organize tasks so that they do not contain large fan-like

basic idea is to pin down k = Θ(n) nodes, v1, . . ., vk, on the path structures. The point to appreciate is that such fan-like

Pb(s, t) and let P be the portion of the path from s to vk. We structures are not purely graph-theoretic abstractions;

show that from each vi the shortest path Qi intersects with P they arise in real settings whenever a task has a series of

only at vi. This is illustrated in Figure 3. “branches” (as in Akerlof’s story) that allows an agent to

Once we have this structure, we obtain an Fk minor by par- repeatedly put off completing the task. The theorems are

titioning P into segments such that each vi is in a distinct seg- a way of formalizing the idea that such sets of repeated

ment, and then we contract each segment. Since all the Qi’s branches are the crux of the reason why present-biased are connected to t, we can contract them together (without the individuals incur unnecessary inefficiency in complet-

vi’s). Finally, as all segments of P are connected to one another, ing large tasks. And correspondingly, organizing tasks

and each segment is connected to t by Qi, we get a fan Fk. in a way that breaks this type of branching—for example, For simplicity we normalize the edges costs such that with intermediate deadlines or subgoals—can be a way of

d(s, t) = 1. We choose the nodes v1, . . ., vk such that for every reducing inefficiency. i i, vi is the last node on the path P such that d(vi, t) £ b . With this choice it is not hard to show that the shortest path from 4. COLLECTIONS OF HETEROGENEOUS AGENTS

vi to t (Qi) intersects with P only at vi. In particular, Qi can not Thus far we have focused on the behavior of a single agent

intersect with P at any node before vi since this will create with a given present-bias parameter b. Now we consider all possible values of b, and ask the following basic ques-

tion: how large can the set {Pb(s, t) : b ∈ [0, 1]} be? In c This sketch is based on the full proof in our original paper, and also draws other words, if for each b, an agent with parameter b were 17 on the exposition in Tim Roughgarden’s lecture notes about our paper. to construct an s-t path in G, how many different paths would be constructed across all the agents? Bounding this quantity tells us how many genuinely “distinct” types of Figure 3. The construction of an -minor in Theorem 3.3. Fk behaviors there are for the instance defined by G. Let P(G)

denote the set {Pb (s, t) : b ∈ [0, 1]}. Despite the fact that b

v3 comes from the continuum [0, 1], the set P(G) is clearly P finite, since G only has finitely many s-t paths. The ques- v2 Q3 tion is whether we can obtain a nontrivial upper bound on

v1 the size of P(G), and in particular one that does not grow Q2 Q exponentially in the number of nodes n. In fact this is pos- v0 Q 1 0 sible, and our main goal in this section is to prove the following theorem. s t Theorem 4.1. For every directed acyclic graph G, the size of

104 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3

P(G) is O(n2). Moreover, there exists a graph for which the size activities invalid, as the deadline did in Figure 2(b) ). Let us of P(G) is Ω(n2). say that a subgraph G¢ of G motivates the agent if in G¢ with Proof idea. We use the following procedure to “dis- reward r, the agent reaches the goal node t. (We will also cover” all the paths in P(G). We start by taking b = 0 and let refer to G¢ as a motivating subgraph.) Note that it is possible

P0 the path that the agent with b = 0 takes. Now, we gradu- for the full graph G to be a motivating subgraph. ally increase b till we reach b* such that an agent with b* will It would be natural to conjecture that if there is any sub- take a different path. We claim that there is at least one edge graph G¢ of G that motivates the agent, then there is a moti-

(v, u) in P0 that will not take part in any path that an agent vating subgraph consisting simply of an s-t path P that the with b > b* will take. More generally, each time that we dis- agent follows. In fact this is not the case. Figure 4 shows a cover a new path, we essentially delete at least one edge from graph illustrating a phenomenon that we find somewhat the graph. Hence the number of paths in P(G) is bounded by surprising a priori, though not hard to verify from the exam- the number of edges in the graph. ple. In the graph G depicted in the figure, an agent with Theorem 4.1 tells us that the effect of present-bias on the b = 1/2 will reach the goal t. However, there is no proper path that an agent takes is in some sense limited as the spe- subgraph of G in which the agent will reach the goal. The cific value ofb an agent has determines which path will the point is that the agent starts out expecting to follow the path agent take from a precomputed set of at most O(n2) different s-a-b-t, but when it gets to node a it finds the remainder paths. Since this O(n2) quantity is much smaller in general of the path a-b-t too expensive to justify the reward, and it than the full set of possible paths, it says that the possible switches to a-c-t for remainder. With just the path s-a-b-t in heterogeneity in agent behavior based on different levels isolation, the agent would get stuck at a; and with just s-a-c-t, of present bias is not as extensive as it might initially seem. the agent would never start out from s. It is crucial the agent Furthermore, quantifying this heterogeneity is a first step mistakenly believes the upper path is an option in order to in designing efficient task graphs for populations of agents eventually use the lower path to reach the goal. that are heterogeneous. It is interesting, of course, to consider real-life analogues of this phenomenon. In some settings, the structure in 5. MOTIVATING AN AGENT TO REACH THE GOAL Figure 4 could correspond to deceptive practices on the part We now consider the version of the model with rewards: of the designer of G—in other words, inducing the agent to there is a reward at t, and the agent has the additional option reach the goal by misleading them at the outset. But there of quitting if it perceives—under its present-biased evalua- are other settings in real life where one could argue that the tion—that the value of the reward is not worth the remain- type of deception represented here is more subtle, not any ing cost in the path. one party’s responsibility, and potentially even salutary. For Note that the presence of the reward does not affect the example, suppose the graph schematically represents the agent’s choice of path, only whether it continues along the learning of a skill such as a musical instrument. There’s path. Thus we can clearly determine the minimum reward r the initial commitment corresponding to the edge (s, a), required to motivate the agent to reach the goal in G by sim- and then the fork at a where one needs to decide whether ply having it construct a path to t according to our standard to “get serious about it” (taking the expensive edge (a, b) ) or fixed-goal model, identifying the node at which it perceives not (taking the cheaper edge (a, c) ). In this case, the agent’s the remaining cost to be the greatest (due to present bias trajectory could describe the story of someone who derived this might not be s), and assigning this maximum perceived personal value from learning the violin (the lower path) cost as a reward at t. even though at the outset they believed incorrectly that they A more challenging question is suggested by the possi- would be willing to put the work into becoming a concert bility of deleting nodes and edges from G; recall that Figure violinist (the upper path). 2(b) showed a basic example in which the instructor of a course was able to motivate a student to finish the course- The structure of minimal motivating subgraphs work by deleting a node from the underlying graph. (This Given that there is sometimes no single path in G that is deletion essentially corresponded to introducing a deadline motivating, how rich a subgraph do we necessarily need to for the first piece of work.) This shows that even if the reward motivate the agent? Let us say that a subgraph G* of G is a remains fixed, in general it may be possible for a designer minimal motivating subgraph if (i) G* is motivating, and (ii) to remove parts of the graph, thereby reducing the set of no proper subgraph of G* is motivating. Thus, for example, options available to the agent, so as to get the agent to reach the goal. We now consider the structure of the subgraphs Figure 4. A minimal subgraph for getting an agent to reach t. that naturally arise from this process.

Motivating subgraphs: A fundamental example b Reward 2 The basic set-up we consider is the following. Suppose the 6 12 agent in the reward model is trying to construct a path from s 2 a t s to t in G; the reward r is not under our control—perhaps it 3 6 is defined by a third party, or represents an intrinsic reward that we can not augment—but we are able to remove nodes c and edges from the graph (essentially by declaring certain

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 105 research highlights

in Figure 4, the graph G is a minimal motivating subgraph of on which it is placed. Now the question is to place rewards itself; no proper subgraph of G is motivating. on the nodes or edges of an instance G such that the agent Concretely, then, we can ask the following question: what reaches the goal t while claiming as little total reward as pos- can a minimal motivating subgraph look like? For example, sible; this corresponds to the designer’s objective to pay out could it be arbitrarily dense with edges? as little as possible while still motivating the agent to reach In fact, minimal motivating subgraphs necessarily have the goal. Following our paper, Albers and Kraft2 and Tang a sparse structure, which we now describe in our next theo- et al.20 studied different versions of this question and showed rem. To set up this result, we need the following definition. that the problem of assigning intermediate rewards in an Given a directed acyclic graph G and a path P in G, we say that optimal way is NP-complete. A version worth mentioning is a path Q in G is a P-bypass if the first and last nodes of Q lie on the one in which the designer only cares about minimizing P, and no other nodes of Q do; in other words, P ∩ Q is equal the rewards that the agent actually takes. Such a formulation to the two ends of Q. We now have of the problem comes with the danger that a designer would be able to create “exploitive” solutions in which the agent is Theorem 5.1. If G* is a minimal motivating subgraph, then it motivated by intermediate rewards that it will never claim, contains an s-t path P* with the properties that because these rewards are on nodes that the agent will never (i) Every edge of G* is either part of P* or lies on a reach. P*-bypass in G*; and (ii) Every node of G* has at most one outgoing edge that 6. CONCLUSION AND SUBSEQUENT WORK does not lie on P*. We have developed a graph-theoretic model in which an agent constructs a path from a start node s to a goal node Proof sketch. Roughly speaking, there are two types t in an underlying graph G representing a sequence of of edges that should be included in a minimal motivating tasks. Time-inconsistent agents may plan an s-t path that subgraph: edges that the agent will actually take (these is different from the one they actually follow, and this type are the edges of the path P*) and edges that at some point of behavior in the model can reproduce a range of qualita- the agent (wrongly) believes that it will take (these are tive phenomena including procrastination, abandonment the edges on the P*-bypasses). It is clearly the case that of long-range tasks, and the benefits of a reduced set of an edge e that the agent never plans to take can be safely options. Our results provide characterizations for a set of removed from the graph without affecting the agent’s basic structures in this model, including for graphs achiev- decisions. Furthermore, since all the bypass edges are ing the highest cost ratios between time-inconsistent agents only used in shortest path computations it is impossible and shortest paths, and we have investigated the structure * for a node v in P to have two neighbors w1 and w2 not on of minimal graphs on which an agent is motivated to reach * * P such that the agent at some v1 on P plans to follow a the goal node.

path that includes the edge (v, w1) and an agent at some v2 There is a wide range of broader issues for further work. * on P plans to follow a path that includes the edge (v, w2). These include finding structural properties beyond our

This is simply because if (v, w1) + d(w1, t) £ (v, w2) + d(w2, t) graph-minor characterization that have a bearing on the

the agent will choose the path that contains (v, w1) both cost ratio of a given instance; obtaining a deeper under-

when standing at v1 and at v2 and otherwise it will choose standing of the relationship between agents with different

the path that contains (v, w2) in both cases. levels of time-inconsistency as measured by different values After the publication of our original paper, Tang et al.20 of b; and developing algorithms for designing graph struc- and Albers and Kraft2 independently showed that determin- tures that motivate effort as efficiently as possible, including whether a task graph admits a motivating subgraph is ing for multiple agents with diverse time-inconsistency NP-complete. One way of circumventing these hardness properties. results is to identify task graphs that are more common in In work following the initial appearance of our paper, practice and asking whether on these graphs the motivat- the results were extended in several subsequent lines of ing subgraph problem can be solved in polynomial time. research. We have discussed several of these further results Alternatively, we can consider approximation algorithms. in the text thus far, including a tighter graph-minor char- Albers and Kraft2 considered a variant of this question: acterization of instances with high cost ratio,20 and a set of what is the minimum reward r for which a motivating sub- hardness results and approximation algorithms for moti- graph exists? They showed that this problem can not be vating subgraphs.2, 3, 20 Further results beyond these have approximated by a factor better than , and they pre- considered the behavior of different types of agents. Gravin sented a approximation algorithm. Interestingly, et al.8 studied the behavior of a present-biased agent whose the subgraph achieving the approximation is a path. present-bias parameter is not fixed over time; rather, after Surprisingly, in a different paper, Albers and Kraft3 were each step it re-samples the parameter from some fixed disable to break the barrier and presented a 2-approxima- tribution. In Kleinberg et al.,10 the authors of the present tion algorithm for a more powerful designer who is able to paper together with Manish Raghavan studied the behavior increase the costs of edges. of sophisticated agents (building on a formalization from An alternative way to motivate an agent to reach t is to O’Donoghue and Rabin14), who are aware of their present place intermediate rewards on specific nodes or edges; the bias and can take it into account in their planning. Such agent will claim each reward if it reaches the node or edge agents are not able to ignore or disregard their present

106 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3

bias—they still prefer avoiding costs in the present time agents with present bias. In Proc. Econ. Studies 35, 2 (Apr. 1968), 201–208. ACM Conf. Econ. Comp. (2016). 17. Roughgarden, T. Lecture 19: Time- period—but their sophistication does mean that they can 11. Kleinberg, J., Oren, S., Raghavan, M. inconsistent planning, 2016. http:// take measures to reduce the future effects of present bias in Planning with multiple biases. In theory.stanford.edu/tim/f16/l/l19.pdf. Proc. ACM Conf. Econ. Comp. 18. Russell, S.L., Norvig, P. Artificial 11 their plans. Finally Kleinberg et al., studied the behavior (2017). Intelligence: A Modern Approach. of agents that exhibit two biases concurrently: present bias, 12. Laibson, D. Golden eggs and Prentice Hall (Upper Saddle River NJ, hyperbolic discounting. Q. J. Econ. USA, 1994). as in this paper, and sunk-cost bias, which is the tendency to 112, 2 (1997), 443–478. 19. Strotz, R.H. Myopia and inconsistency 13. Nikolova, E., Kelner, J.A., Brand, M., in dynamic utility maximization. Rev. reason about costs already incurred in the formulation of Mitzenmacher, M. Stochastic Econ. Studies 23 (1955). plans for the future. shortest paths via quasi-convex 20. Tang, P., Teng, Y., Wang, Z., Xiao, S., maximization. In Proc. 14th European Xu, Y. Computational issues in time- Symposium on Algorithms (2006), inconsistent planning. In Proc. 31st Acknowledgments 552–563. AAAI Conf. Artificial Intelligence 14. O’Donoghue, T., Rabin, M. Doing it now (2017). We thank Supreet Kaur, Sendhil Mullainathan, and Ted or later. American Econ. Rev. 89 (1999). O’Donoghue for valuable discussions and suggestions. 15. O’Donoghue, T., Rabin, M. Procrastination on long-term projects. J. Econ. Behav. Org. 66 (2008). 16. Pollak, R.A. Consistent planning. Rev. References linear and combinatorial programming. 1. Akerlof, G.A. Procrastination and PhD thesis, U. Michigan (1983). obedience. American Econ. 6. Diestel, R. Graph Theory, 3 edn. Rev.: Papers and Proc. 81 (1991). Springer (Berlin/Heidelberg, 2005). Jon Kleinberg ([email protected]), Sigal Oren ([email protected]), Ben 2. Albers, S., Kraft, D. Motivating time- 7. Frederick, S., Loewenstein, G., Cornell University, Ithaca, NY, USA. Gurion University of the Negev, Be’er inconsistent agents: A computational O’Donoghue, T. Time discounting Sheva, Israel. approach. In Intl. Conf. Web and and time preference. J. Econ. Lit. 40, 2 Internet Econ. (2016). (June 2002), 351–401. 3. Albers, S., Kraft, D. On the value 8. Gravin, N., Immorlica, N., Lucier, B., of penalties in time-inconsistent Pountourakis, E. Procrastination planning. In Proc. 44th Intl. Colloq. with variable present bias. In Proc. on Automata, Languages and ACM Conf. Econ. Comp. (2016). Programming (2017). 9. Kaur, S., Kremer, M., Mullainathan, S. 4. Ariely, D., Wertenbroch, K. Procrastination, Self-control and the development deadlines, and performance: of work arrangements. American self-control by precommitment. Econ. Rev.: Papers and Proceedings Psych. Sci. 13 (2002). 100 (2002). 5. Carstensen, P. The complexity of 10. Kleinberg, J., Oren, S., Raghavan, M. some problems in parametric Planning problems for sophisticated © 2018 ACM 0001-0782/18/3 $15.00

He was the conscience of the computing industry...and paid for it.

The first full-length biography of Edmund Berkeley, computing pioneer, social activist, and founding member of the ACM. It is an historical narrative of a man ultimately in favor of engineering peace, instead of war, and how his career was ultimately damaged by politicians determined to portray him as a Communist sympathizer. Berkeley’s life work provides a lens to understand social and political issues surrounding the early development of electronic computers which ties directly to current debates about the use of autonomous intelligent systems.

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 107 research highlights

DOI:10.1145/3176242 To view the accompanying paper, Technical Perspective visit doi.acm.org/10.1145/3176244 rh more than two years, and could have On Heartbleed: been exploited during that time. A third A Hard Beginnyng Makth a Good Endyng possible reason is ignorance on the JOHN HEYWOOD (1497–1580) part of sysadmins: while patching is By Kenny Paterson part of the everyday sysadmin culture, dealing with certificates and keys is THE SSL/TLS PROTOCOL suite has become the tem administrators should revoke their not. The paper points out a more sys- de facto secure protocol for communi- public key certificates, generate new tematic study is needed in order to un- cations on the Web, protecting billions key pairs, and request their Certifica- derstand certificate management and of communications sessions between tion Authorities (CAs) to issue new how it is tackled by sysadmins. browsers and servers on a daily basis. We certificates. Private keys, among the More broadly, the Heartbleed inci- use it every time we access our social me- security “crown jewels” for SSL/TLS, dent has had a lasting and net-positive dia feeds, or whenever an app running had potentially been compromised. impact on the security of the Web. The on our mobile device wants to contact its The following paper examines to bug led to a much closer inspection of home server. It has become an almost in- what extent revocation and reissuance the OpenSSL code base, leading to oth- visible part of the Web’s security infra- of certificates happened post-Heart- er problems being subsequently dis- structure, supported by an eclectic mix of bleed. The short and surprising answer: covered. It also led to a wider debate technologies including public key cryp- not so much. Zhang et al. cleverly use the about the wisdom of having a security tography, certificates, and the Web PKI. Heartbleed incident as an opportunity monoculture, about the quality of the So when a serious security vulnera- to perform a natural experiment, track- OpenSSL code, and about the way in bility is discovered in the SSL/TLS pro- ing the rate at which large websites from which the OpenSSL project was being tocol itself, or in one of the main imple- the Alexa top 1M chose to revoke and re- run. And it triggered a debate about the mentations like OpenSSL, one would issue certificates. They carefully assess responsibilities of large companies naturally expect a rapid response—sys- the extent to which those sites would who make free use of open source soft- tem administrators would roll into ac- have been vulnerable to Heartbleed, and ware like OpenSSL without contribut- tion, patching their software as quickly then use the open nature of the Web to ing materially to its development. as possible, and taking any other reme- collect information about when those Heartbleed resulted in major indus- dial actions that might be necessary. sites’ certificates were actually changed, try players forming the Core Infrastruc- The following paper by Zhang et al. and whether the previous certificates ture Initiative, a project intended to paints a very different picture in the con- were properly revoked. To a security- fund critical elements of the global in- text of the most famous SSL/TLS vulner- conscious reader, the final statistics formation infrastructure. OpenSSL has ability of all, Heartbleed. The Heartbleed make depressing reading. For example, in turn significantly revised its opera- vulnerability resides in the OpenSSL im- of roughly 107,712 vulnerable websites tions, expanded the development team, plementation of the Heartbeat protocol. (that is, websites for which the private and heavily refactored the codebase. The Heartbeat protocol is an extension key could have been exposed), only In parallel, Google initiated its own of SSL/TLS for checking the “liveness” of 26.7% had reissued certificates by the fork of OpenSSL called BoringSSL. The a connection. The Heartbleed bug lay in end of April 2014, while 60% of those naming is considered, with “boring” OpenSSL’s failure to correctly perform sites did not properly revoke their old being intended to convey an impres- bounds checking when processing certificates (meaning that, had the cor- sion of “no nasty surprises.” In October Heartbeat messages. An attacker, situat- responding private keys been exposed, 2015, Google announced it had ed anywhere on the planet, could induce the sites would still be vulnerable). switched over to BoringSSL across its a server to return large amount of data to The authors discuss some of the entire codebase (comprising several the attacker from arbitrary (but uncon- reasons why such low rates of revoca- billion lines of code). trolled) portions of its stack. This memo- tion and reissuance were seen. They The “Let’s Encrypt” initiative has ry leak would allow the attacker to learn a also report anecdotal evidence started to address the problem of help- vulnerable server’s private key, with di- gleaned from surveying system ad- ing sysadmins to better manage certifi- sastrous security consequences. ministrators. A key reason is the pro- cates. Let’s Encrypt is a free CA, which Heartbleed became public in early cesses are manual rather than being simplifies the process of obtaining cer- April 2014. The Internet community rap- automated. One may also advance the tificates and switching on SSL/TLS for idly developed free Heartbleed scanning argument that customers pay CAs for websites. In mid-2017, its 100-millionth services, and published statistics on certificates, and security budgets are certificate was issued, and Mozilla’s te- vulnerable servers. Some websites were always under pressure. One possible lemetry indicated more than half of all actually attacked, though all hosts in perception is that the window of expo- HTTP connections were protected. the Alexa top 500 were patched within sure was small, because most vulnera- 48 hours. As well as immediately patch- ble systems were patched quickly. This Kenny Paterson is a professor of Information Security at Royal Holloway, University of London, U.K. ing OpenSSL to remove the vulnerability, belies the fact that the vulnerability security experts recommended that sys- was present in the OpenSSL code for Copyright held by author.

108 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 DOI:10.1145/3176244 Analysis of SSL Certificate Reissues and Revocations in the Wake of Heartbleed By Liang Zhang, David Choffnes, Tudor Dumitras¸, Dave Levin, Alan Mislove, Aaron Schulman, and Christo Wilson

Abstract between browsers and servers. A recent study suggests A properly managed public key infrastructure (PKI) is 0.2% of SSL connections to Facebook correspond to such critical to ensure secure communication on the Internet. man-in-the-middle attacks.10 After considerable research Surprisingly, some of the most important administrative into understanding and improving the speed at which steps—in particular, reissuing new X.509 certificates and software is patched,14,22 much of software patching has revoking old ones—are manual and remained unstudied, become automated. However, the web’s PKI requires a sur- largely because it is difficult to measure these manual pro- prising amount of manual administration. To revoke a cer- cesses at scale. tificate, website administrators must send a request to their We use Heartbleed, a widespread OpenSSL vulnerability Certificate Authority (CA), and this request may be manually from 2014, as a natural experiment to determine whether reviewed before the certificates are finally added to a list that administrators are properly managing their certificates. All browsers (are supposed to) check. Such operations occur at domains affected by Heartbleed should have patched their human timescales (hours or days) instead of computer ones software, revoked their old (possibly compromised) certifi- (seconds or minutes). An important open question is: when cates, and reissued new ones, all as quickly as possible. We private keys are compromised, how long are SSL clients find the reality to be far from the ideal: over 73% of vulnerable exposed to potential attacks? certificates were not reissued and over 87% were not revoked Historically, these manual processes have been difficult three weeks after Heartbleed was disclosed. Our results to measure: how can one measure, at scale, how long these also show a drastic decline in revocations on the weekends, processes take if we do not know how often, or precisely even immediately following the Heartbleed announcement. when, administrators realize their keys are compromised? These results are an important step in understanding the In this paper, we use a widespread security vulnerability manual processes on which users rely for secure, authenti- from 2014, Heartbleed, as a natural experiment: the moment cated communication. Heartbleed was announced, all administrators of vulnerable servers should have initiated their manual processes as quickly as possible.3 This natural experiment allows us 1. INTRODUCTION to measure at scale the manual administration of the web’s Server authentication is the cornerstone of secure com- PKI. In particular, this paper focuses on the response to munication on the Internet; it is the property that allows the public announcement of Heartbleed, in terms of how client applications such as online banking, email, and quickly certificates were reissued and whether or not the e-commerce to ensure the servers with whom they com- certificates were eventually revoked. municate are truly who they say they are. In practice, Our results expose incomplete and slow administrative server authentication is made possible by the globally practices that ultimately weaken the security of today’s PKI. distributed Public Key Infrastructure (PKI). The PKI lever- On the positive side, we also identify ways in which the PKI ages cryptographic mechanisms and X.509 certificates to can be strengthened. Our hope is that, through better under- establish the identities of popular websites. This mecha- standing how the PKI operates in practice, the security and nism works in conjunction with other network protocols— research community can take concrete steps toward improv- particularly Secure Sockets Layer (SSL) and Transport Layer ing this system on which virtually all Internet users rely. Security (TLS)—to provide secure communications, but the PKI plays a key role: without it, a browser could establish 2. BACKGROUND a secure connection with an attacker that impersonates a In this section, we review the relevant background of SSL/ trusted website. TLS and the PKI, and we describe the Heartbleed vulnerabil- The secure operation of the web’s PKI relies on respon- ity that serves as our natural experiment. sible administration. When a software vulnerability is discovered, administrators must act quickly and deploy the patch to prevent attackers from exploiting the vulnerability. The original version of this paper was published in Similarly, after a potential key compromise, website admin- the 2014 ACM Internet Measurement Conference (IMC’14). istrators must revoke the corresponding certificates to prevent attackers from intercepting encrypted communications

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 109 research highlights

2.1. Certificates date or until they are revoked. The security of any PKI is thus One of the critical components of the PKI is a certificate: critically dependent on the timeliness of certificate revoca- a signed attestation binding a human-understandable tions. However, requesting a revocation is a surprisingly subject (a domain name or business name) to a public key. manual process, typically requiring an administrator to visit Certificates are signed by a CA, who in turn has its own cer- a website, fill in a form, provide a reason for the revocation, tificate, etc., forming a logical chain that terminates at a and wait for a representative at the CA to manually inspect self-signed root certificate. By issuing a certificate, a CA is the request before issuing the revocation. essentially asserting “this subject is the sole owner of the While it seems natural to assume that certificates are reis- private key corresponding to this certificate’s public key.” sued at precisely the moment the old certificate is revoked, Thus, if someone can prove knowledge of the private key in fact today’s PKI protocols make no such requirement. (e.g., by using it to sign a message), then this proves that As our study will demonstrate, reissues can happen before, this someone is the subject. As a result, anyone who has during, or after a revocation—or even without revoking the knowledge of the private key can pretend to be that subject. old certificate at all. To the best of our knowledge, we are the The assumption that only legitimate subjects hold the first to correlate revocations with reissues. corresponding private keys is central to the PKI’s ability to authenticate servers. Unfortunately, keys can be compro- 2.4. Heartbleed mised. For example, software vulnerabilities in SSL imple- Heartbleed is a buffer over-read vulnerability discovered in mentations have resulted in predictable keys22 or the ability OpenSSL15 versions 1.0.1 (released March 14, 2012) through to read sensitive server-side data.6 1.0.1f. The vulnerability stems from a bug in OpenSSL’s When a private key is compromised, a responsible admin- implementation of the TLS Heartbeat Extension.17 The istrator must do at least three things: first, the administrator intended functionality of TLS Heartbeat is to allow a cli- must patch the vulnerable software. Second, because the ent to test a secure communication channel by sending a certificate’s private key has been compromised, the admin- “heartbeat” message consisting of a string and the 16-bit istrator must generate a new key pair and ask their CA to reis- payload_length of this string. Unfortunately, vulner- sue a new certificate with this new private key. However, the able OpenSSL versions fail to check that the payload_ old, compromised certificate would still exist, and it could length supplied by the client matches the length of the be used by a malicious party to undetectably impersonate provided string. This allows a malicious client to craft a the website. Thus, there is a critical third step an admin- heartbeat message containing a 1-byte string and 216 − 1 as istrator must do in response to a key compromise: revoke the payload_length. In this case, OpenSSL will allocate the old certificate. It is important to note that the final two a 64KB block of heap memory, memcpy() 64KB of data steps necessarily involve the CA, and many CAs charge for into it, starting with the 1-byte string, and finally send the these actions, which may lead to perverse incentives for site contents of the entire buffer to the client. This allows the owners. malicious client to read up to 216 − 2 bytes of the server’s heap memory, although the client cannot chose which 2.2. Certificate reissue memory is read. When a website stops using a certificate—for instance, By repeatedly exploiting Heartbleed, an attacker can because the certificate has been compromised, or because extract sensitive data from the server, including SSL pri- it expired—they must obtain a new certificate. This proc- vate keys.19 To make matters worse, OpenSSL does not log ess is referred to as reissuing the certificate. To do so, the heartbeat messages, giving attackers free reign to undetect- system administrator must contact the CA who signed their ably exploit Heartbleed. Given the severity and undetectable certificate and request a new certificate and signature. In nature of Heartbleed, site operators were urged to immedi- cases where the private key may have been compromised, ately update their OpenSSL software and revoke and reissue the administrator should also choose a new public/private their certificates.3 key pair (since reissuing the certificate with the old public Why study Heartbleed? Heartbleed was first discovered key does nothing to mitigate attacks that leverage the leaked by Neel Mehta from Google on March 21, 2014. On April 7, private key). the bug became public and the OpenSSL project released a patched version (1.0.1g) of the OpenSSL library.8 2.3. Certificate revocation The significance of this timeline, and of Heartbleed A certificate revocation is another signed attestation from a in general, is that it represents a point in time after which CA, which essentially states “this certificate should no lon- the administrators of all vulnerable servers should have (1) ger be considered valid.” CAs are responsible for making patched their server, (2) revoked their old certificate, and revocations available for download, and typically do so with (3) issued a new one. The scope of this vulnerability—it is Certificate Revocation Lists (CRLs). Browsers (are supposed estimated that up to 17% of all HTTPS web servers were to) download CRLs to check if a presented certificate has vulnerable13—makes it an ideal case study for evaluating been revoked; the longer an administrator waits to revoke large-scale properties of SSL security in the face of private compromised certificates, the longer users are susceptible key compromise. As a result, Heartbleed acts as a natural to man-in-the-middle attacks. experiment, allowing us to measure how completely and The PKI uses a default-valid model, where potentially quickly administrators took steps to secure their keys. While compromised certificates remain valid until their expiration such events are (sadly) not uncommon,22 the intense press

110 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3

coverage surrounding Heartbleed reduces the likelihood After validation, we are left with 628,692 leaf certificates that administrators failed to take action because they were (40.0% of all certificates advertising Alexa domains and 3.2% unaware of the vulnerability. of all certificates). We refer to this set of certificates as the Leaf Set, each of which has a valid chain. We refer to the set 3. DATA AND METHODS of all CA certificates on these chains (not including the leaf We now describe the data sets that we collected and our certificates) as the CA Set, which contains 910 unique certifi- methodology for determining a host’s SSL certificate, when cates. The Leaf Set certificates cover 166,124 (16.6%) of the it was in use, if and when the certificate was revoked, and Alexa Top-1M domains. This is the set of certificates and if the host was (or is still) vulnerable to the Heartbleed bug. chains that we use in the remainder of the paper.

3.1. Certificate data source 3.3. Collecting CRLs We obtain our collection of SSL certificates from (roughly) To determine if and when certificates were revoked, we weekly scans of the entire IPv4 address space made available extracted the CRL URLs out of all Leaf Set certificates. We by Rapid7.16 We use scans collected between October 30, found 626,659 (99.7%) of these certificates to include at 2013 and April 28, 2014. There are a total of 28 scans during least one well-formed, reachable CRL URL. For certificates this period, giving an average of 6.7 days (with a minimum that included multiple CRL URLs, we included them all. We of 3 days and maximum of 9 days) between successive scans. found a total of 1,386 unique CRL URLs (most certificates The scans found an average of 26.9 million hosts respond- use a unified CRL provided by the signing CA, so the small ing to SSL handshakes on port 443 (an average of 9.12% of number of CRLs is not surprising). We downloaded all of the entire IPv4 address space). Across all of the scans, we these CRLs on May 6, 2014, and found 45,268 (7.2%) of the observed a total of 19,438,865 unique certificates (includ- Leaf Set certificates to be revoked. ing all leaf and CA certificates). In the sections below, we We also collected the CRL URLs for all certificates in the describe how we filtered and validated this data set; an over- CA set. We found that 884 (97.1%) of the certificates in the view of the process is provided in Figure 1. CA Set included a reachable CRL; the union of these URLs comprised 246 unique reachable URLs. We downloaded 3.2. Filtering data these CRLs on May 6, 2014, as well. We found a total of seven To focus on web destinations that are commonly accessed CA certificates that were revoked, which invalidated 60 cer- by users, we use the Alexa Top-1M domains1 as observed on tificates in the Leaf Set (< 0.01%). April 28, 2014. We first extract all leaf (non-CA) certificates that advertise a Common Name (CN) that is in one of the 3.4. Inferring the Heartbleed vulnerability domains in the Alexa list (e.g., we would include certificates Finally, we wish to determine if a site was ever vulnerable to for facebook.com, www.facebook.com, as well as *.dev.face- the Heartbleed OpenSSL vulnerability (and if it continued book.com). This set represents 1,573,332 certificates (8.1% to be vulnerable at the end of the study). Doing so allows of all certificates). us to reason about whether the site operators should have Unfortunately, despite leaf certificates having a CN in the reissued their SSL certificate(s) and revoked their old one(s). Alexa list, many may not be valid (e.g., expired certificates, Determining if a host is currently vulnerable to Heartbleed forged certificates, certificates signed by an unrecognized is relatively easy, as one can simply send an improperly-for- root, etc.). We removed these invalid certificates4 by running matted SSL heartbeat message with a payload_length of openssl verify on each certificate (and its correspond- 0 to test for vulnerability without exfiltrating any data.6 ing chain). We configure OpenSSL to trust the root CA certifi- However, determining if a site was vulnerable in the cates included by default in the OS X 10.9.2 root store12; this past—but has since updated their OpenSSL code—is more includes 222 unique root certificates. challenging. We observe that only three of the common TLS

Figure 1. Workflow from raw scans of the IPv4 address space to valid certificates (and corresponding CRLs) from the Alexa Top-1M domains. The Rapid7 data after February 5, 2014 did not include the intermediate (CA) certificates, necessitating additional steps and data to perform validation.

Input data Filtering Final CRLs Rapid7 Top-1M domains Valid chains certiﬁcates CRL URLs Revoked certs

4,313,480 910 CA Set October 30, 2013 CA certificates CA certificates 246 unique 7 revoked 910 CRL URLs CA certificates to CA certificates February 5, 2014 9,640,973 1,212,837 477,557 leaf certificates leaf certificates leaf certificates Leaf Set 1386 unique 45,268 revoked 628,692 February 10, 2014 CRL URLs leaf certificates 360,495 leaf certificates to 3,240,205 151,135 leaf certificates leaf certificates leaf certificates as of April 28, 2014 May 6, 2014

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 111 research highlights

implementations have ever supported SSL Heartbeats17: and (c) we observe at least one IP address switch from the old OpenSSL,15 GnuTLS,20 and Botan.2 Thus, if a host supports certificate to the new. We define the date of the certificate the SSL Heartbeat extension, we know that it is running one reissue to be the date of the certificate’s death. For the sake of these three implementations. Botan is targeted for client- of clarity, we refer to the old certificate that was replaced as side TLS, and we know of no popular web server that uses the the retired certificate. Botan TLS library. GnuTLS has support for the SSL Heartbeat Certificate revocation: A certificate is revoked if its serial extension, but it is not enabled by default. Furthermore, number appears in any of the certificate’s CRLs. The date of GnuTLS supports the Max Fragment Length SSL extension,7 revocation is provided in the CRL entry. which is enabled by default, while OpenSSL has never sup- In Figure 3, we present the number of certificate births, ported this extension. Thus, if we observe a host that sup- deaths, reissues, and revocations per day over time (please ports SSL Heartbeat but not the Max Fragment Length, we note the y-axis is in log scale). Births almost always outnumber declare that host to have been running a vulnerable version deaths, meaning that the total number of certificates in-the- of OpenSSL. wild is growing. Furthermore, we see an average of 29 certifi- To collect the list of sites that were ever vulnerable to cate revocations per day before Heartbleed; after Heartbleed, Heartbleed, we extracted the IP addresses in the April 28, this jumps to an average of 1,414 revocations per day. 2014 Rapid7 scan that were advertising a certificate with a CN in the Alexa Top-1M list. We found 5,951,763 unique 4.2. Server patching IP addresses in this set. We then connected to these IP We present a brief analysis of the number of certificates addresses on port 443, determined the SSL extensions that hosted by machines that were ever vulnerable to Heartbleed. the host supported, and checked whether the host was still Of the 428,552 leaf certificates that were still alive on the last vulnerable to the Heartbleed vulnerability. scan, we observe 122,832 (28.6%) of them advertised by a host that was likely vulnerable to Heartbleed at some point 4. ANALYSIS in time. These certificates come from 70,875 unique Alexa We now examine the collected SSL certificate data, begin- Top-1M domains. Of these certificates, 11,915 (from 10,366 ning with a few definitions. unique domains) were on hosts that were still vulnerable at the time of our crawl (April 30, 2014). This result dem- 4.1. Definitions onstrates that even in the wake of a well-publicized, severe We are concerned with the evolution of SSL certificates (i.e., security vulnerability, around 10% of vulnerable sites did not when are new certificates created, old ones retired, etc.). To address the issue three weeks after the fact. aid in understanding this evolution, we define the following notions: Certificate birth: We define the birth of an SSL certificate Figure 2. Example of lifetime, for certificates for m.scotrail.co.uk. All to be the date of the first scan where we observed any host hosts except one switch to a new certificate after February 10, 2014. advertising that certificate. 20 Certificate death: Defining the death of a certificate is 18 16 more complicated, as we observe a number of instances 14 tificate where many hosts advertise a given certificate, and then 12 10 all but a few of the hosts switch over to a new certificate 8 Inferred reissue date (presumably, the site intended to retire the old certifi- tising cer 6 4 Old certificate adver cate, but failed to update some of the hosts). To handle Number of unique hosts 2 New certificate 0 these cases, we calculate the maximum number of hosts 11/2013 12/2013 01/2014 02/2014 03/2014 04/2014 05/2014 that were ever advertising each certificate. We then define Date the death of an SSL certificate to be the last date that the number of hosts advertising the certificate was above 10% of that certificate’s maximum. The 10% threshold prevents us from incorrectly classifying certificates that are Figure 3. The rate of certificate births, deaths, reissues, and revocations increased sharply after the Heartbleed announcement. still widely available as dead, even if the certificate has been reissued. Birth Death Reissue Revoke 100,000 An example of certificate lifetime for m.scotrail.co.uk is Heartbleed shown in Figure 2. All hosts except one switch to a new certif- 10,000 icate after February 10, 2014; this lone host finally switches on April 28, 2014. In this case, we would consider the death 1000

date of the old certificate to be February 10, 2014. 100 Based on these definitions, we can now define the notion of a certificate reissue and revocation: 10 Certificate reissue: We consider a certificate to be reissued Number of certificates/day 1 if the following three conditions hold: (a) we observe the cer- 11/2013 12/2013 01/2014 02/2014 03/2014 04/2014 05/2014 tificate die, and (b) we observe a new certificate for the same Date Common Name born within 10 days of the certificate’s death,

112 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3

In Figure 4, we present the fraction of domains that have in the three weeks following the announcement; this is 8.9% at least one SSL host that was ever vulnerable to Heartbleed of all certificates that were alive at the time Heartbleed was (or still was as of April 30, 2014). We can observe a slight announced. Figure 5 examines the fraction of sites that increase in likelihood of ever being vulnerable for the most have at least one Heartbleed-induced certificate reissue, as popular sites, but the distribution quickly stabilizes. The a function of Alexa rank; we can observe a strong correlation increased likelihood of being vulnerable is likely because with Alexa rank. Higher-ranked sites are much more likely to these sites have larger numbers of hosts. This trend is mir- have reissued at least one certificate due to Heartbleed (even rored in the hosts that are still vulnerable on April 30, 2014. though they are only slightly more likely to have been vulnerable, as observed in Figure 4). This result complements 4.3. Certificate reissues previous studies’ findings that more popular websites often We now examine the reissuing of SSL certificates in the exhibit more sound administrative practices.5, 9 wake of Heartbleed. Not all SSL certificate reissues that we Reissues with same key. System administrators who be- observe following Heartbleed’s announcement are due to the lieve that their SSL private key may have been compromised Heartbleed vulnerability. In particular, reissues can happen should generate a new public/private key pair when reissu- for at least two other reasons: first, the old certificate could ing their certificate. We now examine how frequently this is be expiring soon. For example, before Heartbleed, we observe done, both in the case of normal certificate reissues and for that 50% of certificates are reissued within 60 days of their Heartbleed-induced reissues. expiry date. Second, a site may periodically reissue certifi- Before Heartbleed, we observe that reissuing a certificates as a matter of policy (regardless of expiration date). cate using the same key pair is quite common; up to 53% of For example, we observed that Google typically reissues the all reissued certificates do so. This high-level of key reuse google.com certificate every two weeks. is at least partially due to system administrators re-using In this study, we wish to distinguish a Heartbleed-induced the same Certificate Signing Request (CSR) when request- certificate reissue from a reissue that would have happened ing the new certificate from their CA. After Heartbleed, we anyway. We define a certificate reissue to be Heartbleed- observe a significant drop in the frequency of reissuing cer- induced if all three of the following conditions hold: tificates with the same key, that is, sites are generating a new key pair more frequently. However, if we focus on the 1. The date of reissue was on or after April 7, 2014 (the day Heartbleed-induced reissues, we observe that a non-trivial Heartbleed was announced). fraction (4.1%) of these certificates are reissued with the 2. The certificate that is reissued was not going to expire same key (thereby defeating the purpose of reissuing the for at least 60 days. This eliminates certificates that certificate). In fact, we observe a total of 912 such certifi- were likely to be reissued in the near future anyway. cates coming from 747 distinct Alexa domains; these cer- 3. We do not observe more than two other reissues for tificates may represent cases where administrators believe certificates with that CN in the time before Heartbleed. they have correctly responded to Heartbleed, but their cer- This implies that certificates with that name do not typi- tificates remain as vulnerable as if they had not reissued cally get reissued more than once every three months. at all. Vulnerable certificates. Finally, we examine the certifi- Thus, for the examples discussed so far, we do not con- cates that should have been reissued (regardless of whether sider the reissue of the retired certificate in Figure 2 to be they actually were); we refer to these certificates as vulner- Heartbleed-induced (as it happened before Heartbleed), able certificates. We declare a certificate to be vulnerable if and we do not consider any of the google.com reissues to be the following three conditions hold: Heartbleed-induced (because we observed a total of 12 reissues of that certificate prior to Heartbleed). 1. Its date of birth was before April 7, 2014. Heartbleed-induced reissues. Overall, we observe 36,781 2. It has not expired as of April 30. certificate reissues that we declare to be Heartbleed-induced 3. It was advertised by at least one host that was (or is) vulnerable to Heartbleed.

Figure 4. Fraction of domains that have at least one host that was ever vulnerable to Heartbleed as a function of Alexa rank, as well as Figure 5. Fraction of domains that have at least one Heartbleed- domains that continued to be vulnerable at the end of the study. induced reissue/revocation as a function of Alexa rank.

0.6 0.3 Was ever vulnerable Reissue 0.5 Was still vulnerable on April 30, 2014 0.25 Revocation

0.4 0.2

0.3 0.15

0.2 tbleed-induced 0.1 action of domains action of sites with reissue/revocation Fr hear

0.1 Fr 0.05

0 0 0 200k 400k 600k 800k 1M 0 200k 400k 600k 800k 1M Alexa site rank (bins of 10,000) Alexa site rank (bins of 10,000)

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 113 research highlights

In other words, these certificates are vulnerable because who is responsible, these weekend delays are problematic their private keys could have been stolen by attackers. for online security, since vulnerabilities (and the attackers Overall, we find 107,712 vulnerable certificates. Of these, who exploit them) do not take weekends off. only 28,652 (26.7%) have been reissued as of April 30. The Heartbleed-induced revocations. Similar to certificate remaining 79,060 (73.3%) vulnerable certificates that have reissues, not all certificate revocations after April 7, 2014 not been reissued come from 55,086 different Alexa Top-1M are necessarily due to Heartbleed (e.g., the site could have domains. Thus, the vast majority of SSL certificates that exposed their private key due to a different vulnerability). were potentially exposed by the Heartbleed bug remain in- We therefore define a Heartbleed-induced revocation to be a use over three weeks after the vulnerability was announced. certificate revocation where the certificate had a Heartbleed- induced reissue (see Section 4.3). 4.4. Certificate revocation Overall, we observe 14,726 Heartbleed-induced revoca- We now turn to investigating certificate revocation before, tions; this corresponds to 40% of all Heartbleed-induced during, and after the revelation of Heartbleed. Recall that it reissued certificates. Thus, 60% of all certificates that were is critical that a vulnerable certificate be revoked: even if a reissued due to Heartbleed were not revoked, implying that, site reissues a new certificate, if an attacker gained access if the certificate’s private key was actually stolen, the attacker to the vulnerable certificate’s private key, then that attacker still would be able to impersonate the victim without any cli- will be able to impersonate the owner until either the cer- ents being able to detect it. tificate expires or is revoked. We study both revocation and Figure 5 presents the fraction of sites that have at least expiration here, and correlate them with rates of reissue. one Heartbleed-induced certificate revocation, as a function Overall revocation rates. Figure 3 shows the number of of Alexa rank. Similar to reissues, sites with high rank are certificate revocations over time. As noted above, the average slightly more likely to revoke. Ideally, the two lines in Figure 5 jumps from 29 revocations per day to 1,414 post-Heartbleed. should be coincident, that is, all sites reissuing certificates However, the spike on April 16, 2014 is somewhat mislead- due to Heartbleed should also have revoked the retired cer- ing, as it was largely due to the mass-revocation of 19,384 tificates. This result highlights a serious gap in security best- CloudFlare certificates.18 practices across all of the sites in the Alexa Top-1M. To mitigate this issue, we plot in Figure 6 the number Finally, we examine revocation delay, or the number of unique domains that revoked at least one certificate over of days between when a certificate is reissued and it is time. We make three interesting observations: First, the mag- revoked. Figure 7 presents the cumulative distribution of nitude of the Heartbleed-induced spike is greatly reduced, the revocation delay for both Heartbleed-induced and non- but we still observe an up-to-40-fold increase in the number Heartbleed-induced revocations. To make the distributions of domains issuing revocations per day. Second, we observe comparable, we only look at differences between –10 and that the number of domains issuing revocations falls closer 10 days (recall that Heartbleed-induced reissues and revo- to its pre-Heartbleed level by April 28, suggesting that within cations can only occur after April 7, 2014, limiting that 3 weeks most of the domains that will revoke their certificate in distribution). We observe that Heartbleed-induced revoca- direct response to Heartbleed already have. tions appear to happen slightly more quickly, though not Third, we observe three “dips” in the post-Heartbleed to the extent one might expect, given the urgent nature of revocation rate on April 13, April 20, and April 27—all the vulnerability. We also observe that revocation almost weekends, indicating that far fewer revocations occur on always happens after reissue, which makes sense, since this the weekend relative to the rest of the week. This periodic- preserves the availability of HTTPS websites. This result ity can also be (less-easily) observed in the pre-Heartbleed contradicts previous assumptions5 that revocations and time frame. It is reasonable to assume revocations dip on reissues occur simultaneously. weekends because humans are involved in the revocation Expirations are not enough. To demonstrate how long the process, however it is not clear who is responsible for the effects of the Heartbleed vulnerability will be felt if sites do delays: is it site administrators or CRL maintainers at CAs not revoke their vulnerable certificates, we analyze vulner- (or both) who are not working on weekends? Regardless of able certificates that, by the end of our data collection, were

Figure 6. The rate of domains revoking certificates spiked after Figure 7. Heartbleed-induced revocations were issued slightly faster Heartbleed, but dropped closer to normal after three weeks. than other revocations.

1200 1 Heartbleed Non-Heartbleed-induced 1000 0.8 Heartbleed-induced

800 e tiﬁcates 0.6 Weekends 600 0.4

400 Cumulativ

200 0.2 fraction of cer Number of domains/day 0 0 03/06 03/13 03/20 03/27 04/03 04/10 04/17 04/24 05/01 −10 −5 0 5 10 Date Days from reissue to revocation

114 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3

reissued but never revoked. Although we find that 60% of the thorough response. Interestingly, while EV certificates were certificates expire within a year, there are vulnerable certifi- revoked more quickly, their non-EV counterparts caught cates that are valid for up to 5 years after Heartbleed was an- up within 10 days; however, EV certificates were reissued nounced. In fact, 10% of the vulnerable certificates still had both more quickly and more thoroughly. We expect that over 3 years of validity remaining. We conclude from this the underlying cause of this observation is a self-selection that, given the meager rates of revocation, it would be helpful effect, that is, security-conscious sites are more likely to for CAs to shift to shorter expiry times in their certificates. seek out EV certificates in the first place. Nonetheless, there Reissues and revocation speed. Next, we examine how are still many vulnerable EV certificates that have not been quickly sites responded to Heartbleed. Figure 8 shows the fraction reissued three weeks after the event (67%) and that have not of vulnerable certificates that were not reissued or revoked over been revoked three weeks after (87%). the three weeks following the Heartbleed announcement. In this figure, the initial y values do not all start at 1.0 for reis- 5. CONCLUDING DISCUSSION sues: this is because, with the coarse granularity of our data, we In this paper, we studied how SSL certificates are reissued know the range of time during which some certificates were and revoked in response to a widespread vulnerability, reissued, but not the precise day. We therefore provide the Heartbleed, that enabled undetectable key compromise. We most optimistic possibility: if we know a certificate was reissued conducted large-scale measurements and developed new between days d and d + k, we assume it was reissued on day d. methodologies to determine how the most popular one mil- This figure presents a bleak view of howthoroughly lion domains reacted to this vulnerability in terms of certifi- sites revoke and reissue their certificates (note that the cate management, and how this impacts security for clients. y-axis begins at 0.60). Three weeks after the revelation of We found that the vast majority of vulnerable certificates Heartbleed, over 87% of all certificates we found to be vul- have not been reissued. Further, of those domains that reis- nerable were not revoked, and over 73% of them were not sued certificates in response to Heartbleed, 60% did not reissued. We also found that the revocation rate follows a revoke their vulnerable certificates—if they do not eventu- pattern previously observed in earlier studies on the spread ally become revoked, 20% of those certificates will remain of patches14, 22: there is an exponential drop-off, followed by valid for two or more years. The ramifications of these find- a gradual decline. This behavior is even more pronounced ings are alarming: Web browsers will remain potentially when looking farther beyond the Heartbleed announce- vulnerable to malicious third parties using stolen keys for ment: 16 weeks after the announcement, there were still a long time to come. Additionally, we found that domains 86% who had not revoked and 70% who had not reissued. with EV certificates performed only marginally better than Extended validation certificates. Recall that one of the ma- other domains with respect to reissuing and revocation. jor roles of a CA is to validate the identity of the subjects who Our results are, in some ways, in line with previous stud- purchase certificates. Extended Validation (EV) certificates ies on the rates at which administrators patched vulner- are a means by which CAs can express that this identity- able software22—for instance, revocation rates followed a verification process has followed (presumably) more strin- sharp exponential drop-off shortly after the vulnerability gent criteria. Many browsers present EV certificates differently was made public, and tapered off soon thereafter. However, in the address bar. unlike software bugs, we find that the vast majority of cer- EV certificates are standard X.509 certificates that are not, tificates remain vulnerable to attacks, as they have still not in and of themselves, more secure, but the rationale is that been reissued or revoked. These findings indicate that the with a more thorough verification process by the CAs, these current practices of certificate management are misaligned certificates deserve greater trust. That said, there remains with what is necessary to secure the PKI. concern as to whether this trust is well-placed. We close by investigating the rate at which vulnerable EV certificates were 5.1. Surveying system administrators revoked and reissued as compared all certificates overall. To help better understand the reasons behind the lack of Overall, Figure 8 shows EV certificates follow similar prompt certificate reissues and revocations, we informally trends to the entire corpus, with a slightly faster and more surveyed a few systems administrators. We asked what steps they had taken in response to Heartbleed: did they patch, reissue, and revoke, and if not, then why not? We received Figure 8. Many vulnerable certificates were not revoked and reissued seven responses. Most reported manually patching their after Heartbleed (note that the y-axis does not begin at zero). systems, but some relied on managed servers or automatic 1 updates and therefore took no Heartbleed-specific steps. 0.95 Not revoked (All)

tiﬁcates There was some variance in when patches were applied, 0.9 Not revoked (EV) due to a combination of scheduled reboots and delayed 0.85 responses from vendors, but the majority of patches were 0.8 applied quickly. 0.75 Not reissued (All) For revoking and reissuing, however, we saw a wide spec-

not revoked/reissued 0.7 Not reissued (EV) ac. of vulnerable cer trum of behavior. The few who reissued and revoked did

Fr 0.65 04/07 04/11 04/15 04/19 04/23 04/27 so within 48 hours. Many neither revoked nor reissued; a Date common reason provided was that the vulnerable hosts were not hosting sensitive data or services. Along similar

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 115 research highlights

lines, others reported having reissued the certificate but not 5.4. Open source revoking, explaining that the certificate is only for internal Our analysis relied on both existing, public sources of data use. Finally, others reported that they did not perceive reis- and those we collected ourselves. We make all of our data suing and revoking as important because they had patched and our analysis code available to the research community quickly after the bug was publicly announced (recall, how- at https://securepki.org. ever, that the vulnerability was introduced over 2 years prior).

Our results from this small survey should be viewed References anecdotally—more extensive surveys on certificate admin- 1. Alexa Top 1 Million Domains. http:// end-to-end measurement of istration would be an important area of future work6— s3.amazonaws.com/alexa-static/top- certificate revocation in the web’s 1m.csv.zip. PKI. In ACM Internet Measurement but they do shed light on the root causes of why revoking 2. Botan SSL Library. http://botan. Conference (IMC) (2015). and reissuing are not on equal footing with patching. randombit.net. 12. Mac OS X 10.9.2 Root Certificates. 3. CERT Vulnerability Note VU#720951: http://support.apple.com/kb/ While administrators almost universally understand the OpenSSL TLS heartbeat extension HT6005. read overflow discloses sensitive 13. Mutton, P. Half a million widely importance of patching after a vulnerability, many do not information. http://www.kb.cert.org/ trusted websites vulnerable to appreciate or know about the importance of revoking and vuls/id/720951. heartbleed bug, 2014. http://news. 4. Chung, T., Liu, Y., Choffnes, D., Levin, D., netcraft.com/archives/2014/04/08/ reissuing certificates with new keys. Of those administra- Maggs, B.M., Mislove, A., Wilson, C. half-a-million-widely-trusted- tors who do understand the importance, some reported Measuring and applying invalid SSL websites-vulnerable-to-heartbleed- certificates: The silent majority. bug.html. push-back from others who perceived the process as being In ACM Internet Measurement 14. Nappa, A., Johnson, R., Bilge, L., overly complex. In sum, this points to the need for broader Conference (IMC) (2016). Caballero, J., Dumitras, T. The attack 5. Durumeric, Z., Kasten, J., Bailey, M., of the clones: A study of the impact of education on the treatment of certificates, and perhaps Halderman, J.A. Analysis of the shared code on vulnerability patching. more assistance from CAs to help ensure that all the pre- HTTPS certificate ecosystem. In IEEE Symposium on Security and In ACM Internet Measurement Privacy (S&P) (2015). scribed steps are taken. Conference (IMC) (2013). 15. OpenSSL Project. https://www. 6. Durumeric, Z., Kasten, J., Li, F., openssl.org. Amann, J., Beekman, J., Payer, M., 16. Rapid7 SSL Certificate Scans. https:// 5.2. Lessons learned Weaver, N., Halderman, J.A., Paxson, V., scans.io/study/sonar.ssl. Our results suggest several changes to common PKI prac- Bailey, M. The matter of Heartbleed. 17. Seggelmann, R., Tuexen, M., Williams, M. In ACM Internet Measurement Transport Layer Security (TLS) and tices that may improve security. First, low revocation rates Conference (IMC) (2014). Datagram Transport Layer Security and long expiration dates form a dangerous combination. 7. Eastlake, D III. Transport Layer (DTLS) Heartbeat Extension, Feb. Security (TLS) Extensions: Extension 2012. IETF RFC-6520. Techniques that automate revocation would vastly reduce Definitions, Jan. 2011. IETF RFC- 18. Sullivan, N. The Heartbleed 6066. Aftermath: all CloudFlare certificates the period during which clients are vulnerable to malicious 8. Grubb, B. Heartbleed disclosure revoked and reissued, 2014. http:// third parties. Similarly, adopting short certificate expira- timeline: who knew what and when, blog.cloudflare.com/the-heartbleed- 21 2014. http://www.smh.com.au/it-pro/ aftermath-all-cloudflare-certificates- tion dates (as suggested by Topalovic et al. ) by default will security-it/heartbleed-disclosure- revoked-and-reissued. significantly reduce the validity period of vulnerable certifi- timeline-who-knew-what-and-when- 19. Sullivan, N. The Results of the 20140415-zqurk.html. CloudFlare Challenge, 2014. http:// cates. Second, mechanisms that enable simultaneous reis- 9. Holz, R., Braun, L., Kammenhuber, N., blog.cloudflare.com/the-results-of- sue-and-revoke for certificates will make it less likely that Carle, G. The SSL landscape – the-cloudflare-challenge. A thorough analysis of the X.509 PKI 20. The GnuTLS Transport Layer Security invalid certificates are accepted by clients.Third , we have using active and passive measurements. Library. http://www.gnutls.org. found that many domains continue to serve old, vulnerable In ACM Internet Measurement 21. Topalovic, E., Saeta, B., Huang, L.-S., Conference (IMC) (2011). Jackson, C., Boneh, D. Toward short- certificates even after they reissue. Given the large number 10. Huang, L.S., Rice, A., Ellingsen, E., lived certificates. InWeb 2.0 Security of certificates and hosts using them per domain in our data- Jackson, C. Analyzing forged SSL & Privacy (W2SP) (2012). certificates in the wild. InIEEE 22. Yilek, S., Rescorla, E., Shacham, H., set, we believe administrators would benefit from tools that Symposium on Security and Privacy Enright, B., Savage, S. When private more easily track and validate the set of certificates they are (S&P) (2014). keys are public: Results from the 11. Liu, Y., Tome, W., Zhang, L., Choffnes, D., 2008 Debian OpenSSL vulnerability. using. Levin, D., Maggs, B.M., Mislove, A., In ACM Internet Measurement Schulman, A., Wilson, C. An Conference (IMC) (2009). 5.3. Future work This paper is, we believe, the first step towards understand- Liang Zhang, David Choffnes, Alan Dave Levin and Tudor Dumitras¸ ing the manual process of reissuing and revoking certifi- Mislove, and Christo Wilson ({liang, ({dml@cs, tdumitra@umiacs}.umd.edu), cates in the wake of a vulnerability. Several interesting open choffnes, amislove, cbw}@ccs.neu.edu), University of Maryland, College Park, Northeastern University, Boston, MA, USA. MD, USA. problems remain. Because our data focuses on the server and CA side of the PKI ecosystem, we are unable to draw Aaron Schulman ([email protected]), Stanford University, Stanford, CA, USA. any direct conclusions as to what clients experience. A host- centered measurement study would allow us to understand not only when revocations were added to CRLs, but when clients actually received the CRLs. Moreover, our study opens many questions as to why the certificate reissue and revocation processes are so extensively mismanaged. Our results reinforce previous findings that site popularity is correlated with good security practices, but even the highest ranked Alexa websites show relatively anemic rates of reissues and revocations. Understanding the root causes for why the PKI is mismanaged is an important step toward developing a secure infrastructure. © 2018 ACM 0001-0782/18/3 $15.00

116 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 CAREERS

Harvard John A. Paulson School of didates must demonstrate an outstanding schol- Application Details: Engineering and Applied Sciences arly record of research, exhibited by high-impact ˲˲Submit the following documents (in a single Tenured Professor in Computer Science peer-reviewed publications, a forward-looking, PDF) online via: https://faces.comp.nus.edu.sg vigorous research agenda and a demonstrated • A cover letter that indicates the position ap- The Harvard John A. Paulson School of Engineer- history of securing significant, competitive exter- plied for and the main research interests ing and Applied Sciences (SEAS) seeks applicants nal funding. • Curriculum Vitae for a position at the tenured level in the area of An exceptional researcher is sought to lead • A teaching statement Artificial Intelligence with Societal Impact, with and expand the research enterprise of our school • A research statement an expected start date of July 1, 2018. and contribute to the department’s growing data ˲˲Provide the contact information of 3 referees We seek a computer scientist whose research science academic program. All areas of data sci- when submitting your online application, or, ar- accomplishments include fundamental advanc- ence will be considered including data mining, range for at least 3 references to be sent directly es in AI and impact through applications that statistical machine learning, descriptive, predic- to [email protected] improve societal well-being. We seek candidates tive, and prescriptive analytics, cloud computing, ˲˲Application reviews will commence immedi- who have a strong research record and a commit- distributed databases, high performance com- ately and continue until the position is filled ment to undergraduate and graduate teaching puting, data visualization, or other areas involv- If you have further enquiries, please contact and training. We particularly encourage applica- ing the collection, organization, management, the Search Committee Chair, Weng-Fai Wong, at tions from historically underrepresented groups, and extraction of knowledge from massive, com- [email protected]. including women and minorities. plex, heterogeneous datasets. Data may include Computer Science at Harvard is enjoying a text, images, video, sensor and instrument data, period of substantial growth in numbers of stu- clickstream data, social media interactions, neu- Southern University of Science and dents and faculty hiring, and in expanded facili- roimaging data, genomics, proteomics, or me- Technology (SUSTech) ties. We benefit from outstanding undergraduate tabolomics data, etc. Professor Position in Computer Science and and graduate students, world-leading faculty, an The overarching responsibility is to expand Engineering excellent location, significant industrial collabo- the research portfolio of the SoIC. Full details of ration, and substantial support from the Harvard this position can be found at https://indiana.peo- The Department of Computer Science and Engi- Paulson School. For more information, see http:// pleadmin.com/postings/5287 neering (CSE, http://cse.sustc.edu.cn/en/), South- www.seas.harvard.edu/computer-science. Questions pertaining to this position can be ern University of Science and Technology (SUS- The associated Center for Research on Com- directed to Jeff Hostetler, Assistant to the Dean at Tech) has multiple Tenure-track faculty openings putation and Society (http://crcs.seas.harvard. [email protected] at all ranks, including Professor/Associate Profes- edu/), Berkman Klein Center for Internet & Soci- The School of Informatics and Computing is sor/Assistant Professor. We are looking for out- ety (http://cyber.harvard.edu), Data Science Ini- eager to consider applications from women and mi- standing candidates with demonstrated research tiative (https://datascience.harvard.edu/), and In- norities. Indiana University is an Affirmative Action/ achievements and keen interest in teaching. stitute for Applied Computational Science (http:// Equal Opportunity Employer. IUPUI is an Affirma- Applicants should have an earned Ph.D. de- iacs.seas.harvard.edu) foster connections among tive Action/Equal Opportunity Institution M/F/D/V. gree and demonstrated achievements in both computer science and other disciplines through- research and teaching. The teaching language at out the university. SUSTech is bilingual, either English or Putong- Candidates are required to have a doctoral de- National University of Singapore (NUS) hua. It is perfectly acceptable to use English in all gree in computer science or a related area. Sung Kah Kay Assistant Professorship lectures, assignments, exams. In fact, our exist- Required application documents include a ing faculty members include several non-Chinese cover letter, CV, a statement of research interests, The Department of Computer Science at the Na- speaking professors. a teaching statement, and up to three represen- tional University of Singapore (NUS) invites appli- As a State-level innovative city, Shenzhen has tative papers. Candidates are also required to cations for the Sung Kah Kay Assistant Professor- identified innovation as the key strategy for its submit the names and contact information for ship. Applicants can be in any area of computer development. It is home to some of China’s most at least three references. Applicants can apply science. This prestigious chair was set up in mem- successful high-tech companies, such as Huawei online at https://academicpositions.harvard.edu/ ory of the late Assistant Professor Sung Kah Kay af- and Tencent. SUSTech considers entrepreneur- postings/8037. ter his untimely demise early in his career at NUS. ship as one of the main directions of the univer- We are an equal opportunity employer and Candidates should be early in their academic ca- sity. Strong supports will be provided to possible all qualified applicants will receive consideration reers and yet demonstrate outstanding research new initiatives. SUSTech encourages candidates for employment without regard to race, color, re- potential, and strong commitment to teaching. with experience in entrepreneurship to apply. ligion, sex, sexual orientation, gender identity, The Department enjoys ample research fund- The Department of Computer Science and national origin, disability status, protected veteran ing, moderate teaching loads, excellent facilities, Engineering at SUSTech was founded in 2016. status, or any other characteristic protected by law. and extensive international collaborations. We have It has 17 professors, all of whom hold doctoral a full range of faculty covering all major research ar- degrees or have years of experience in overseas eas in computer science and boasts a thriving PhD universities. Among them, two were elected into IU School of Informatics and program that attracts the brightest students from the “1000 Talents” Program in China; three are Computing, IUPUI the region and beyond. More information is avail- IEEE fellows; one IET fellow. The department is Associate Dean for Research able at www.comp.nus.edu.sg/careers. expected to grow to 50 tenure track faculty mem- NUS is an equal opportunity employer that bers eventually, in addition to teaching-only pro- The Indiana University School of Informatics and offers highly competitive salaries, and is situated fessors and research-only professors. Computing (SoIC), IUPUI campus, invites appli- in Singapore, an English-speaking cosmopolitan SUSTech is a pioneer in higher education cations for a tenured associate or full professor in city that is a melting pot of many cultures, both reform in China. The mission of the University the growing field of data science (or related area) the east and the west. Singapore offers high-qual- is to become a globally recognized research uni- to fill the position of Associate Dean for Research. ity education and healthcare at all levels, as well versity which emphasizes academic excellence The appointment will begin August 1, 2018. Can- as very low tax rates. and promotes innovation, creativity and entre-

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 117 CAREERS

preneurship. Set on five hundred acres of wooded and can send all application materials, includ- Inquiries: landscape in the picturesque Nanshan (South ing confidential letters of recommendation, Search Committee Chair, Dr. Kazuo Hotate, Vice Mountain) area, the campus offers an ideal en- free of charge. To apply, visit https://apply.in- President & Professor. vironment for learning and research. SUSTech terfolio.com/45234. (Phone) +81-52-809-1821 (E-mail) hotate- is committed to increase the diversity of its fac- Swarthmore College actively seeks and wel- [email protected] ulty, and has a range of family-friendly policies comes applications from candidates with ex- The above documentation should be sent to: in place. The university offers competitive sala- ceptional qualifications, particularly those with Mr. Masashi Hisamoto ries and fringe benefits including medical insur- demonstrable commitments to a more inclusive Toyota Technological Institute ance, retirement and housing subsidy, which are society and world. Swarthmore College is an 2-12-1, Hisakata, Tempaku-ku, Nagoya, 468- among the best in China. Salary and rank will Equal Opportunity Employer. Women and mi- 8511 Japan commensurate with qualifications and experi- norities are encouraged to apply. ence. More information can be found at http:// Please write “Application for Principal Profes- talent.sustc.edu.cn/en sor” on the envelope. We provide some of the best start-up pack- Toyota Technological Institute The application documents will not be returned. ages in the sector to our faculty members, includ- Principal Professor ing one PhD studentship per year, in addition to a significant amount of start-up funding. Toyota Technological Institute has one opening Western Michigan University To apply, please provide a cover letter iden- for “Principal Professor” positions in its School Assistant/Associate Professor in Computer tifying the primary area of research, curriculum of Engineering. For more information, please re- Science vitae, and research and teaching statements, and fer to the following website: http://www.toyota-ti. forward them to [email protected]. ac.jp/english/employment/index.html. Applications are invited for a tenure-track position at the assistant or associate professor level in Research fields: Science and technology for the area of applied information security in the De- Swarthmore College advanced instrumentation and/or information partment of Computer Science at Western Michi- Computer Science Department processing gan University (Kalamazoo, MI) starting August Visiting Faculty Positions in Computer Science Examples: New devices and systems for advanced 2018 or January 2019. information processing, communication, and/ Applicants must have a Ph.D. in Computer The Computer Science Department invites ap- or sensing; Leading instrumentation technolo- Science or a closely related field. We are looking plications for multiple visiting positions at the gies for ultra-sensitive measurements and/or for candidates with expertise in applied informa- rank of Assistant Professor to begin Fall semester bio-medical studies and diagnosis; Science and tion security to support our new M.S. in Informa- 2018. technology of cyber-physical systems. tion Security. The program is offered fully online For the visiting position, strong applicants in and in cooperation with the Department of Busi- any area will be considered. Priority will be given Qualifications: A successful candidate must ness Information Systems. to complete applications received by February 15, have a Ph. D. degree or the equivalent in a rel- Successful candidates will be capable of es- 2018. evant field; he/she must possess outstanding tablishing an active research program leading Applications will continue to be accepted af- competence to promote world-class research to funding, supervising graduate students, and ter these dates until the positions are filled. program(s) as well as to conduct excellent teaching courses at both the undergraduate and The Computer Science Department currently teaching and research supervision for graduate graduate levels in information security. Other has eight tenure-track faculty and four visiting and undergraduate students, so as to fulfill his/ duties include development of undergraduate faculty. Faculty teach introductory courses as well her mission as a superb leader in research and and graduate courses, advising and service at as advanced courses in their research areas. We education. the University, College, Department and profes- have grown significantly in both faculty and stu- sional society levels. dents in the last five years. Presently, we are one Positions: Principal Professor Application screening will start immediately of the most popular majors at the College and The “Principal Professor” will serve as the head and the position will remain open until filled. expect to have over 70 Computer Science majors of a “unit laboratory,” that consists of the Prin- Successful candidates must earn their Ph.D. de- graduating this year. cipal Professor, one associate professor, and gree by the time of employment. QUALIFICATIONS: Applicants must have three post-doctoral fellows. A start-up grant of The Department has 260 undergraduates, teaching experience and should be comfortable about one hundred million Japanese yen (ca. 50 M.S. students and 45 Ph.D. students. Cur- teaching a wide range of courses at the introduc- one million US dollars) is available. In addition, rent active research areas include security, pri- tory and intermediate level. Candidates should a research budget of about ten million Japanese vacy, networks, embedded systems/internet additionally have a strong commitment to involv- yen (ca. one hundred thousand US dollars) will be of things, compilers, computational biology, ing undergraduates in their research. A Ph.D. in given each year to promote research programs for massive data analytics, scientific computing, Computer Science at or near the time of appoint- a period of five years. At the end of this five-year parallel computing, formal verification, paral- ment is required. The strongest candidates will term, the principal professor will be given a for- lel debugging, and data mining. More infor- be expected to demonstrate a commitment to mal evaluation. mation regarding Western Michigan Univer- creative teaching and an active research program sity, the College of Engineering and Applied that speaks to and motivates undergraduates Number of Positions Available: One Sciences and the Department of Computer from diverse backgrounds. Start date: April 1, 2019 or on the date of the earli- Science are available at http://www.wmich. APPLICATION INSTRUCTIONS: Applica- est convenience edu, http://www.wmich.edu/engineer, and tions should include a cover letter, vita, teach- http://wmich.edu/cs, respectively. ing statement, research statement, and three Documents: The Carnegie Foundation for the Advance- letters of reference, at least one (preferably two) 1. A curriculum vitae ment of Teaching has placed WMU among the of which should speak to the candidate’s teach- 2. A list of publications 76 public institutions in the nation designated as ing ability. In your cover letter, please briefly de- 3. Copies of 5 representative papers research universities with high research activity. scribe your current research agenda; what would 4. An outline of research and educational WMU is an Equal Opportunity/Affirmative Ac- be attractive to you about teaching diverse stu- accomplishments (about 3-pages) tion Employer. Minorities, women, veterans, in- dents in a liberal arts college environment; and 5. A future plan of educational and research dividuals with disabilities and all other qualified what background, experience, or interests are activities (about 3- pages) individuals are encouraged to apply. likely to make you a strong teacher of Swarth- 6. Names of two references, including phone To do so, please visit: http://wmich.edu/hr/ more College students. numbers and e-mail addresses jobs and provide a cover letter, curriculum vitae, This institution is using Interfolio’s Fac- 7. An application form (available on our statement of research goals, teaching statement, ulty Search to conduct this search. Applicants website) and names and contact information of at least to this position receive a free Dossier account Deadline: May 15, 2018 three references.

118 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 last byte

[CONTINUED FROM P. 120] When I show you a segment of a In one sense, it’s very practical. video and I ask what happens next, you Deep learning has been successful might be able to predict to some ex- not just because it works well, but tent, but not exactly; there are proba- also because it automates part of the bly several different outcomes that are ACM Journal on process of building and designing in- possible. So when you train a system to telligent systems. In the old days, ev- predict the future, and there are sev- erything was manual; you had to find eral possible futures, the system takes Computing and a way to express all of human knowl- an average of all the possibilities, and edge in a set of rules, which turns out that’s not a good prediction. Cultural Heritage to be extremely complicated. Even Adversarial training allows us to in the more traditional realm of ma- train a system where there are multiple chine learning, part of the system was correct outputs by asking it to make a trained, but most of it was still done prediction, then telling it what should ACM JOCCH publishes by hand, so for classical computer vi- have been predicted. One of the central sion systems, you had to design a way ideas behind this is that you train two papers of significant to pre-process the image to get it into neural networks simultaneously; there and lasting value in all a form that your learning algorithm is one neural net that does the predic- could digest. tion and there’s a second neural net areas relating to the that essentially assesses whether the With deep learning, on the other hand, prediction of the first neural net looks use of ICT in support you can train an entire system more or probable or not. less from end to end. of Cultural Heritage, Yes, but you need a lot of labeled You recently helped found the Partner- seeking to combine data to do it, which limits the number ship on AI, which aims to develop and of applications and the power of the share best practices and provide a plat- the best of computing system, because it can only learn what- form for public discussion. ever knowledge is present within your There are questions related to science with real labeled datasets. The more long-term the deployment and perception of attention to any reason for trying to train or pre-train a AI within the public and govern- learning system on unlabeled data is ment, questions about the ethics of aspect of the cultural that, as you said, animals and humans testing, reliability, and many other build models of the world mostly by things that we thought went beyond heritage sector. observation, and we’d like machines to a single company. do that as well, because accumulating massive amounts of knowledge about Thanks to rapid advances in the field, the world is the only they will eventually many of these questions are coming acquire a certain level of common sense. up very quickly. It seems like there’s a lot of excitement, but also a lot of ap- What about adversarial training, in prehension in the public about where which a set of machines learn together AI is headed. by pursuing competing goals? Humans make decisions under This is an idea that popped up a few what’s called bounded rationality. We years ago in Yoshua Bengio’s lab with are very limited in the time and effort Ian Goodfellow, one of his students at we can spend on any decision. We are the time. One important application is biased, and we have to use our bias predictions. If you build a self-driving because that makes us more efficient, car or any other kind of system, you’d though it also makes us less accurate. like that system to be able to predict To reduce bias in decisions, it’s better what’s going to happen next—to simu- to use machines. That said, you need late the world and see what a particular to apply AI in ways that are not biased, sequence of actions will produce with- and there are techniques being devel- out actually doing it. That would allow oped that will allow people to make For further information it to anticipate things and act accord- sure that the decisions made by AI sys- ingly, perhaps to correct something or tems have as little bias as possible. or to submit your plan in advance. manuscript, Leah Hoffmann is a technology writer based in Piermont, How does adversarial training address NY, USA. visit jocch.acm.org the problem of prediction in the presence of uncertainty? © 2018 ACM 0001-0782/18/3 $15.00

MARCH 2018 | VOL. 61 | NO. 3 | COMMUNICATIONS OF THE ACM 119 last byte

DOI:10.1145/3178314 Leah Hoffmann Q&A The Network Effect The developer of convolutional neural networks looks at their impact, today and in the long run.

DEEP LEARNING MIGHT be a booming And the last thing would be auton- field these days, but few people re- omous AI systems whose behavior is member its time in the intellectual not directly controlled by a person. In wilderness better than Yann LeCun, other words, they are designed not just director of Facebook Artificial Intelli- to do one particular task, but to make gence Research (FAIR) and a part-time decisions and adapt to different cir- professor at New York University. Le- cumstances on their own. Cun developed convolutional neural networks while a researcher at Bell How does the interplay work between Laboratories in the late 1980s. Now, research and product? the group he leads at Facebook is us- There’s a group called Applied Ma- ing them to improve computer vision, chine Learning, or AML, that works to make predictions in the face of un- closely with FAIR and is a bit more on certainty, and even to understand nat- the application side of things. That ural language. group did not exist when I joined Face- book, but I pushed for its creation, be- Your work at FAIR ranges from long- cause I saw this kind of relationship term theoretical research to applica- work very well at AT&T. Then AML be- tions that have real product impact. came a victim of its own success. There We were founded with the idea of of content, or if two people are likely to was so much demand within the com- making scientific and technological be friends with one another. pany for the platforms they were devel- progress, but I don’t think the Face- oping, which basically enabled all kinds book leadership expected quick re- What are some of the things going of groups within Facebook to use ma- sults. In fact, many things have had on at FAIR that most interest or ex- chine learning in their products, that a fairly continuous product impact. cite you? they ended up moving away from FAIR. In the application domain, our group It’s all interesting! But I’m person- Recently we reorganized this a little works on things like text understand- ally interested in a few things. bit. A lot of the AI capability is now being, translation, computer vision, im- One is marrying reasoning with ing moved to the product groups, and age understanding, video understand- learning. A lot of learning has to do with AML is refocusing on the advanced ing, and speech recognition. There are perceptions, which are relatively simple development of things that are close also more esoteric things that have had things that people can do without think- to research. In certain areas like com- an impact, like large-scale embedding. ing too much. But we haven’t yet found puter vision, there is a very, very tight good recipes for training systems to do collaboration, and things go back and This is the idea of associating every ob- tasks that require a little bit of reason- forth really quickly. In other areas that ject with a vector. ing. There is some work in that direc- are more disruptive or for which there Yes. You describe every object tion, but it’s not where we want it. is no obvious product, it’s more like, on Facebook with a list of numbers, Another area that interests me is ‘let us work on it for a few years first’. whether it’s a post, news item, photo, unsupervised learning—teaching ma- comment, or user. Then, you use op- chines to learn by observing the world, Let’s talk about unsupervised learning, erations between vectors to see if, say, say by watching videos or looking at which, as you point out elsewhere, is two images are similar, or if a person is images without being told what objects much closer to the way that humans ac-

likely to be interested in a certain piece are in these images. tually learn. [CONTINUED ON P. 119] RESEARCH OF FACEBOOK COURTESY IMAGE

120 COMMUNICATIONS OF THE ACM | MARCH 2018 | VOL. 61 | NO. 3 Dr. Jerald has recognized a great need in our community and filled it. The VR Book is a scholarly and comprehensive treatment of the user interface dynamics surrounding the development and application of virtual reality. I have made it required reading for my students and research colleagues. Well done!”

- Prof. Tom Furness, University of Washington, VR Pioneer