Pdf 2019, Tel Aviv, Israel, July 22-25, 2019 Zhang, C.-H

AI MATTERS, VOLUME 5, ISSUE 3 5(3) 2019 AI Matters

Annotated Table of Contents An AI assignment for building a Fake News Detec- tor Welcome to AI Matters 5(3) Amy McGovern, co-editor & Iolanda AI Education Matters: A First Intro- Leite, co-editor duction to Modeling and Learning us- Full article: http://doi.acm.org/10.1145/3362077.3362078 ing the Data Science Workﬂow Welcome and summary Marion Neumann Full article: http://doi.acm.org/10.1145/3362077.3362083 ACM SIGAI Activity Report Modeling and Learning using the Data Science Sven Koenig, Sanmay Das, Rose- Workﬂow mary Paradis, John Dickerson, Yolanda Gil, Katherine Guo, Benjamin Kuipers, AI Policy Matters Iolanda Leite, Hang Ma, Nicholas Mat- Larry Medsker tei, Amy McGovern, Larry Medsker, Todd Neller, Marion Neumann, Plamen Full article: http://doi.acm.org/10.1145/3362077.3362084 Petrov, Michael Rovatsos & David Stork Policy issues relevant to SIGAI Full article: http://doi.acm.org/10.1145/3362077.3362079 ACM SIGAI Activity Report Advancing Non-Convex and Con- strained Learning: Challenges and Opportunities Help Communities Solve Real-World Problems with AI – Become a Tech- Tianbao Yang novation Mentor! Full article: http://doi.acm.org/10.1145/3362077.3362085 Tara Chklovski Latest research trends in AI Full article: http://doi.acm.org/10.1145/3362077.3362080 Mentors needed to help families, schools and Considerations for AI Fairness for communities to learn, play, and create with AI People with Disabilities Shari Trewin, Sara Basson, Michael Events Muller, Stacy Branham, Jutta Trevi- ranus, Daniel Gruen, Daniel Hebert, Na- Michael Rovatsos talia Lyckowski & Erich Manser Full article: http://doi.acm.org/10.1145/3362077.3362081 Full article: http://doi.acm.org/10.1145/3362077.3362086 Upcoming AI events Contributed article AI Education Matters: Building a Fake News Detector The Intersection of Ethics and AI Michael Guerzhoy, Lisa Zhang & Annie Zhou Georgy Noarov Full article: http://doi.acm.org/10.1145/3362077.3362087 Full article: http://doi.acm.org/10.1145/3362077.3362082 Winning essay from the 2018 ACM SIGAI Student

1 AI MATTERS, VOLUME 5, ISSUE 3 5(3) 2019

Essay Contest • to publish in print on condition of acceptance by the editor Artiﬁcial Intelligence: The Societal • to digitize and post your article in the elec- Responsibility to Inform, Educate, tronic version of this publication and Regulate • to include the article in the ACM Digital Li- Alexander D. Hilton brary and in any Digital Library related ser- Full article: http://doi.acm.org/10.1145/3362077.3362088 vices • to allow users to make a personal copy of Winning essay from the 2018 ACM SIGAI Student the article for noncommercial, educational Essay Contest or research purposes

The Necessary Roadblock to Artiﬁ- However, as a contributing author, you retain cial General Intelligence: Corrigibility copyright to your article and ACM will refer requests for republication directly to you. Yat Long Lo, Chung Yu Woo & Ka Lok Ng Submit to AI Matters! Full article: http://doi.acm.org/10.1145/3362077.3362089 Winning essay from the 2018 ACM SIGAI Student We’re accepting articles and announce- Essay Contest ments now for the next issue. Details on the submission process are available at http://sigai.acm.org/aimatters. AI Fun Matters Adi Botea Full article: http://doi.acm.org/10.1145/3362077.3362090 AI generated Crosswords

Links

SIGAI website: http://sigai.acm.org/ Newsletter: http://sigai.acm.org/aimatters/ AI Matters Editorial Board Blog: http://sigai.acm.org/ai-matters/ Twitter: http://twitter.com/acm sigai/ Amy McGovern, Co-Editor, U. Oklahoma Edition DOI: 10.1145/3362077 Iolanda Leite, Co-Editor, KTH Sanmay Das, Washington Univ. in Saint Louis Alexei Efros, Univ. of CA Berkeley Join SIGAI Susan L. Epstein, The City Univ. of NY Yolanda Gil, ISI/Univ. of Southern California Students $11, others $25 Doug Lange, U.S. Navy For details, see http://sigai.acm.org/ Kiri Wagstaff, JPL/Caltech Beneﬁts: regular, student Xiaojin (Jerry) Zhu, Univ. of WI Madison Also consider joining ACM. Contact us: [email protected] Our mailing list is open to all.

Notice to Contributing Authors to SIG Newsletters

By submitting your article for distribution in this Special Interest Group publication, you hereby grant to ACM the following non-exclusive, per- petual, worldwide rights:

2 AI MATTERS, VOLUME 5, ISSUE 3 5(3) 2019

Contents Legend

Book Announcement

Ph.D. Dissertation Brieﬁng

AI Education

Event Report

Hot Topics

Humor

AI Impact

AI News

Opinion

Paper Precis´

Spotlight

Video or Image Details at http://sigai.acm.org/aimatters

3 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Welcome to AI Matters 5(3) Amy McGovern, co-editor (University of Oklahoma; [email protected]) Iolanda Leite, co-editor (Royal Institute of Technology (KTH); [email protected]) DOI: 10.1145/3362077.3362078

Issue overview Elimination for SAT and QSAT” (https://jair. org/index.php/jair/article/view/10942). This pa- Welcome to the third issue of the fifth volume per describes fundamental and practical re- of the AI Matters Newsletter. With this issue, sults on a range of clause elimination pro- we want to welcome our new SIGAI Execu- cedures as preprocessing and simplification tive Committee. Elections were completed this techniques for SAT and QBF solvers. Since its Spring, and we have a new leadership team publication, the techniques described therein in place. Sanmay Das of Washington Uni- have been demonstrated to have profound im- versity in St. Louis (former vice-chair) is the pact on the efficiency of state-of-the-art SAT new chair, Nicholas Mattei of Tulane Univer- and QBF solvers. The work is elegant and ex- sity the new vice-chair, and John Dickerson of tends beautifully some well-established theo- the University of Maryland the new secretary- retical concepts. In addition, the paper gives treasurer. Nicholas and John were formerly new emphasis and impulse to pre- and in- active as the appointed AI and Society and La- processing techniques - an emphasis that res- bor Market officers respectively. Sven Koenig onates beyond the two key problems, SAT and will transition into the role of past-chair, and QBF, covered by the authors. continue to serve on the EC in that role. We would also like to note that SIGAI and The new officers want to express their sincere AAAI will be jointly presenting a new annual thanks to Sven Koenig and to Rosemary Par- award for the best doctoral dissertation in AI. adis (the former secretary/treasurer) for the The award will be presented at AAAI, and wealth of novel initiatives they spearheaded in nominations for the inaugural award are due the last three years and the untiring energy by November 15, 2019. Please see http://sigai. they brought to their roles. SIGAI is deeply acm.org/awards/nominations.html for details and indebted to them! information on how to submit a nomination! We would like to mention that there has been a lot of activity in the space of significant awards This issue is full of great new articles and sto- in AI. The inaugural SIGAI Industry Award for ries for you! We open with the annual report of Excellence in Artificial Intelligence (AI) was SIGAI. We then bring you a story from a new presented at IJCAI 2019. The award went to way to teach kids and families about AI: Tech- the Real World Reinforcement Learning Team novation Familes’ AI challenge, which brings from Microsoft, for identification and develop- AI into the home by educating parents and ment of cutting-edge research on contextual- children about AI and providing an opportunity bandit learning that led to new decision sup- for them to prototype AI solutions to real-world port tools that were broadly integrated into a problems. They are seeking new mentors for broad range of Microsoft products. John Lang- this year’s challenges! ford and Tyler Clintworth received the award In our regular articles, Michael Rovatsos re- on behalf of the Microsoft team and presented ports on upcoming AI events and we have two a talk on the work at IJCAI. For more on this submissions for AI Education. First, Michael award, please see https://sigai.acm.org/awards/ Guerzhoy talks about building a fake news de- industry award.html tector. Second, Marion Neumann talks about We also congratulate Marijn Heule, Matti bringing AI and ML to a younger audience, Jarvisalo,¨ Florian Lonsing, Martina Seidl and much like CS for all, instead of focusing on se- Armin Biere who have been awarded the 2019 niors and graduate students. In the policy col- IJCAI-JAIR prize for their 2015 paper “Clause umn, Larry Medsker summarizes recent policies covering face recognition (how much data Copyright c 2019 by the author(s). should we record and share?), upcoming AI

4 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 regulation, and more. Our ﬁnal regular column Submit to AI Matters! is our AI crosswords from Adi Botea. Enjoy! Thanks for reading! Don’t forget to send We have a new regular column where we your ideas and future submissions to AI invite researchers to present latest research Matters! We’re accepting articles and an- trends in AI. In the inaugural article of this col- nouncements now for the next issue. De- umn, Tianbao Yang describes challenges and tails on the submission process are avail- opportunities of non-convex and constrained able at http://sigai.acm.org/aimatters. learning.

In our contributed articles, Shari Trein et al. describe some of the opportunities and risks Amy McGovern is co- across four emerging AI application areas: editor of AI Matters. She employment, education, public safety, and is a Professor of com- healthcare, identified in a workshop with par- puter science at the Uni- ticipants experiencing a range of disabilities. versity of Oklahoma and Finally, this issue features the second set of an adjunct Professor of winning essays from the 2018 ACM SIGAI meteorology. She directs Student Essay Contest. In addition to hav- the Interaction, Discovery, ing their essay appear in AI Matters, the con- Exploration and Adapta- test winners received either monetary prizes tion (IDEA) lab. Her re- or one-on-one Skype sessions with leading AI search focuses on ma- researchers. chine learning and data mining with applications to high-impact weather. Iolanda Leite is co-editor Special Issue: AI For Social Good of AI Matters. She is an Assistant Professor at the Recognizing the potential of AI in solv- School of Electrical En- ing some of the most pressing challenges gineering and Computer facing our society, we are excited to an- Science at the KTH Royal nounce that the next Newsletter of AI Mat- Institute of Technology in ters will be a special issue on the theme Sweden. Her research in- of “AI for Social Good.” We solicit arti- terests are in the areas of cles that discuss how AI applications and/or Human-Robot Interaction and Artificial Intelli- innovations have resulted in a meaning- gence. She aims to develop autonomous so- ful impact on a societally relevant prob- cially intelligent robots that can assist people lem, including problems in the domains of over long periods of time. health, agriculture, environmental sustainability, ecological forecasting, urban planning, climate science, education, social welfare and justice, ethics and privacy, and assistive technology for people with disabilities. We also encourage submissions on emerging problems where AI advances have the potential to influence a transfor- mative change, and perspective articles that highlight the challenges faced by current standards of AI to have a societal impact and opportunities for future research in this area. More details to be coming soon on http://sigai.acm.org/aimatters. Please get in touch with us if you have any questions!

5 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

ACM SIGAI Activity Report Sven Koenig (elected; ACM SIGAI Chair) Sanmay Das (elected; ACM SIGAI Vice-Chair) Rosemary Paradis (elected; ACM SIGAI Secretary/Treasurer) John Dickerson (elected; ACM SIGAI Labor Market Officer) Yolanda Gil (appointed; ACM SIGAI Past Chair) Katherine Guo (appointed; ACM SIGAI Membership and Outreach Officer) Benjamin Kuipers (appointed; ACM SIGAI Ethics Officer) Iolanda Leite (appointed; ACM SIGAI Newsletter Editor-in-Chief) Hang Ma (appointed; ACM SIGAI Information Officer) Nicholas Mattei (appointed; ACM SIGAI AI and Society Officer) Amy McGovern (appointed; ACM SIGAI Newsletter Editor-in-Chief) Larry Medsker (appointed; ACM SIGAI Public Policy Officer) Todd Neller (appointed; ACM SIGAI Education Activities Officer) Marion Neumann (appointed; ACM SIGAI Diversity Officer) Plamen Petrov (appointed; ACM SIGAI Industry Liaison Officer) Michael Rovatsos (appointed; ACM SIGAI Conference Coordination Officer) David Stork (appointed; ACM SIGAI Award Officer) DOI: 10.1145/3362077.3362079

Abstract ACM SIGAI also added two new officers this year to be able to serve its membership better, We are happy to present the annual activity namely Iolanda Leite from the Royal Institute report of the ACM Special Interest Group on of Technology (Sweden) as second newslet- AI (ACM SIGAI), covering the period from July ter editor-in-chief and Marion Neumann from 2018 to June 2019. Washington University in St. Louis (USA) as diversity officer, thus increasing diversity in The scope of ACM SIGAI consists of the the ACM SIGAI leadership committee by in- study of intelligence and its realization in creasing both the number of international of- computer systems (see also its web-site at ficers and the number of female officers and sigai.acm.org). This includes areas such also furthering the internationalization of the as ACM SIGAI newsletter. Marion will be covering diversity also as part of the ACM SIGAI autonomous agents, cognitive modeling, newsletter. ACM SIGAI started officers meet- computer vision, constraint programming, hu- ings at major AI conferences already in 2018 man language technologies, intelligent user and continued the new practice in 2019 (so interfaces, knowledge discovery, knowledge representation and reasoning, machine learn- far at the AAAI Conference), in addition to all- ing, planning and search, problem solving and officers teleconferences and a monthly ACM robotics. SIGAI leadership teleconference.

Members come from academia, industry and government agencies worldwide. ACM SIGAI Meetings recently added two new ACM SIGAI chap- ACM SIGAI decided to participate on a trial ters, namely one professional chapter in La- basis in ACM’s voluntary carbon-offset pro- guna Nigel (USA) and one student chapter at gram for conferences. Introduction of this the SRM Institute of Science & Technology in scheme will give conference participants the Chennai. option of making voluntary contributions to off- Copyright c 2019 by the author(s). set the carbon footprint of their trips to confer-

6 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 ences when they register online. ACM SIGAI • iWOAR 2018 plans to test this scheme at upcoming edi- • ICPRAM 2018 tions of the AAAI/ACM AI, Ethics and Soci- • IEA/AIE 2019 ety (AIES) and ACM Intelligent User Interfaces • (IUI) conferences in cooperation with AAAI FW 2018 and SIGCHI, respectively, and hopes that it • BIOSTEC 2019 will enable the ACM SIGAI and wider ACM • RecSys 2019 membership to contribute to the environmen- • FW 2019 tal sustainability of our communities. • ICAART 2019 ACM SIGAI continues to support AIES, which • AAMAS 2019 it co-founded in 2017 to fill a scientific void. As • iWOAR 2019 AI is becoming more pervasive in our lives, its • impact on society is more significant, raising KMIKS 2019 ethical concerns and challenges regarding is- • ICPRAM 2019 sues such as value alignment, safety and se- • FDG 2019 curity, data handling and bias, regulations, ac- • ICAIL 2019 countability, transparency, privacy and work- • IC3K 2019 force displacement. Only a multi-disciplinary • and multi-stakeholder effort can find the best IJCCI 2019 ways to address these concerns, by includ- • IEA/AIE ’20 ing experts from various disciplines, such as • AAMAS 2020 ethics, philosophy, economics, sociology, psychology, law, history and politics. AIES was ACM SIGAI also organizes – jointly with the co-located with AAAI 2019 in Honolulu and will Association for the Advancement of AI (AAAI) again be co-located with AAAI 2020 in New – the annual joint job fair at the AAAI con- York City. ference, where attendees can find out about job and internship opportunities from repre- ACM SIGAI sponsored the following confer- sentatives from industry, universities and other ences in addition to AIES 2019: organizations. The AAAI/ACM SIGAI job fair was held at AAAI 2019 in Honolulu, co- • WI 2018 organized by the ACM SIGAI labor market of- • ASE 2018 ficer. Twenty-six employers formally attended, • IVA 2018 while a handful of exhibitors who did not for- • HRI 2019 mally sign up also took part. Hundreds of CVs and resumes were collected before, dur- • IUI 2019 ing and after the job fair from students, post- doctoral researchers and other job seekers and it will sponsor the following conferences via the job fair web-site; these were shared coming up in 2019 and 2020: with interested employers. This year, the or- • IVA 2019 ganizers also purchased a dedicated domain aaaijobfair.com • ( ) to allow present and fu- K-CAP 2019 ture firms and participants to view previous it- • ASE 2019 erations of the job fair. The ACM SIGAI la- • WI 2019 bor market officer believes that we can use in- • sights from AI to create an even better func- ASE 2020 tioning job market and works actively toward • HRI 2020 designing the job market of the future. Toward • IUI 2020 that end, he has begun to gather requirements with a large number of chairs of top computer ACM SIGAI approved the following in- science departments in the USA as well as in cooperation and sponsorship requests from Israel and Europe and is working to formulate events covering a wide thematic and geo- a model that will be translated into a larger job graphical range across the international AI fair (in terms of participating employers as well community: as applicants) in the near future.

7 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

ACM SIGAI also co-sponsors – jointly with candidates in AI. This new annual award will AAAI – the annual joint doctoral consortium at be presented at the AAAI Conference on AI the AAAI conference, which provides an op- in the form of a certiﬁcate and is accompa- portunity for Ph.D. students to discuss their nied by the option to present the dissertation research interests and career objectives with at the AAAI conference as well as to submit a the other participants and a group of estab- six-page summary to both the AAAI proceed- lished AI researchers who act as their men- ings and the ACM SIGAI newsletter. The nom- tors. The AAAI/ACM SIGAI doctoral consor- ination deadline for the inaugural AAAI/ACM tium was held at AAAI 2019 in Honolulu. SIGAI Doctoral Dissertation Award will be announced later this year and is expected to be in late Fall 2019. Awards

ACM SIGAI sponsors the ACM SIGAI Au- Public Policy Activities tonomous Agents Research Award, an annual award for excellence in research in the area of ACM SIGAI promotes the discussion of poli- autonomous agents. The recipient is invited to cies related to AI through posts in the AI Mat- give a talk at the International Conference on ters blog, helps to identify external groups Autonomous Agents and Multiagent Systems with common interests in AI public policy, (AAMAS). The 2019 ACM SIGAI Autonomous encourages ACM SIGAI members to part- Agents Research Award was presented at AA- ner in policy initiatives with these organiza- MAS 2019 in Montreal to Carles Sierra, the tions, disseminates public policy ideas to the vice-director of the AI Research Institute of the ACM SIGAI membership through articles in Spanish National Research Council, for sem- the ACM SIGAI newsletter and ensures that inal contributions to research on negotiation every technologist is educated, trained and and argumentation, computational trust and empowered to prioritize ethical considerations reputation and artificial social systems. in the design and development of autonomous and intelligent systems. ACM SIGAI partici- ACM SIGAI also sponsors the ACM SIGAI pates in the ACM US Technology Policy Com- Industry Award for Excellence in AI, a new mittee (ACM USTPC), formerly USACM, via annual award which is given annually to an the ACM SIGAI public policy officer and in a individual or team in industry who created variety of other policy efforts, including those a fielded AI application in recent years that of other societies (such as the IEEE Global Ini- demonstrates the power of AI techniques via tiative on Ethics of Autonomous and Intelligent a combination of the following features: nov- Systems). ACM USTPC addresses US public elty of application area, novelty and techni- policy issues related to computing and infor- cal excellence of the approach, importance mation technology and regularly educates and of AI techniques for the approach and ac- informs US Congress, the US Administration tual and predicted societal impact of the ap- and the US courts about significant develop- plication. The inaugural ACM SIGAI Industry ments in the computing field and how those Award for Excellence in AI will be presented at developments affect public policy. For exam- the International Joint Conference on AI (IJ- ple, the ACM SIGAI public policy officer joined CAI) 2019 in Macau to the Real World Rein- the comments of ACM USTPC on the draft forcement Learning Team from Microsoft for of the “20-Year Roadmap for AI Research in the identification and development of cutting- the US” of the Computing Community Con- edge research on contextual-bandit learning, sortium. He also studies how organizations the manifest cooperation between research collect and analyze data and whether these and development efforts, the applicability of practices are consistent with recommenda- the decision support throughout the broad tions by the USTPC working group on algo- range of Microsoft products and the quality of rithmic accountability, transparency and bias. the final systems. He represented ACM SIGAI via his talks “Fu- ACM SIGAI also recently created – jointly with ture of Work, AI Education, and Public Policy” AAAI – the joint AAAI/ACM SIGAI Doctoral at EAAI 2019 and “Transparency, Accessibil- Dissertation Award to recognize and encour- ity, and Ethics in AI” at Dalhousie University age superior research and writing by doctoral in 2019. He was also the moderator of the

8 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 panel “Are We Ready for AI” at the Annual • Matthew Sun and Marissa Gerchick – The Consumer Assembly of the Consumer Feder- Scales of (Algorithmic) Justice: Tradeoffs and ation of America in 2019. Remedies • Annie Zhou – The Intersection of Ethics and AI Educational Activities ACM SIGAI supported the “Birds of the ACM SIGAI held a second ACM SIGAI Stu- Feather” undergraduate research challenge dent Essay Contest focused on AI ethics (or- organized by the ACM SIGAI education officer ganized by the ACM SIGAI AI and society offi- at the Symposium on Educational Advances cer), after the success of the first such compe- in AI (EAAI) 2019. Six research and liberal tition in 2017. Students could win cash prizes arts institutions participated with seven papers of US$500 or Skype conversations with se- and one poster presentation that passed peer nior AI researchers from academia or industry review. ACM SIGAI contributed US$500 of (including the director of Microsoft Research award funding for the best papers. The ACM Labs and the director of research at Google) if SIGAI education officer intends to announce their essays provided good answers to one or a Gin Rummy undergraduate research chal- both of the following topic areas (or any other lenge at EAAI 2020. question in this space that they considered im- ACM SIGAI also started discussions with the portant): ACM Special Interest Group on Computer Sci- ence Education (ACM SIGCSE) on a collabo- • What requirements, if any, should be imposed ration for disseminating pointers to resources on AI systems and technology when interacting for AI educators and creating incentives for the with humans who may or may not know that they production and dissemination of assignments are interacting with a machine? For example, should they be required to disclose their identi- on AI ethics. ties? If so, how? • What requirements, if any, should be imposed Member Communication on AI systems and technology when making decisions that directly affect humans? For exam- ACM SIGAI communicates with its members ple, should they be required to make transparent via email announcements, the ACM SIGAI decisions? If so, how? newsletter “AI Matters,” the AI Matters blog and webinars: This year, ACM SIGAI received 18 submissions, of which eight were selected for publica- ACM SIGAI maintains a more than 4,000 tion and prizes. The winning essays are listed email address long mailing list for AI-related below in alphabetical order by author. ACM announcements to its members and friends. SIGAI intents to hold a third ACM SIGAI stu- ACM SIGAI publishes four issues of its dent Essay Contest later this year. newsletter AI Matters per year. The ACM SIGAI newsletter is distributed via the • Janelle Berscheid and Francois Roewer- ACM SIGAI mailing list but also openly Despres – Beyond Transparency: A Proposed available on the ACM SIGAI web-site (at Framework for Accountability in Decision- sigai.acm.org/aimatters/). AI Matters Making AI Systems features articles of general interest to the AI • Gage Garcia – AI Education: A Requirement for community and added not only an additional a Strong Democracy editor-in-chief but also additional column edi- • Alexander Hilton – AI: The Societal Responsibil- tors in the past year. Recent columns, led by ity to Inform, Educate, and Regulate these and other column editors, have included • Michelle Seng Ah Lee – Context-Conscious AI Interviews (organized by the ACM SIGAI Fairness in Using Machine Learning to Make diversity officer), AI Amusements, AI Educa- Decisions tion (written or organized by the ACM SIGAI • Yat Long Lo, Chung Yu Woo and Ka Lok Ng – education officer), AI Policy Issues (written The Necessary Roadblock to Artificial General by the ACM SIGAI public policy officer) and Intelligence: Corrigibility AI Events (written by the ACM SIGAI confer- • Grace McFassel – Embedding Ethics: Design of ence coordination officer). The editors-in-chief Fair AI Systems have recently added an AI crossword puzzle

9 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

(thanks to Adi Botea from IBM’s Ireland Re- The ACM SIGAI conference coordination of- search Laboratory) and are about to add a col- ficer recently started to test a new open stu- umn on current research trends in AI, written dent award travel scheme. Beyond provid- by recent grantees of research funds (such as ing a ringfenced allocation to specific confer- NSF CAREER or European Research Council ences, he created a process by which any awards). AI Matters has also started to pub- ACM SIGAI student member who intends to lish the winning student essays of the second attend an ACM (and, in exceptional cases, ACM SIGAI Student Essay Contest. even a non-ACM) event can apply for travel support through the ACM SIGAI web-site. In ACM SIGAI also maintains an AI Matters blog the first few months since the inception of the sigai.acm.org/aimatters/blog/ (at ) scheme, students have already been offered as a forum for important announcements and financial support of about US$8,000 in total. news. For example, the ACM SIGAI public policy officer posts new information every two ACM SIGAI also recently created the AI Activ- weeks in the blog to survey and report on ities Fund, a new initiative to empower ACM current AI policy issues and raise awareness SIGAI members and friends to organize ac- about the activities of other organizations that tivities with a strong outreach component to share interests with ACM SIGAI. either students, researchers or practitioners not working on AI technologies or to the pub- After a hiatus due to the illness of one lic in general. The purpose of the inaugural of the organizers, ACM SIGAI recently re- call for funding proposals was to help ACM started the ACM SIGAI webinars with a SIGAI members and friends to promote a bet- webinar on “Advances in Socio-Behavioral ter understanding of current AI technologies, Computing” and several more in prepara- including their strengths and limitations as well tion. The webinars are streamed live but the as their promise for the future. Examples of videos can still be watched on demand at fundable activities included (but were not lim- learning.acm.org/webinar/. ited to) AI technology exhibits or exhibitions, ACM SIGAI is also a founding member of AI holding meetings with panels on AI technology Hub (at aihub.org), a new non-profit sibling (including on AI ethics) with expert speakers, to Robohub (at robohub.org) dedicated to creating podcasts or short films on AI tech- connecting the AI communities of the world by nologies that are accessible to the public and bringing together experts in AI research, start- holding AI programming competitions. ACM ups, business and education from across the SIGAI was looking for evidence that the infor- globe. Content-area specialists will curate all mation presented by the activities would be of incoming AI news articles to make sure that high quality, accurate, unbiased (for example, reporting is truthful, fair and balanced, and not influenced by company interests) and at in-house editors will ensure that all content the right level for the intended audience. The meets the highest editorial standards for lan- inaugural call for proposals supported the fol- guage and clarity. AI Hub is expected to come lowing initiatives: a workshop on “AI for All us- online in Summer or Fall 2019. ACM SIGAI ing the R Programming Language” organized will provide content to AI Hub and, conversely, by the Indian Institute of Technology in Goa, AI Hub will provide AI news to the ACM SIGAI the “Bee Network of AI” organized by the Uni- members. versidad Mayor in Chile and “Co-Opting AI: Public Conversations about Design, Inequality and Technology” organized by New York Uni- Financial Member Support versity. ACM SIGAI so far had concentrated its financial support on travel scholarships to ACM Additional Member Services SIGAI student members to allow them to attend conferences if they are otherwise missing ACM SIGAI also supports its members in the financial resources to do so. The amounts additional ways. For example, it nominates of the scholarships vary but are generally in them for awards or supports their nominations. the range of US$1,000 to US$10,000 per con- ACM SIGAI is proud of the fact that many AI ference, depending on the conference size. researchers in the past year received ACM

10 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 honors, such as becoming ACM senior members, distinguished members and fellows as well as receiving other awards. Three AI researchers received the A.M. Turing Award in 2018. ACM SIGAI also actively supports the Re- search Highlight Track of the Communications of the ACM (CACM) by nominating publications of recent, signiﬁcant and exciting AI research results that are of interest to the computer science research community in general to the Research Highlight Track. This way, ACM SIGAI helps to make important AI research results visible to many computer scientists. Additional ACM SIGAI membership beneﬁts include reduced registration fees at many of the co-sponsored and in-cooperation conferences and access to the proceedings of many of these conferences in the ACM Digital Li- brary.

Planning for the Future ACM SIGAI held elections for a new chair, vice chair and secretary/treasurer in Spring 2019. Sanmay Das (the current ACM SIGAI vice chair) was elected ACM SIGAI chair, Nicholas Mattei (the current ACM SIGAI AI and society officer) was elected ACM SIGAI vice-chair, and John Dickerson (the current ACM SIGAI labor market officer) was elected ACM SIGAI secretary/treasurer. Sven Koenig (the current ACM SIGAI chair) will transition to his new role as ACM SIGAI past chair. We are looking forward to the new leadership committee shaping the future of ACM SIGAI. In general, ACM SIGAI intends to reach out to more AI groups worldwide that could benefit from ACM support, such as providing financial support, making the proceedings widely accessible in the ACM Digital Library and providing speakers via the ACM Distinguished Speakers program. ACM SIGAI also intends to reach out more to other disciplines that share an interest in AI, for example, in terms of research methodologies or applications.

11 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Help Communities Solve Real-World Problems with AI – Become a Technovation Mentor! Tara Chklovski (Founder and CEO, Technovation; [email protected]) DOI: 10.1145/3362077.3362080

Abstract thereby strengthening local capacity in areas that may not have access to technology pro- Join other AI professionals as a mentor in fessionals or universities. Technovation’s AI program for families. Share your expertise with adults and children who In the first year of Technovation Families’ AI are curious about artificial intelligence and competition, 7,500 people across 70 chap- how it can be used to address real issues in ters in 13 countries participated, developing their communities. Help people all around the 200 AI-based solutions to problems in their world learn to not only use AI, but to create communities. These solutions ranged from solutions with AI that improve their lives and image-recognition software that scans chil- their communities. dren’s drawings for signs of bullying and a wearable swimming cap for kids to detect early signs of drowning, to a tool to detect and re- Technovation Families move invasive algae from a local lake (Figure 1 and Figure 2). Technovation, a global technology education nonprofit, is seeking mentors for its second season of Technovation Families, an AI- focused program for families with children ages 8-13. Join peers working in Computer Science fields and be part of the world’s largest AI mentoring program. Support families around the world as they learn about AI and develop AI-based prototypes addressing problems they identify in their communities. Started in 2018, Technovation Families’ AI challenge brings families, schools, communi- Figure 1: Jeff Dean (Head of Google Brain) lis- ties, and mentors together to learn, play, and tening to a father and son team describing their create with AI. They apply what they learn to image-recognition prototype that emits ultrasonic solve a real-world problem in their commu- frequencies when it sees a dog. nity as part of a global competition. Mentors guide learners of all ages through Neural Net- works, data, self-driving car algorithms, and machine learning and training models to recognize images, text, and emotions through an IBM-Watson based platform Machine Learn- ing for Kids. Local educators and CS experts offer encouragement and guidance throughout as families learn about – and use – AI tools for the first time. Mentors especially are a critical touch-point for families who are developing their confidence as problem solvers and inven- tors, and who have big ideas for applying AI to community problems, but lack technical knowl- Figure 2: Six coaches from Bolivia, Palestine, edge, experience, and confidence in their abil- Spain, United States, Pakistan and Kazakhstan ities. Mentors are able to volunteer remotely, who coached their communities to create winning Copyright c 2019 by the author(s). AI-based inventions.

12 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Technovation partners with industry leaders lem to solve in their local community, and build including Google, NVIDIA, Intel, General Mo- an AI agent to solve it. Sign-up today to help tors, and the Patrick J. McGovern Foundation, families make the world a better place with AI! to bridge the AI knowledge and conﬁdence gap for children and adults around the world (Figure 3).

Figure 3: Mother and sons from Bolivia explaining to a judge how their Raspberry-Pi powered, image Figure 4: First 5-weeks of project-based Techno- recognition system sucks up invasive weeds from vation Families AI curriculum that introduces learn- Lake Titicaca. ers to AI, Machine Learning, and building Image and Text Recognition Systems. The Technovation Families program is built on a community-based model that involves parents and caregivers so that the adults (in addition to the children) can reignite their curios- ity and develop as lifelong learners. After participating in the program, more than 91% of the parents surveyed believed their child developed a sustained interest and growing interest in AI and ∼85% of parents wanted to continue investing effort into improving their local communities (Chklovski, Jung, Fofang, & Gonzales, 2019). Mentors beneﬁt too. Through Technovation’s programs, mentors’ communications and presentation skills improve, as do their professional relationships with colleagues and leaders. And, their work with Technovation partic- Figure 5: Last 5-weeks of the Technovation Fami- ipants stretches and grows their creativity, or- lies AI curriculum that helps participants apply their learning to create AI-based prototypes that ad- ganizational, and project management skills. dress problems in their community. Recently, the second season of Technovation Families launched debuting an expanded curriculum developed in partnership with a com- References mittee of AI researchers, and industry profes- Chklovski, T., Jung, R., Fofang, J. B., & Gon- sionals. The updated curriculum includes ad- zales, P. (2019). Implementing a 15- ditional information about AI, good and bad week ai-education program with under- datasets, machine learning and ethical inno- resourced families across 13 global com- vation (Figure 4 and Figure 5). Through 10 munities. In International joint confer- fun, hands-on lessons, families learn founda- ence on artiﬁcial intelligence, macau. tional AI concepts, identify a meaningful prob-

13 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Tara Chklovsk Tara Chklovski is CEO and founder of global tech education nonproﬁt Tech- novation. A frequent advocate for STEM education, she’s presented at the White House STEM Inclusion Summit, SXSW EDU, UNESCO’s Mobile Learning Week, and led the education track at the 2019 UN AI for Good Global Summit.

14 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Events Michael Rovatsos (University of Edinburgh; [email protected]) DOI: 10.1145/3362077.3362081

This section features information about up- perimental and/or theoretical studies yielding coming events relevant to the readers of AI new insights that advance Pattern Recognition Matters, including those supported by SIGAI. methods are especially encouraged. We would love to hear from you if you are Submission deadline: October 4, 2019 are organizing an event and would be interested in cooperating with SIGAI. For more information about conference support visit 13th International Joint Conference on sigai.acm.org/activities/requesting sponsor- Biomedical Engineering Systems and ship.html. Technologies (BIOSTEC 2020) 2nd International Conference on Valetta, Malta, February 24-26, 2020 Artificial Intelligence & Virtual Reality http://www.biostec.org/ (AIVR 2019) The purpose of BIOSTEC is to bring together researchers and practitioners, including en- San Diego, CA, December 9-11, 2019 gineers, biologists, health professionals and http://ieee-aivr.org informatics/computer scientists, interested in The AIVR conference, now in its second run, both theoretical advances and applications of is a unique event, addressing researchers and information systems, artificial intelligence, sig- industries from all areas of AI as well as Vir- nal processing, electronics and other engi- tual, Augmented, and Mixed Reality. It pro- neering tools in knowledge areas related to bi- vides an international forum for the exchange ology and medicine. BIOSTEC is composed between those fields to present advances in of five co-located conferences, each special- the state of the art, identify emerging re- ized in a different knowledge area. search topics, and together define the future Submission deadline: October 4, 2019 of these exciting research domains. We invite researchers from VR, as well as Augmented Reality (AR) and Mixed Reality (MR) to participate and submit their work to the program. 12th International Conference on Likewise, any work on AI that has a relation to Agents and Artificial Intelligence any of these fields or potential for the usage in (ICAART 2020) any of them is welcome. Valetta, Malta, February 24-26, 2020 http://www.icaart.org/ 9th International Conference on The purpose of ICAART is to bring together Pattern Recognition Applications and researchers, engineers and practitioners inter- Methods (ICPRAM ’20) ested in the theory and applications in the ar- Setubal,´ Portugal, February 22-24, 2020 eas of Agents and Artificial Intelligence. Two http://www.icpram.org/ simultaneous related tracks will be held, cov- The International Conference on Pattern ering both applications and current research Recognition Applications and Methods would work. One track focuses on Agents, Multi- like to become a major point of contact be- Agent Systems and Software Platforms, Dis- tween researchers, engineers and practition- tributed Problem Solving and Distributed AI in ers on the areas of Pattern Recognition, general. The other track focuses mainly on both from theoretical and application perspec- Artificial Intelligence, Knowledge Representa- tives. Contributions describing applications tion, Planning, Learning, Scheduling, Percep- of Pattern Recognition techniques to real- tion Reactive AI Systems, and Evolutionary world problems, interdisciplinary research, ex- Computing and other topics related to Intelli- gent Systems and Computational Intelligence. Copyright c 2019 by the author(s). Submission deadline: October 4, 2019

15 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

15th ACM/IEEE International for research in autonomous agents and Conference on Human-Robot multi-agent systems. The AAMAS conference Interaction (HRI 2020) series was initiated in 2002 as the merging of three respected scientific meetings: the Cambridge, UK, March 23-26, 2020 International Conference on Multi-Agent http://humanrobotinteraction.org/2020/ Systems (ICMAS), the International Work- HRI 2020 is the 15th annual conference shop on Agent Theories, Architectures, and for basic and applied human-robot interac- Languages (ATAL), and the International tion research. Researchers from across the Conference on Autonomous Agents (AA). The world present their best work to HRI to ex- aim of the joint conference is to provide a change ideas about the theory, technology, single, high-profile, internationally-respected data, and science furthering the state-of-the- archival forum for scientific research in the art in human-robot interaction. Each year, the theory and practice of autonomous agents HRI Conference highlights a particular area and multi-agent systems. AAMAS 2020 is through a theme. The theme of HRI 2020 is the 19th edition of the AAMAS conference, “Real World Human-Robot Interaction”. The and the first time AAMAS will be held in New HRI conference is a highly selective annual Zealand. The conference solicits papers international conference that aims to show- addressing original research on autonomous case the very best interdisciplinary and mul- agents and their interaction, including agents tidisciplinary research in human-robot inter- that interact with humans. In addition to the action with roots in and broad participation main track, there will be two special tracks: from communities that include but are not lim- Blue Sky Ideas and JAAMAS. ited to robotics, artificial intelligence, human- Submission deadline: November 15, 2020 computer interaction, human factors, design, and social and behavioral sciences. Submis- sion deadline: October 1, 2019 33rd International Conference on Industrial, Engineering and Other 22nd International Conference on Applications (IEA/AIE ’20) Enterprise Information Systems (ICEIS Kitakyushu, Japan, July 21-24, 2020 2020) https://jsasaki3.wixsite.com/ieaaie2020 IEA/AIE 2020 continues the tradition of em- Prague, Czech Republic, May 5-7, 2020 phasizing on applications of applied intelligent http://www.iceis.org/ systems to solve real-life problems in all ar- The purpose of ICEIS is to bring together re- eas including engineering, science, industry, searchers, engineers and practitioners inter- automation & robotics, business & finance, ested in the advances and business appli- medicine and biomedicine, bioinformatics, cy- cations of information systems. Six simul- berspace, and human-machine interactions. taneous tracks will be held, covering differ- Submission deadline: December 15, 2019 ent aspects of Enterprise Information Systems Applications, including Enterprise Database Technology, Systems Integration, Artificial In- 35th IEEE/ACM International telligence, Decision Support Systems, Infor- Conference on Automated Software mation Systems Analysis and Specification, Engineering (ASE 2020) Internet Computing, Electronic Commerce, Melbourne, Australia, September 21-25, 2020 Human Factors and Enterprise Architecture. https://www.deakin.edu.au/ase2020 Submission deadline: December 13, 2019 The 35th IEEE/ACM International Conference on Automated Software Engineering (ASE 19th International Conference on 2020) will be held in Melbourne, Australia from September 21 to 25, 2020. The conference Autonomous Agents and Multi-Agent is the premier research forum for automated Systems (AAMAS 2020) software engineering. Each year, it brings Auckland, New Zealand, May 9-13, 2020 together researchers and practitioners from https://aamas2020.conference.auckland.ac.nz/ academia and industry to discuss founda- AAMAS is the leading scientific conference tions, techniques, and tools for automating

16 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 the analysis, design, implementation, testing, and maintenance of large software systems.

Michael Rovatsos is the Conference Coordi- nation Ofﬁcer for ACM SIGAI, and a faculty member at the Univer- sity of Edinburgh. His research in multiagent systems and human- friendly AI. Contact him at [email protected].

17 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

AI Education Matters: Building a Fake News Detector Michael Guerzhoy (Princeton University, University of Toronto, and the Li Ka Shing Knowl- edge Institute, St. Michael’s Hospital; [email protected]) Lisa Zhang (University of Toronto Mississauga; [email protected]) Georgy Noarov (Princeton University; [email protected]) DOI: 10.1145/3362077.3362082

Introduction Fake news is a salient societal issue, the subject of much recent academic research, and, as of 2019, a ubiquitous catchphrase. In this article, we explore using the task of detecting fake news to teach supervised machine learning and data science, as demonstrated in our Model AI Assignment1(Neller et al., 2019). We ask students to build a series of increasingly complex classifiers that cate- gorize news headlines into “fake” and “real” and to analyze the classifiers they have built. Students think about the data, the validity of the problem posed to them, and the assumptions behind the models they use. Students ( | ) can compete in a class-wide competition to Figure 1: Visualizing P fake keyword for a naive Bayes model trained on our training set. Larger build the best fake news detector. text corresponds to larger conditional probabilities. To help instructors incorporate fake news detection into their course, we briefly review recent research on fake news and the task of patterns (Shu, Sliva, Wang, Tang, & Liu, fake news detection. We then describe the 2017). High-quality datasets of fake and assignment design, and reflect on the in-class real news are scarce. Several medium-scale fake news detection competition we ran. datasets have recently been collected, with fake news either obtained from the web (often with the help of fact-checking resources Fake News Research such as PolitiFact.com) or written to order Fake news is an old issue (Mansky, 2018), by Amazon Mechanical Turk workers (Wang, but the role it may have played in the 2016 2017), (Perez-Rosas,´ Kleinberg, Lefevre, & US Presidential Election has sparked renewed Mihalcea, 2018). interest in the phenomenon (Lazer et al., The definition of the concept of “fake news” 2018), (Allcott & Gentzkow, 2017). Re- itself has proven elusive. See (Tandoc, Lim, & search on fake news is focused on under- Ling, 2018) for an overview of the definitions standing its audience and societal impact, recently used in literature. how it spreads on social media, and who its consumers are (Grinberg, Joseph, Friedland, Swire-Thompson, & Lazer, 2019), (Nelson & Teaching Supervised Learning via Taneja, 2018). Fake News Fake news can be detected based on tex- In our assignment, the task is to classify tual features and social network propagation news headlines as “real” or “fake”. Students Copyright c 2019 by the author(s). build and compare several standard classi- 1http://modelai.gettysburg.edu/ fiers: naive Bayes, logistic regression, and 2019/fakenews/ a decision tree. All three classifiers use the

18 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 presence/absence of keywords as their fea- why, students need to think about how human ture set. The detection of fake news headlines language works). using naive Bayes is directly analogous to the Another part of the assignment involves pro- classic spam filtering task. See (Russell & ducing new data via the naive Bayes genera- Norvig, 2009) for an exposition and (Sahami, tive model. The goal is for students to gain a Dumais, Heckerman, & Horvitz, 1998) for the deeper understanding of generative models. paper that introduced the idea. Our pedagogical approach emphasizes having students analyze the models they build. In Datasets particular, we ask students to obtain keywords The dataset students use in the principal part whose presence or absence most strongly in- of the assignment was compiled by combining dicates that a headline is “fake” or “real”. data from several sources. It consists of 1298 To find the most important keywords for clas- “fake news” headlines and 1968 “real news” sifying a headline as “real” using naive Bayes, headlines, all containing the word “Trump”. students need to decide whether they should “Fake” headlines are challenging to collect; as use P (real|keyword) or P (keyword|real).We students see, most headlines labeled as such hope they gain a deeper understanding of could be argued to be merely tendentious or naive Bayes in the process. We use PyTorch hyperbolic rather than fake. to implement logistic regression, and suggest We have curated a smaller private test set of (as would be natural for our students) that stu- headlines that we have verified to be either dents use multinomial logistic regression with real or fake2. That test set is used in our fake 2 outputs when predicting “fake”/“real”. This news detection competition and is available to 2 +2 results in k coefficients for a vocabulary instructors upon request. of k keywords. Identifying the most important keywords based on these 2k +2numbers nudges students towards understanding the Fake News Detection Contest details of the model. We also ask students to derive the logistic regression coefficients that For interested students, we ran an optional correspond to the naive Bayes classifier they fake news detection competition. The authors fit. As a final step, students fit a decision tree of the best-performing entries would earn a to the data and again identify the most impor- small amount of points towards their course tant features according to the model. grade. Participating students could follow any approach they liked. We encouraged aug- As they fit a series of increasingly complex menting the given training set with more data, classifiers, students observe overfitting first- engineering useful features, training classi- hand: training performance increases with fiers of the students’ own choice, and using classifier complexity, while validation perfor- ensemble methods. Gratifyingly, some con- mance decreases. Beating naive Bayes turns testants were able to engineer useful features out to be quite difficult (though doable). Stu- and use modern text classification algorithms dents attempt to do that in the competition to beat the naive Bayes baseline. phase. The source code for many modern text classification systems is widely available and some- Teaching Data Science via Fake News times comes with pre-trained weights. Stu- dents would often adapt, train, or fine-tune the When using the assignment in a data sci- systems for their submissions.3 ence rather than a machine learning course, 2While we could not fact-check the headlines to we place more emphasis on statistical model- journalistic standards, we made sure that the truth ing and careful examination of the data. We or falseness of the headlines was not in serious ask students to inspect the dataset in order dispute. to analyze it qualitatively and discuss its lim- 3Training deep learning systems is often re- itations. Students are also asked to check source intensive. We refer students to services whether the dataset conforms to the naive such as AWS, Microsoft Azure, and Google Cloud Bayes assumption (it does not; to figure out Platform, where they are eligible for free credits.

19 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Conclusion Text Categorization: Papers from the 1998 workshop (Vol. 62, pp. 98–105). Through building a fake news detector in Shu, K., Sliva, A., Wang, S., Tang, J., & Liu, class, we are able to teach some of the foun- H. (2017). Fake news detection on so- dational methods of supervised learning in a cial media: A data mining perspective. compelling and coherent manner. The dataset SIGKDD Explorations Newsletter, 19(1), we collected can be used in a class that em- 22–36. phasizes rigorous thinking about data science Tandoc, E. C., Lim, Z. W., & Ling, R. (2018). problems. We share our experience of run- Defining “fake news”. Digital Journalism, ning an in-class fake news detection competi- 6(2), 137-153. tion. Wang, W. Y. (2017). “Liar, liar pants on fire”: A new benchmark dataset for fake news References detection. In Proceedings of the 55th An- nual Meeting of the Association for Com- Allcott, H., & Gentzkow, M. (2017). Social putational Linguistics. media and fake news in the 2016 election. Journal of Economic Perspectives, 31(2), 211–36. Michael Guerzhoy Grinberg, N., Joseph, K., Friedland, L., Swire- is a Thompson, B., & Lazer, D. (2019). Fake Lecturer at Princeton Uni- news on twitter during the 2016 us pres- versity, an Assistant Pro- idential election. Science, 363(6425), fessor (Status Only) at 374–378. the University of Toronto, Lazer, D. M., Baum, M. A., Benkler, Y., Berin- and a Scientist at the Li sky, A. J., Greenhill, K. M., Menczer, F., Ka Shing Knowledge In- . . . others (2018). The science of fake stitute, St. Michael’s Hos- news. Science, 359(6380), 1094–1096. pital. His professional in- Mansky, J. (2018). The age-old problem of terests are in computer “fake news”. Smithsonian Magazine. science and data science education and in ap- https://www.smithsonianmag plications of machine learning to healthcare. .com/history/age-old-problem Lisa Zhang is an Assis- -fake-news-180968945/. tant Professor, Teaching Neller, T. W., Sooriamurthi, R., Guerzhoy, M., Stream (CLTA) at the Zhang, L., Talaga, P., Archibald, C., . . . University of Toronto others (2019). Model AI Assignments Mississauga. Her current 2019. In Proceedings of the AAAI Con- research interests are in ference on Artificial Intelligence (Vol. 33, the intersection of com- pp. 9751–9753). puter science education Nelson, J. L., & Taneja, H. (2018). The small, and machine learning. disloyal fake news audience: The role of audience availability in fake news con- Georgy Noarov sumption. New Media & Society, 20(10), is a stu- 3720-3737. dent at Princeton Uni- Perez-Rosas,´ V., Kleinberg, B., Lefevre, A., & versity concentrating in Mihalcea, R. (2018). Automatic detec- mathematics and pursu- tion of fake news. In Proceedings of the ing a certificate in statis- 27th International Conference on Com- tics and machine learn- putational Linguistics (pp. 3391–3401). ing. His research areas include algorithmic game Russell, S. J., & Norvig, P. (2009). Artificial In- theory and combinatorial telligence: A Modern Approach. Pearson optimization. Education Limited. Sahami, M., Dumais, S., Heckerman, D., & Horvitz, E. (1998). A bayesian approach to filtering junk e-mail. In Learning for

20 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

AI Education Matters: A First Introduction to Modeling and Learn- ing using the Data Science Workﬂow Marion Neumann (Washington University in St. Louis; [email protected]) DOI: 10.1145/3362077.3362083

Introduction Traditionally artificial intelligence (AI) and machine learning (ML) courses are taught at the senior and graduate level in higher-education computer science curricula following the mastery learning strategy, cf. Figure 1. This makes Figure 1: Mastery learning paradigm. sense, since most AI and ML models and the theory behind them require a substantial understanding of probability and statis- we propose to gently introduce AI/ML con- tics, as well as advanced calculus and ma- cepts focusing on example applications rather trix algebra. To understand Logistic Regres- than computational problems by incorporating sion as a probabilistic classifier performing course modules into introductory CS courses maximum-likelihood or maximum-a-posteriori or design an entire course early on the cur- estimation, for example, students need to un- riculum. The goal of such intro-level mod- derstand joint and conditional probability dis- ules or courses is to expose students to AI/ML tributions. In order to derive the back propa- problems and introduce basic techniques to gation algorithm to train Neural Networks stu- solve them without relying on the computa- dents need to understand partial derivatives tional and mathematical prerequisite knowl- and inner and outer tensor products. These edge. More concretely, the module or course are just two of many examples where sub- may be designed as combined lecture and stantial mathematical background – typically lab sessions, where a new topic is introduced taught at the junior level in a computer science in a lecture unit followed by a lab session, major program – is required. With AI and ML where students get to know a problem in the algorithms being used more widely by enter- context of an application, explore a solution prises across domains, as well as, in applica- method, and tackle a potentially open-ended tions and services we use in our daily lives, it question about evaluation procedures, bene- makes sense to raise awareness about what fits and challenges of the approach, or impli- AI is, what it can and cannot do, and how it cations and ethical considerations when us- is used to solve problems to a broader audi- ing such methods in the real world in a group ence. Very much in the same spirit as the discussion. Lab sessions should be designed https://www.csforall “CS for all” idea ( carefully focusing on the understanding of the .org ), we have to extend our curricula to in- data, the problem, and the results instead of clude introductory courses to AI and ML on model implementation. We will introduce two the early undergraduate level (or even in high- such lab assignments implemented in Python school) to expose students to the ideas and working principles of AI technology. One way to achieve this is to introduce the principles of working with data, modeling, and learning through the data science workflow.

Exposure First Following the exposure – interest – mastery paradigm as illustrated in Figure 2,

21 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Figure 3: Data science workflow using sentiment analysis as an example application. and Jupyter notebooks in the next section. Iris dataset (https://archive.ics.uci .edu/ml/datasets/Iris). We interac- The main aim of our assignments is to en- tively introduce the use of expressions, vari- gage the students’ interest to acquire the pre- ables, strings, printing, lists, dictionaries, con- requisite knowledge in order to move forward trol flow, and functions in Python to students and gain a deeper understating of specific AI that are already familiar with a programming and ML techniques. Since we propose these language from an introductory CS course. units for a course that is taught very early The second lab aims at motivating students to in the CS curriculum, we face the challenge acquire skills such as using statistics to model that students do not have a lot of program- and analyze data, knowing how to design and ming experience nor a deep understanding of use algorithms to store, process, and visual- data structures and algorithms. Therefore, we ize data, while not forgetting the importance developed the lab assignments using Jupyter of domain expertise. We begin by establish- notebooks which nicely combine illustrative in- ing the example problem to be studied based structions and executable starter code. on the Iris dataset. The next step is to acquire After having worked though the data science and process the data, where students practice workflow using illustrative applications that are how to load data and process strings into nu- both easy to understand and relevant in the meric arrays using numpy. Then, we explain real-world, our hope is that students develop different plotting methods such as box plots, the motivation to study traditional prerequisite histograms, and scatter plots for data explo- classes for AI and ML courses like probability ration leveraging matplotlib. Finally, we and statistics, matrix/linear algebra, and algo- split the data into training and test set, build rithm analysis perceiving them useful to mas- a model, use it for predictions, and evaluate ter AI/ML instead of a nuisance. the results using sklearn. The main learning objectives are to get to know and practice Python in the context of a realistic data sci- Two Model AI Assignments ence and machine learning application. Introduction to Python for Data Science Introducing the Data Science Workflow We provide an interactive guided lab to using Sentiment Analysis introduce Python for data science (DS),1 which can also be used for any course The second interactive lab guides students that introduces modeling and learning us- through a basic data science workflow by ex- ing Python, such as introduction to AI or ploring sentiment analysis.2 The data science ML courses. We provide two Jupyter note- workflow along with the example sentiment books, one introducing the basics of Python analysis application is depicted in Figure 3. and the other the DS workflow using the The lab assignment focuses on introducing 1http://modelai.gettysburg.edu/ 2http://modelai.gettysburg.edu/ 2019/intro2py/ 2019/intro2ds/

22 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 the machinery using a given dataset of movie there is no unified way to tackle this issue, reviews. We further provide a follow-up home- however, we believe that students should be work assignment reiterating some of the steps encouraged to work in teams for the lab as- and highlighting data acquisition and explo- signments, whereas homework assignments ration with Twitter data. After introducing senti- should be worked on individually. This way ment analysis, we explain a simple rule-based both teamwork and communication skills as approach to predict the sentiment of textual re- well as knowledge retention are facilitated. views using three handcrafted examples. This Both labs were perceived as useful by our stu- introduction shows simple means to prepro- dents. 97% answered Ye s to the question cess text data and exemplifies the use of lists “Did you like the lab.” for the introduction to of positive and negative expressions to com- Python lab and 81% for the sentiment analysis pute a sentiment score. Then students will lab. The most common reasons stated by stu- implement the approach to predict the senti- dents that didn’t like the second lab were that ment of movie reviews and evaluate the re- they where overwhelmed by unfamiliar code sults. The lab concludes with a discussion and that it was too long. From the students’ of the limitations of the rule-based approach answers to our quiz and exam questions we and a quick introduction to sentiment clas- can also confirm that they understand basic sification via machine learning. The home- Python processes to handle data, implement work assignment reiterates over the process and apply simple learning models, and visual- of building and analyzing a sentiment predictor ize and interpret their results. with the focus on collecting and preprocessing their own dataset scraped from Twitter using the python-twitter API. The main learn- Pedagogical Resources ing objective of this activity is getting to know the inference problem and walking through the In addition to Jupyter notebooks constituting entire data science workflow to tackle it. Since the lab and homework assignments, we de- the module only requires minimal program- veloped lecture materials in form of slides ming background it is an ideal precursor to in- and worksheets for each module. The first troducing machine learning in an AI, ML, or lecture covers an introduction to data sci- DS course. It may also be used in a introduc- ence and machine learning, and the second tion to Python course as a module focusing on one introduces sentiment analysis, text pro- using libraries and APIs. cessing, and classification respectively. The slides are interactive with gaps to be filled in by the instructor during the lectures and the Our Experiences worksheets contain in-class activities for stu- We incorporated both lab assignments into dents to engage with the presented materi- our “Introduction to Data Science” course for als. Those resources are available from the sophomore students at Washington University authors upon request. in St. Louis. One of the challenges we faced Useful textbooks that specifically focus on in- was that our students had different levels of troducing data science topics and techniques Python experience, from no experience at all are: (51%) over some experience (36%) to quite proficient (13%). This led to a large variance • Python Data Science Handbook Vander- in the times needed to complete the labs. To Plas (2016) introduces essential tools and deal with this issue we propose to add some libraries such as Jupyter notebooks, numpy, optional challenge problems to the assign- pandas, scikit-learn, and matplotlib for work- ment that are not required for the homework or ing with data. will be introduced later in the course. Another • challenge was that some students preferred Data Science from Scratch Grus (2019)fo- to work in groups where others did the labs cuses on implementing learning algorithms on their own. However, both strategies can and data processing routines from scratch. result in slower or faster pace given the stu- • Data Science for Business Provost and dents’ working style, group composition, and Fawcett (2013) showcases interesting real- amount of group discussion. Unfortunately, world use cases and emphasizes data-

23 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

analytic thinking while not being too technical.

The ﬁrst two books focus on implementations in Python, whereas the third one details concepts and techniques without code examples.

References Grus, J. (2019). Data science from scratch: ﬁrst principles with python. O’Reilly Me- dia. Provost, F., & Fawcett, T. (2013). Data science for business: What you need to know about data mining and data- analytic thinking. ” O’Reilly Media, Inc.”. VanderPlas, J. (2016). Python data science handbook: essential tools for working with data. ” O’Reilly Media, Inc.”.

Marion Neumann is a Senior Lecturer at Washington University in St. Louis and the SIGAI diversity ofﬁcer. She teaches Machine Learn- ing, Cloud Computing, Analysis of Networked Data, and Introduction to Data Science. Her research interests include graph-based machine learning and analyzing networked data as well as measuring and analyzing student emotions in large computing courses using sentiment analysis.

24 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

AI Policy Matters Larry Medsker (The George Washington University; [email protected]) DOI: 10.1145/3362077.3362084

Abstract Oakland, CA, are the first three cities to limit use of FR to identify people. AI Policy Matters is a regular column in AI • In “Facial recognition may be banned from Matters featuring summaries and commen- public housing thanks to proposed law” tary based on postings that appear twice a month in the AI Matters blog (https:// CNET reports that a bill will be introduced to sigai.acm.org/aimatters/blog/). We address the issue that “landlords across the welcome everyone to make blog comments country continue to install smart home tech- so we can develop a rich knowledge base of nology and tenants worry about unchecked information and ideas representing the SIGAI surveillance.” members. • A call for a more comprehensive ban on FR has been launched by the digital rights About Face group Fight for the Future, seeking a complete Federal ban on government use of fa- Face recognition (FR) research has made cial recognition surveillance. great progress in recent years and has been prominent in the news. In public policy, many Beyond legislation against FR research and are calling for a reversal of the trajectory for FR banning certain products, work is in progress systems and products. In the hands of peo- to enable safe and ethical use of FR. A more ple of good will, using products designed for general example that could be applied to FR safety and training systems with appropriate is the MITRE work The Ethical Framework data, FR benefits society and individuals. The for the Use of Consumer-Generated Data in Verge reports the use in China of unique facial Health Care, which “establishes ethical val- markings of pandas to identify individual an- ues, principles, and guidelines.” imals. FR research includes work to mitigate negative outcomes, as with the Adobe and UC AI Regulation Berkeley work on Detecting Facial Manipula- tions in Adobe Photoshop for automatic de- With AI in the news so much over the past tection of facial images that have been manip- year, the public awareness of potential prob- ulated by splicing, cloning, and removing ob- lems arising from the proliferation of AI sys- jects. tems and products has led to increasing calls for regulation. The popular media, and even Intentional and unintentional application of technical media, do contain misinformation systems that are not designed and trained for and misplaced fears, but plenty of legitimate ethical use are a threat to society. Screening issues exist even if their relative importance is for terrorists could be good, but FR lie and sometimes misunderstood. Policymakers, re- fraud detection systems sometimes do not searchers, and developers need to be in dia- work properly. The safety of FR is currently log about the true needs for and potential dan- an important issue for policymakers, but regu- gers of regulation. From our policy perspec- lations could have negative consequences for tive, the significant risks from AI systems in- AI researchers. As with many contemporary clude misuse and faulty unsafe designs that issues, conflicts arise because of conflicting can create bias, non-transparency of use, and policies in different countries. Recent and cur- loss of privacy. Some AI systems are known rent legislation is attempting to restrict FR use to discriminate against minorities, unintention- and possibly inhibit FR research; for example, ally and not. • San Francisco, CA, Somerville, MA, and An important discussion we should be having Copyright c 2019 by the author(s). is if governments, international organizations,

25 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 and big corporations, which have already re- artificial intelligence (AI) because they under- leased dozens of non-binding guidelines for stand that AI is a foundational technology that the responsible development and use of AI, can boost competitiveness, increase produc- are the best entities for writing and enforc- tivity, protect national security, and help solve ing regulations. Non-binding principles will not societal challenges. This report compares make some companies developing and apply- China, the European Union, and the United ing AI products accountable. An important States in terms of their relative standing in point in this regard is to hold companies re- the AI economy by examining six categories of sponsible for the product design process itself, metrics: talent, research, development, adop- not just for testing products after they are in tion, data, and hardware. It finds that de- use. spite the bold AI initiatives in China, the United States still leads in absolute terms. China Introduction of new government regulations is comes in second, and the European Union a long process and subject to pressure from lags further behind. This order could change lobbyists, and the current US administration in coming years as China appears to be mak- is generally inclined against regulations any- ing more rapid progress than either the United way. We should discuss alternatives like clear- States or the European Union. Nonetheless, inghouses and consumer groups endorsing AI when controlling for the size of the labor force products designed for safety and ethical use. in the three regions, the current U.S. lead be- If well publicized, the endorsements of re- comes even larger, while China drops to third spected non-partisan groups including profes- place, behind the European Union. This re- sional societies might be more effective and port also offers a range of policy recommen- timely than government regulations. dations to help each nation or region improve its AI capabilities.” The European Union has released its Ethics Guidelines for Trustworthy AI, and a second document with recommendations on how to US and G20 AI Policy boost investment in Europe’s AI industry is to be published. In May, 2019, the Organi- The G20 AI Ministers from the Group of zation for Economic Cooperation and Devel- 20 major economies conducted meetings on opment (OECD) issued their first set of in- trade and the digital economy. They pro- ternational OECD Principles on Artificial In- duced guiding principles for using artificial intelligence, which are embraced by the United telligence based on principles adopted earlier State and leading AI companies. by the 36-member OECD and an additional six countries. The G20 guidelines call for users and developers of AI to be fair and account- The AI Race able, with transparent decision-making processes and to respect the rule of law and val- China, the European Union, and the United ues including privacy, equality, diversity and States have been in the news about strate- internationally recognized labor rights. Mean- gic plans and policies on the future of AI. while, the principles also urge governments The U.S. National Artificial Intelligence Re- to ensure a fair transition for workers through search and Development Strategic Plan,was training programs and access to new job op- released in June, 2019, as an update of the portunities. report by the Select Committee on Artificial In- telligence of The National Science and Tech- Bipartisan Legislators On Deepfake Videos nology Council. The Computing Community Consortium (CCC) recently released the AI Senators introduced legislation intended to Roadmap Website. lessen the threat posed by “deepfake“ videos, which use AI technologies to manipulate orig- Now, the Center for Data Innovation has is- inal videos and produce misleading informa- sued a Report comparing the current stand- tion. With this legislation, the Department of ings of China, the European Union, and the Homeland Security would conduct an annual United States. Here is a summary of their pol- study of deepfakes and related content and icy recommendations: “Many nations are rac- require the department to assess the AI tech- ing to achieve a global innovation advantage in nologies used to create deepfakes. This could

26 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 lead to changes in regulations or to new regu- enabling AI research and expanding public- lations impacting the use of AI. private partnerships to accelerate advances in AI. A goal is to promote opportunities for sus- Hearing on Societal and Ethical Impacts tained investment in AI research and development and transitions into practical capabilities, The House Science, Space and Technology in collaboration with academia, industry, inter- Committee held a hearing on June 26th about national partners, and other non-Federal enti- the societal and ethical implications of artificial ties. intelligence, now available on video. The Na- tional Artificial Intelligence Research and De- Continued points from the seven Strategies in velopment Strategic Plan, released in June, is the previous Executive Order in February in- an update of the report by the Select Com- clude mittee on Artificial Intelligence of The National Science and Technology Council. • support for the development of instructional materials and teacher professional develop- On February 11, 2019, the President signed ment in computer science at all levels, with Executive Order 13859: Maintaining Ameri- emphasis at the K–12 levels, can Leadership in Artificial Intelligence. Ac- • consideration of AI as a priority area within cording to Michael Kratsios, Deputy Assis- existing Federal fellowship and service pro- tant to the President for Technology Policy, grams, this order “launched the American AI Initia- tive, which is a concerted effort to promote • development of AI techniques for human and protect AI technology and innovation in augmentation, the United States. The Initiative implements • emphasis on achieving trust: AI system de- a whole-of-government strategy in collabora- signers need to create accurate, reliable tion and engagement with the private sector, systems with informative, user-friendly inter- academia, the public, and like-minded interfaces. national partners. Among other actions, key directives in the Initiative call for Federal agen- The National Science and Technology Coun- cies to prioritize AI research and development cil (NSTC) is functioning again. NSTC is investments, enhance access to high-quality the principal means by which the Executive cyberinfrastructure and data, ensure that the Branch coordinates science and technology Nation leads in the development of technical policy across the diverse entities that make up standards for AI, and provide education and the Federal research and development enter- training opportunities to prepare the American prise. A primary objective of the NSTC is to workforce for the new era of AI.” ensure that science and technology policy decisions and programs are consistent with the The first seven strategies continue from the President’s stated goals. The NSTC prepares 2016 Plan, reflecting the reaffirmation of the research and development strategies that are importance of these strategies by multiple re- coordinated across Federal agencies aimed spondents from the public and government, at accomplishing multiple national goals. The with no calls to remove any of the strategies. work of the NSTC is organized under commit- The eighth strategy is new and focuses on tees that oversee subcommittees and working the increasing importance of effective partner- groups focused on different aspects of science ships between the Federal Government and and technology. More information is available. academia, industry, other non-Federal enti- The Office of Science and Technology Pol- ties, and international allies to generate tech- icy (OSTP) was established by the National nological breakthroughs in AI and to rapidly Science and Technology Policy, Organization, transition those breakthroughs into capabili- and Priorities Act of 1976 to provide the Pres- ties. ident and others within the Executive Office Strategy 8: Expand Public–Private Partner- of the President with advice on the scientific, ships to Accelerate Advances in AI is new in engineering, and technological aspects of the the June, 2019, plan and reflects the grow- economy, national security, homeland secu- ing importance of public-private partnerships rity, health, foreign relations, the environment,

27 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 and the technological recovery and use of re- Larry Medsker is Re- sources, among other topics. OSTP leads in- search Professor of teragency science and technology policy co- Physics and was founding ordination efforts, assists the Office of Man- director of the Data Sci- agement and Budget with an annual review ence graduate program and analysis of Federal research and devel- at The George Wash- opment in budgets, and serves as a source of ington University. He is scientific and technological analysis and judg- a faculty member in the ment for the President with respect to major GW Human-Technology policies, plans, and programs of the Federal Collaboration Lab and Government. More information is available. Ph.D. program. His research in AI includes work on artificial neural networks, hybrid Groups that advise and assist the NSTC on AI intelligent systems, and the impacts of AI on include society and policy. He is the Public Policy Officer for the ACM SIGAI.

• The Select Committee on Artiﬁcial Intelli- gence addresses Federal AI research and development activities, including those related to autonomous systems, biometric identiﬁcation, computer vision, human computer interactions, machine learning, natural language processing, and robotics. The committee supports policy on technical, national AI workforce issues

• The Subcommittee on Machine Learning and Artiﬁcial Intelligence monitors the state of the art in machine learning (ML) and arti- ﬁcial intelligence within the Federal Govern- ment, in the private sector, and internationally

• The Artiﬁcial Intelligence Research and De- velopment Interagency Working Group coordinates Federal research and development in AI and supports and coordinates activities tasked by the Select Committee on AI and the NSTC Subcommittee on Machine Learning and Artiﬁcial Intelligence.

More information available.

Please join our discussions at the SIGAI Pol- icy Blog.

28 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Advancing Non-Convex and Constrained Learning: Challenges and Opportunities Tianbao Yang (The University of Iowa; [email protected]) DOI: 10.1145/3362077.3362085

Introduction 2006a; Sra, Nowozin, & Wright, 2011; T. Yang, Mahdavi, Jin, Zhang, & Zhou, 2012), etc. In As data gets more complex and applica- spite of great promises brought by these new tions of machine learning (ML) algorithms for learning paradigms, they also bring emerging decision-making broaden and diversify, tra- challenges to the design of computationally ef- ditional ML methods by minimizing an un- ficient algorithms for big data and the analysis constrained or simply constrained convex ob- of their statistical properties. jective are becoming increasingly unsatisfac- tory. To address this new challenge, recent ML research has sparked a paradigm shift Non-Convex Learning in learning predictive models into non-convex learning and heavily constrained learning. In this section, we describe some recent ad- Non-Convex Learning (NCL) refers to a fam- vances in non-convex learning with mention- ily of learning methods that involve optimiz- ing some of our recent related results. We ing non-convex objectives. Heavily Con- will also describe their limitations and point strained Learning (HCL) refers to a family of out future directions. This article will focus on learning methods that involve constraints that studies that are concerned with algorithm de- are much more complicated than a simple sign and analysis for solving NCL and HCL norm constraint (e.g., data-dependent func- problems instead of papers that are purely tional constraints, non-convex constraints), as application-driven. It is notable that the ref- in conventional learning. This paradigm shift erences are not exhaustive due to a large vol- has already created many promising out- ume of related works. comes: (i) non-convex deep learning has Non-Convex Minimization and Deep Learn- brought breakthroughs for learning represen- ing. Deep learning can be formulated as the tations from large-scale structured data (e.g., following non-convex minimization problem: images, speech) (LeCun, Bengio, & Hinton, 2015; Krizhevsky, Sutskever, & Hinton, 2012; minw∈Rd F (w):=Ez[f(w; z)], (1) Amodei et al., 2016; Deng & Liu, 2018); z w (ii) non-convex regularizers (e.g., for enforc- where denotes a random data, and denotes the parameters of the neural network ing sparsity or low-rank) could be more effec- (w; z) tive than their convex counterparts for learn- to be learned, and f denotes the loss ing high-dimensional structured models (C.- function. Due to the success of deep learn- H. Zhang & Zhang, 2012; J. Fan & Li, 2001; ing in many areas, this problem has attracted C.-H. Zhang, 2010; T. Zhang, 2010); (iii) much attention from the community of math- constrained learning is being used to learn ematical programming and machine learning. predictive models that satisfy various con- Research has been conducted in the following straints to respect social norms (e.g., fair- directions. ness) (B. E. Woodworth, Gunasekar, Ohan- • Convergence to stationary points. For nessian, & Srebro, 2017; Hardt, Price, Srebro, general non-convex problems, it is NP-hard et al., 2016; Zafar, Valera, Gomez Rodriguez, to find a global minimizer (Hillar & Lim, & Gummadi, 2017; A. Agarwal, Beygelzimer, 2013). Hence, many studies have focused Dud´ık, Langford, & Wallach, 2018), to improve on finding stationary points of (1)(Nesterov the interpretability (Gupta et al., 2016; Canini, & Polyak, 2006; N. Agarwal, Allen Zhu, Cotter, Gupta, Fard, & Pfeifer, 2016; You, Bullins, Hazan, & Ma, 2017; Carmon, Duchi, Ding, Canini, Pfeifer, & Gupta, 2017), to en- Hinder, & Sidford, 2016; P. Xu, Roosta- hance the robustness (Globerson & Roweis, Khorasani, & Mahoney, 2017; Cartis, Gould, Copyright c 2019 by the author(s). & Toint, 2011b, 2011a; Royer & Wright,

29 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

2017; M. Liu & Yang, 2017b, 2017a; Allen- step size (Y. Xu, Lin, & Yang, 2017), and Zhu, 2017; Kohler & Lucchi, 2017; Reddi et adaptive step sizes (Kingma & Ba, 2015; al., 2017). Typically, two types of station- J. Chen & Gu, 2018; Zhou, Tang, Yang, Cao, ary points are considered, namely first-order &Gu, 2018; Zaheer, Reddi, Sachan, Kale, & stationary point and second-order station- Kumar, 2018; Luo, Xiong, Liu, & Sun, 2019; ary point. A point w∗ is called a first-order Z. Chen et al., 2019). A stagewise geometri- stationary point if it satisfies ∇F (w∗)=0.A cally decreasing step size is usually adopted point w∗ is called a second-order stationary in SGD, SHB and SNAG for deep learning, 2 if it satisfies ∇F (w∗)=0and ∇ F (w∗) 0. which starts from a relatively large step size These studies concentrate on the complex- and decreases by a constant factor after a ity analysis of first or second-order methods. number of iterations. This step size scheme Many first-order methods (e.g., stochastic has achieved the state of the art result on gradient descent (SGD)) have been proved the ImageNet classification task (He, Zhang, to converge to first-order stationary points Ren, & Sun, 2016; Real, Aggarwal, Huang, with a polynomial time complexity. In our &Le, 2019; Tan&Le, 2019). The idea study (Yan, Yang, Li, Lin, & Yang, 2018), we of adaptive step size dates back to Ada- presented the first theoretical result showing Grad (Duchi, Hazan, & Singer, 2011), which that the commonly used stochastic heavy- was proposed for convex optimization. It ball (SHB) method and stochastic Nes- has several variants with Adam (Kingma & terov’s accelerated gradient (SNAG) method Ba, 2015) being one of its most popular vari- for deep learning converge to first-order sta- ants. The adaptive algorithms have been tionary points, and also presented a unified analyzed for non-convex optimization prob- framework that subsumes SGD, SHB and lems (X. Li & Orabona, 2018; J. Chen & Gu, SNAG by varying a single parameter. More- 2018; Zhou et al., 2018; Zaheer et al., 2018; over, in (Y. Xu, Rong, & Yang, 2018)we Luo et al., 2019). presented a unified framework that can promote first-order algorithms to enjoy conver- Limitations and Future Directions. Al- gence to a second-order stationary point by though some nice results have been achieved using our proposed first-order negative cur- in non-convex optimization and learning deep vature finding procedure named NEON. neural networks, there still remain many issues that require further investigation. • Convergence to global minimizers. Re- • cently, several works have proved gradient The gap between practice and theory. descent or stochastic gradient descent can There are several limitations of existing find global minimizers of minimizing an over- analysis: (i) most existing analysis of SGD parameterized deep neural network under uses a very small step size (Ghadimi & Lan, some mild conditions of input data (Allen- 2013; Yan et al., 2018; Davis & Drusvy- Zhu, Li, & Song, 2018; Arora, Cohen, & atskiy, 2018), which is far from being prac- Hazan, 2018; Y. Li & Liang, 2018; Du, Zhai, tical; (ii) most theoretical analysis of non- Poczos, & Singh, 2018; Zou, Cao, Zhou, & convex optimization algorithms focus on op- Gu, 2018). Different from other studies that timization error; however, it is more im- focus on general non-convex minimization portant to consider the generalization per- problems, these recent works explored the formance of a stochastic optimization al- properties for overparameterized deep neu- gorithm; (iii) global analysis of SGD im- ral networks and presented sharp analysis poses strong conditions on the level of over- of (stochastic) gradient descent. parameterization (Allen-Zhu et al., 2018; Arora et al., 2018; Y. Li & Liang, 2018; Du et • Smart Step Sizes or Learning Rates. Step al., 2018; Zou et al., 2018), which is far from sizes or learning rates play an important being practical. To address the first two lim- role in an optimization algorithm for learning itations, we have conducted some prelimi- deep neural networks. Conventional polyno- nary study of SGD with a stagewise geomet- mially decreasing step sizes are observed rically decreasing step size scheme by an- to be non-effective for deep learning. Smart alyzing both the optimization error and the step size schemes have been proposed in- generalization error. Our analysis exhibits cluding stagewise geometrically decreasing that the stagewise geometrically decreasing

30 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

step scheme can leverage some nice prop- one step of Adam for updating the genera- erties of deep neural networks and enjoy tor network. Theoretically, most existing re- faster convergence for both the training error sults of min-max optimization algorithms for and testing error than using a conventional GAN are either asymptotic (Daskalakis, Ilyas, polynomially decreasing step size. Some Syrgkanis, & Zeng, 2017; Heusel, Ramsauer, important theoretical questions that deserve Unterthiner, Nessler, & Hochreiter, 2017; Na- more attention are (i) why do stochastic garajan & Kolter, 2017; Cherukuri, Gharesi- momentum methods exhibit better general- fard, & Cortes, 2017) or their analysis require ization performance than SGD (Yan et al., strong assumptions of the problem (Nagarajan 2018); (ii) how does the adaptive learning & Kolter, 2017; Grnarova, Levy, Lucchi, Hof- rate affect the generalization performance; mann, & Krause, 2017) (e.g., the problem (iii) how can we derive much sharper analy- is concave in maximization). In our recent sis of practical SGD for finding a global min- study (Lin, Liu, Rafique, & Yang, 2018), we imizer of deep learning with good general- proposed new stochastic algorithms based ization performance. on the proximal point framework for solving • Better stochastic algorithms for deep the non-convex non-concave min-max prob- learning. Beyond theoretical questions lem of GAN, and established their complexi- mentioned above, it is also important to de- ties for finding approximate first-order station- sign better stochastic algorithms for deep ary points without convex and concavity as- learning. While most recent studies focus sumptions. on designing better adaptive learning rates, Future studies in this direction should answer however, they have mostly ignored the role the following questions (i) how can we analyze of stochastic gradients itself. The learn- the generalization performance of stochastic ing rate plays its role through multiplying min-max optimization algorithms for GAN? (ii) with stochastic gradients. We believe that does GAN exhibit some nice properties as in it is important to consider the properties of deep learning that facilitates the design of bet- stochastic gradients, which essentially deter stochastic algorithms? (iii) why is the Adam pend on the data. algorithm more effective than SGD for GAN? (iv) how can we design faster stochastic algorithms for solving non-convex non-concave Non-Convex Min-Max Optimization and min-max problems with lower complexities? Generative Adversarial Networks. Re- cently, non-convex non-concave min-max Other Non-Convex Learning Problems. optimization has received increasing attention Beyond regular deep learning and GAN, non- due to its application in generative adversarial convex learning also has some important ap- networks (GAN) (Goodfellow et al., 2014; plications in machine learning. Below, we will Radford, Metz, & Chintala, 2015; Arjovsky, mention several of them. Chintala, & Bottou, 2017). GAN has emerged • to be an important paradigm of unsupervised Learning with Non-convex Regularizers. learning. It learns a generator network and a Learning with a non-convex regularizer can discriminator network in a unified framework be formulated as: by solving a min-max problem of the following minw∈Rd F (w):=Ez[f(w; z)] + R(w) form: where R(w) denotes a regularizer, which min max L(w, u), includes the indicator function of a non- w∈W u∈U convex set. Commonly used non-convex where w denotes the parameter of the gen- regularizers that have been well studied erator network and u denotes the parameter include log-sum penalty (LSP) (Candes,` of the discriminator network. Although many Wakin, & Boyd, 2008), minimax con- variants of GAN have been investigated, the cave penalty (MCP) (C.-H. Zhang, 2010), research on optimization algorithms for GAN smoothly clipped absolute deviation remains rare. In practice, most studies use a (SCAD) (J.Fan&Li, 2001), capped 1 primal-dual variant of Adam for optimization, penalty (T. Zhang, 2010), transformed which runs several steps of Adam for updat- 1 norm (S. Zhang & Xin, 2014). How- ing the discriminator network and then runs ever, there are many other interesting

31 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

non-convex regularizations (Chartrand, Li, Wang, Gong, & Yang, 2019; Y. Fan, Lyu, 2012; Chartrand & Yin, 2016; Wen, Chu, Ying, & Hu, 2017). It is also related to Liu, & Qiu, 2018). For example, one can variance-based regularization and can yield formulate learning a quantized neural net- smaller excess risk bounds (Namkoong & work as a non-convex minimization with Duchi, 2017). When the loss function a non-convex constraint. Although non- f(w, z) is non-convex in terms of w, the smooth non-convex regularization has been above problem is non-convex and concave considered in literature (Attouch, Bolte, & min-max problems. In (Rafique, Liu, Lin, Svaiter, 2013; Bolte, Sabach, & Teboulle, & Yang, 2018), we have proposed efficient 2014; Bot, Csetnek, & Laszl´ o´, 2016; H. Li stochastic algorithms for solving the above & Lin, 2015; Yu, Zheng, Marchetti-Bowick, min-max problems, and demonstrated that & Xing, 2015; L. Yang, 2018; T. Liu, Pong, it gives better performance than SGD for & Takeda, 2018; An & Nam, 2017; Zhong & learning a deep neural network in the pres- Kwok, 2014), existing results are restricted ence of imbalanced data. to deterministic optimization and asymptotic • Learning with Truncated Losses. Learn- or local convergence analysis. In our recent ing with truncated losses has long history works (Y. Xu, Jin, & Yang, 2019; Y. Xu, Qi, in statistics (Wu & Liu, 2007; Belagiannis, Lin, Jin, & Yang, 2019), we have proposed Rupprecht, Carneiro, & Navab, 2015), which new stochastic algorithms for tackling is more robust to outliers and can be formu- learning with a non-smooth non-convex lated as regularizer, and established state-of-the-art 1 n non-asymptotic convergence rates. min φ(f(w, zi)) w∈Rd n • DC programming. Difference-of-Convex i=1 (DC) programming is to solve non-convex where φ(·) is suitable concave truncation minimization problems of the following form: function (Y. Xu, Zhu, et al., 2019; Belagian- nis et al., 2015). The above problem is a min f(w) − g(w) w non-convex minimization problem. In (Y. Xu, Zhu, et al., 2019), we studied SGD for min- where both f and g are convex functions. imizing the above truncated losses and ob- DC programming finds applications in many served improved performance in the pres- machine learning problems (Le Thi, Dinh, ence of various types of outliers and noise. & Belghiti, 2014; Le Thi & Dinh, 2014; Ni- However, it remains a question whether tanda & Suzuki, 2017; Thi, Le, Phan, & SGD converges to a global minimizer. Tran, 2017; Khalaf, Astorino, d’Alessandro, & Gaudioso, 2017). For example, positive unlabeled learning problems can be formu- Heavily Constrained Learning lated as a DC programming (Kiryo, Niu, du As ML is increasingly deployed in various do- Plessis, & Sugiyama, 2017). In (Y. Xu, Qi, mains, more and more problems are being for- et al., 2019), we developed new stochastic mulated as constrained optimization problems DC algorithms for a broad family of DC prob- where constraints are introduced to account lems, and established their complexities. for other factors/concerns beyond the predic- • Distributionally Robust Optimization tion performance (B. Woodworth, Gunasekar, (DRO). DRO is to solve the following Ohannessian, & Srebro, 2017; Hardt et al., min-max problem: 2016; Globerson & Roweis, 2006b; Gupta et n al., 2016; Canini et al., 2016; Globerson & Roweis, 2006b). Recently, there is much in- min max pif(w, zi) w∈Rd p∈P i=1 terest in measuring and ensuring fairness in ML, which is important in domains protected n n where P⊆{p ∈ R , i pi =1,pi ≥ by anti-discrimination law (B. E. Woodworth 0} encodes some constraint that how far et al., 2017; Hardt et al., 2016; Zafar et al., p deviates from the empirical distribution 2017; A. Agarwal et al., 2018). For example, a pî =1/n, i =1,...,n. DRO has financial institution may want to use machine found to be effective in handling imbalanced learning methods to predict whether a partic- data (Namkoong & Duchi, 2016, 2017; Zhu, ular individual will pay back a loan or not for

32 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 making a lending decision. In this case, it is Jana, & Ray, 2018). morally and legally undesirable to discriminate Constrained convex optimization has been based on the person’s race and/or gender. A studied extensively for a few decades and variety of notions of fairness has been consid- different methods, ranging from projected ered in literature, including demographic par- gradient methods, Frank-Wolfe methods (or ity, equality of opportunity, equalized odds, conditional gradient methods), barrier meth- 80% rule, which can be modeled naturally ods, augmented Lagrangian methods, penalty as data dependent equality or inequality con- methods, level-set methods to trust-region straints (B. Woodworth et al., 2017; Hardt et methods, have been developed and studied. al., 2016; Globerson & Roweis, 2006b). However, the design of most existing con- Learning with data dependent constraints strained optimization algorithms suffers from could also arise in Interpretable learning, severe scalability issues in the presence of which requires the prediction or the predictive big data and many complex constraints due to model to be interpretable by a human. For various reasons. example, if ML is used to predict whether a The general constrained learning problem can medication is effective for a client, then the be formulated as: client wants to know why it is effective in order min f0(x), (2) to trust the medication. One way to achieve x∈X interpretable learning is to impose human- (x) ≤ =1 interpretable constraints into the learning pro- s.t. fi ri,i ,...,m (3) cess. For instance, for predicting an individual In (Mahdavi, Yang, Jin, & Zhu, 2012; T. Yang, will pay back a loan or not, it is expected the Lin, & Zhang, 2017), we developed new the- probability of paying back is likely to increase ories of projection reduced (stochastic) first- as the person’s income increases. It can be order methods with only one or a logarith- modeled as a constraint on the monotonicity mic number of projections. In (Lin, Nadara- of the predictive function respect to some fea- jah, Soheili, & Yang, 2019), we developed tures (Gupta et al., 2016). new stochastic level-set methods for a family Learning with complicated and complex con- of finite-sum constrained convex optimization straints can find applications in other scenar- problems which can guarantee the exact fea- ios. In Neyman-Pearson (NP) classification sibility of constraints. Recently, we proposed a paradigm (Rigollet & Tong, 2011), one needs class of subgradient methods for constrained to minimize false negative rate with an up- optimization where the objective function and per bound on false positive rate, where the the constraint functions are non-convex (Ma, upper bound on false positive rate is repre- Lin, & Yang, 2019). sented as a constraint. When the observed However, there still remain many challenging data are subject to some uncertainty (e.g, problems for heavily constrained learning. corruption, missing values, noise contamina- tion), many studies have formulated the task • How to efficiently handle a large number of as a constrained learning problem (Globerson constraints? & Roweis, 2006a; Sra et al., 2011). Recent • How do the constraints affect the general- works also found that imposing constrains ization performance of a learned model? on model parameters of neural networks can • be more effective than using a regularization How to establish stronger convergence for term in the objective for improving the predic- a constrained optimization with non-convex tion performance (Gouk, Frank, Pfahringer, & objectives and non-convex constraints? Cree, 2018; Ravi, Dinh, Lokhande, & Singh, 2018), and can improve the robustness of Acknowledgments learned neural networks to adversarial examples (Cisse, Bojanowski, Grave, Dauphin, & We acknowledge the support by National Sci- Usunier, 2017). The robustness of a neu- ence Foundation Career Award 1844403. Part ral network is very important for applications of the our research cited in this paper was sup- in security critical domains (e.g., autonomous ported by National Science Foundation (IIS- driving) (Carlini & Wagner, 2017; Tian, Pei, 1545995).

33 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

References tion for nonconvex and nonsmooth problems. Mathematical Programming, 146, Agarwal, A., Beygelzimer, A., Dud´ık, M., 459–494. Langford, J., & Wallach, H. (2018). A Bot, R. I., Csetnek, E. R., & Laszl´ o,´ S. C. reductions approach to fair classiﬁcation. (2016, Feb 01). An inertial forward– In Proceedings of the 35th international backward algorithm for the minimization conference on machine learning (icml) of the sum of two nonconvex functions. (pp. –). EURO Journal on Computational Opti- Agarwal, N., Allen Zhu, Z., Bullins, B., Hazan, mization, 4(1), 3–25. E., & Ma, T. (2017). Finding approx- Candes,` E. J., Wakin, M. B., & Boyd, S. P. imate local minima faster than gradient (2008, Dec 01). Enhancing sparsity by descent. In Acm symposium on theory reweighted l1 minimization. Journal of of computing (stoc) (pp. 1195–1199). Fourier Analysis and Applications, 14(5), Allen-Zhu, Z., Li, Y., & Song, Z. (2018). 877–905. A convergence theory for deep learn- Canini, K., Cotter, A., Gupta, M. R., Fard, ing via over-parameterization. CoRR, M. M., & Pfeifer, J. (2016). Fast and ﬂex- abs/1811.03962. ible monotonic functions with ensembles Allen-Zhu, Z. (2017). Natasha 2: Faster non- of lattices. In Proceedings of the 30th convex optimization than sgd. CoRR, international conference on neural infor- /abs/1708.08694/v4. mation processing systems (nips) (pp. Amodei, D., Ananthanarayanan, S., Anubhai, 2927–2935). R., Bai, J., Battenberg, E., Case, C., . . . Carlini, N., & Wagner, D. (2017). Towards Zhu, Z. (2016). Deep speech 2: End- evaluating the robustness of neural net- to-end speech recognition in english and works. In 2017 ieee symposium on se- mandarin. In Proceedings of the 33rd curity and privacy (sp) (pp. 39–57). international conference on international Carmon, Y., Duchi, J. C., Hinder, O., & Sid- conference on machine learning (icml) ford, A. (2016). Accelerated meth- (pp. 173–182). ods for non-convex optimization. CoRR, An, N. T., & Nam, N. M. (2017). Convergence abs/1611.00756. analysis of a proximal point algorithm for Cartis, C., Gould, N. I. M., & Toint, minimizing differences of functions. Op- P. L. (2011a, Dec 01). Adaptive cu- timization, 66(1), 129-147. bic regularisation methods for uncon- Arjovsky, M., Chintala, S., & Bottou, L. (2017). strained optimization. part ii: worst- Wasserstein generative adversarial net- case function- and derivative-evaluation works. In International conference on complexity. Mathematical Programming, machine learning (pp. 214–223). 130(2), 295–319. Arora, S., Cohen, N., & Hazan, E. (2018). On Cartis, C., Gould, N. I. M., & Toint, P. L. the optimization of deep networks: Im- (2011b). Adaptive cubic regularisation plicit acceleration by overparameteriza- methods for unconstrained optimization. tion. arXiv preprint arXiv:1802.06509. part i: motivation, convergence and nu- Attouch, H., Bolte, J., & Svaiter, B. F. (2013). merical results. Mathematical Program- Convergence of descent methods for ming, 127(2), 245–295. semi-algebraic and tame problems: Chartrand, R. (2012). Nonconvex splitting for proximal algorithms, forward–backward regularized low-rank+ sparse decompo- splitting, and regularized gauss–seidel sition. IEEE Transactions on Signal Pro- methods. Mathematical Programming, cessing, 60(11), 5810–5819. 137(1), 91–129. Chartrand, R., & Yin, W. (2016). Nonconvex Belagiannis, V., Rupprecht, C., Carneiro, G., sparse regularization and splitting algo- & Navab, N. (2015). Robust optimiza- rithms. In Splitting methods in communi- tion for deep regression. In Proceedings cation, imaging, science, and engineer- of the ieee international conference on ing (pp. 237–249). Springer. computer vision (pp. 2830–2838). Chen, J., & Gu, Q. (2018). Closing the gener- Bolte, J., Sabach, S., & Teboulle, M. (2014). alization gap of adaptive gradient meth- Proximal alternating linearized minimiza- ods in training deep neural networks.

34 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

arXiv preprint arXiv:1806.06763. mare at test time: Robust learning by Chen, Z., Yuan, Z., Yi, J., Zhou, B., Chen, feature deletion. In Proceedings of the E., & Yang, T. (2019). Universal stage- 23rd international conference on ma- wise learning for non-convex problems chine learning (pp. 353–360). with convergence on averaged solutions. Globerson, A., & Roweis, S. (2006b). Night- In 7th international conference on learn- mare at test time: robust learning by ing representations, ICLR 2019, new or- feature deletion. In Proceedings of the leans, la, usa, may 6-9, 2019. 23rd international conference on ma- Cherukuri, A., Gharesifard, B., & Cortes, J. chine learning (pp. 353–360). (2017). Saddle-point dynamics: con- Goodfellow, I., Pouget-Abadie, J., Mirza, M., ditions for asymptotic stability of saddle Xu, B., Warde-Farley, D., Ozair, S., . . . points. SIAM Journal on Control and Op- Bengio, Y. (2014). Generative adver- timization, 55(1), 486–511. sarial nets. In Advances in neural infor- Cisse, M., Bojanowski, P., Grave, E., Dauphin, mation processing systems (pp. 2672– Y., & Usunier, N. (2017). Parseval 2680). networks: Improving robustness to ad- Gouk, H., Frank, E., Pfahringer, B., & Cree, versarial examples. In Proceedings of M. (2018). Regularisation of neural net- the 34th international conference on ma- works by enforcing lipschitz continuity. chine learning-volume 70 (pp. 854–863). arXiv preprint arXiv:1804.04368. Daskalakis, C., Ilyas, A., Syrgkanis, V., & Grnarova, P., Levy, K. Y., Lucchi, A., Hofmann, Zeng, H. (2017). Training gans with opti- T., & Krause, A. (2017). An online learn- mism. CoRR, abs/1711.00141. ing approach to generative adversarial Davis, D., & Drusvyatskiy, D. (2018). Stochas- networks. CoRR, abs/1706.03269. tic subgradient method converges at the Gupta, M. R., Cotter, A., Pfeifer, J., Voevod- rate o(k−1/4) on weakly convex functions. ski, K., Canini, K. R., Mangylov, A., arXiv preprint arXiv:1802.02988. . . . Esbroeck, A. V. (2016). Mono- Deng, L., & Liu, Y. (2018). Deep learning in tonic calibrated interpolated look-up ta- natural language processing. Springer. bles. Journal of Machine Learning Re- Du, S. S., Zhai, X., Poczos, B., & Singh, A. search (JMLR), 17, 109:1–109:47. (2018). Gradient descent provably op- Hardt, M., Price, E., Srebro, N., et al. (2016). timizes over-parameterized neural net- Equality of opportunity in supervised works. arXiv preprint arXiv:1810.02054. learning. In Advances in neural infor- Duchi, J., Hazan, E., & Singer, Y. (2011). mation processing systems (pp. 3315– Adaptive subgradient methods for on- 3323). line learning and stochastic optimization. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Journal of Machine Learning Research, Deep residual learning for image recog- 12(Jul), 2121–2159. nition. In Proceedings of the ieee con- Fan, J., & Li, R. (2001). Variable se- ference on computer vision and pattern lection via nonconcave penalized likeli- recognition (pp. 770–778). hood and its oracle properties. Journal Heusel, M., Ramsauer, H., Unterthiner, T., of the American Statistical Association, Nessler, B., & Hochreiter, S. (2017). 96(456), 1348–1360. Gans trained by a two time-scale update Fan, Y., Lyu, S., Ying, Y., & Hu, B. (2017). rule converge to a local nash equilib- Learning with average top-k loss. In rium. In Advances in neural information Advances in neural information process- processing systems 30 (nips) (pp. 6629– ing systems 30: Annual conference on 6640). neural information processing systems Hillar, C. J., & Lim, L.-H. (2013, November). 2017, 4-9 december 2017, long beach, Most tensor problems are np-hard. Jour- ca, USA (pp. 497–505). nal of ACM, 60(6), 45:1–45:39. Ghadimi, S., & Lan, G. (2013). Stochastic ﬁrst- Khalaf, W., Astorino, A., d’Alessandro, P., & and zeroth-order methods for nonconvex Gaudioso, M. (2017). A dc optimization- stochastic programming. SIAM Journal based clustering technique for edge de- on Optimization, 23(4), 2341–2368. tection. Optimization Letters, 11(3), 627– Globerson, A., & Roweis, S. (2006a). Night- 640.

35 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Kingma, D. P., & Ba, J. (2015). Adam: A ity. arXiv preprint arXiv:1810.10207. method for stochastic optimization. In Lin, Q., Nadarajah, S., Soheili, N., & Yang, 3rd international conference on learning T. (2019). A data efficient and feasible representations, ICLR 2015, san diego, level set method for stochastic convex ca, usa, may 7-9, 2015, conference track optimization with expectation constraints. proceedings. Retrieved from http:// CoRR, abs/1908.03077. arxiv.org/abs/1412.6980 Liu, M., & Yang, T. (2017a). On noisy negative Kiryo, R., Niu, G., du Plessis, M. C., & curvature descent: Competing with gra- Sugiyama, M. (2017). Positive-unlabeled dient descent for faster non-convex opti- learning with non-negative risk estimator. mization. CoRR, abs/1709.08571. In Advances in neural information pro- Liu, M., & Yang, T. (2017b). Stochastic cessing systems 30 (pp. 1675–1685). non-convex optimization with strong high Kohler, J. M., & Lucchi, A. (2017). Sub- probability second-order convergence. sampled cubic regularization for non- CoRR, abs/1710.09447. convex optimization. In Proceedings of Liu, T., Pong, T. K., & Takeda, A. (2018, the international conference on machine Sep 08). A successive difference-of- learning (icml) (pp. 1895–1904). convex approximation method for a class Krizhevsky, A., Sutskever, I., & Hinton, G. E. of nonconvex nonsmooth optimization (2012). Imagenet classification with deep problems. Mathematical Programming. convolutional neural networks. In Ad- Luo, L., Xiong, Y., Liu, Y., & Sun, X. (2019). vances in neural information processing Adaptive gradient methods with dynamic systems (nips) (pp. 1106–1114). bound of learning rate. arXiv preprint LeCun, Y., Bengio, Y., & Hinton, G. E. (2015). arXiv:1902.09843. Deep learning. Nature, 521(7553), 436– Ma, R., Lin, Q., & Yang, T. (2019). Proxi- 444. mally constrained methods for weakly Le Thi, H. A., & Dinh, T. P. (2014). Dc pro- convex optimization with weakly gramming in communication systems: convex constraints. arXiv preprint challenging problems and methods. Viet- arXiv:1908.01871. nam Journal of Computer Science, 1(1), Mahdavi, M., Yang, T., Jin, R., & Zhu, S. 15–28. (2012). Stochastic gradient descent with Le Thi, H. A., Dinh, T. P., & Belghiti, M. (2014). only one projection. In Advances in neu- Dca based algorithms for multiple se- ral information processing systems (nips) quence alignment (msa). Central Eu- (p. 503-511). ropean Journal of Operations Research, Nagarajan, V., & Kolter, J. Z. (2017). Gradi- 22(3), 501–524. ent descent GAN optimization is locally Li, H., & Lin, Z. (2015). Accelerated proxi- stable. In Advances in neural information mal gradient methods for nonconvex pro- processing systems 30 (nips) (pp. 5591– gramming. In Proceedings of the 28th 5600). international conference on neural infor- Namkoong, H., & Duchi, J. C. (2016). mation processing systems - volume 1 Stochastic gradient methods for dis- (pp. 379–387). tributionally robust optimization with f- Li, X., & Orabona, F. (2018). On the con- divergences. In Advances in neural infor- vergence of stochastic gradient descent mation processing systems (pp. 2208– with adaptive stepsizes. arXiv preprint 2216). arXiv:1805.08114. Namkoong, H., & Duchi, J. C. (2017). Li, Y., & Liang, Y. (2018). Learning overparam- Variance-based regularization with con- eterized neural networks via stochastic vex objectives. In Advances in neu- gradient descent on structured data. In ral information processing systems (pp. Advances in neural information process- 2971–2980). ing systems (neurips) (pp. 8157–8166). Nesterov, Y., & Polyak, B. T. (2006). Cu- Lin, Q., Liu, M., Rafique, H., & Yang, bic regularization of newton method and T. (2018). Solving weakly-convex- its global performance. Math. Program., weakly-concave saddle-point problems 108(1), 177–205. as weakly-monotone variational inequal- Nitanda, A., & Suzuki, T. (2017). Stochas-

36 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

tic difference of convex algorithm and its neural-network-driven autonomous cars. application to training deep boltzmann In Proceedings of the 40th international machines. In Artificial intelligence and conference on software engineering (pp. statistics (pp. 470–478). 303–314). Radford, A., Metz, L., & Chintala, S. Wen, F., Chu, L., Liu, P., & Qiu, R. C. (2018). (2015). Unsupervised representation A survey on nonconvex regularization- learning with deep convolutional genera- based sparse and low-rank recovery in tive adversarial networks. arXiv preprint signal processing, statistics, and ma- arXiv:1511.06434. chine learning. IEEE Access, 6, 69883– Rafique, H., Liu, M., Lin, Q., & Yang, T. (2018). 69906. Non-convex min-max optimization: Prov- Woodworth, B., Gunasekar, S., Ohannessian, able algorithms and applications in ma- M. I., & Srebro, N. (2017). Learning non- chine learning. CoRR, abs/1810.02060. discriminatory predictors. arXiv preprint Ravi, S. N., Dinh, T., Lokhande, V. S. R., arXiv:1702.06081. & Singh, V. (2018). Constrained deep Woodworth, B. E., Gunasekar, S., Ohan- learning using conditional gradient and nessian, M. I., & Srebro, N. (2017). applications in computer vision. arXiv Learning non-discriminatory predictors. preprint arXiv:1803.06453. In Proceedings of the 30th conference Real, E., Aggarwal, A., Huang, Y., & Le, Q. V. on learning theory, COLT 2017, ams- (2019). Regularized evolution for image terdam, the netherlands, 7-10 july 2017 classifier architecture search. In Pro- (pp. 1920–1953). ceedings of the aaai conference on ar- Wu, Y., & Liu, Y. (2007). Robust trun- tificial intelligence (Vol. 33, pp. 4780– cated hinge loss support vector ma- 4789). chines. Journal of the American Statis- Reddi, S. J., Zaheer, M., Sra, S., Poczos, B., tical Association, 102(479), 974–983. Bach, F., Salakhutdinov, R., & Smola, Xu, P., Roosta-Khorasani, F., & Mahoney, A. J. (2017). A generic approach for M. W. (2017). Newton-type meth- escaping saddle points. arXiv preprint ods for non-convex optimization under arXiv:1709.01434. inexact hessian information. CoRR, Rigollet, P., & Tong, X. (2011, November). abs/1708.07164. Neyman-pearson classification, convex- Xu, Y., Jin, R., & Yang, T. (2019). Stochas- ity and stochastic constraints. J. Mach. tic proximal gradient methods for non- Learn. Res., 12, 2831–2855. smooth non-convex regularized prob- Royer, C. W., & Wright, S. J. (2017). Complex- lems. arXiv preprint arXiv:1902.07672. ity analysis of second-order line-search Xu, Y., Lin, Q., & Yang, T. (2017). Stochastic algorithms for smooth nonconvex opti- convex optimization: Faster local growth mization. CoRR, abs/1706.03131. implies faster global convergence. In Sra, S., Nowozin, S., & Wright, S. J. (2011). Proceedings of the 34th international Optimization for machine learning. The conference on machine learning-volume MIT Press. 70 (pp. 3821–3830). Tan, M., & Le, Q. V. (2019). Efficient- Xu, Y., Qi, Q., Lin, Q., Jin, R., & Yang, T. net: Rethinking model scaling for convo- (2019). Stochastic optimization for DC lutional neural networks. arXiv preprint functions and non-smooth non-convex arXiv:1905.11946. regularizers with non-asymptotic conver- Thi, H. A. L., Le, H. M., Phan, D. N., & gence. In Proceedings of the 36th inter- Tran, B. (2017). Stochastic dca for national conference on machine learn- the large-sum of non-convex functions ing, ICML 2019, 9-15 june 2019, long problem and its application to group vari- beach, california, USA (pp. 6942–6951). able selection in classification. In Pro- Xu, Y., Rong, J., & Yang, T. (2018). First-order ceedings of the 34th international con- stochastic algorithms for escaping from ference on machine learning-volume 70 saddle points in almost linear time. In Ad- (pp. 3394–3403). vances in neural information processing Tian, Y., Pei, K., Jana, S., & Ray, B. (2018). systems (neurips) (pp. 5530–5540). Deeptest: Automated testing of deep- Xu, Y., Zhu, S., Yang, S., Zhang, C., Jin, R.,

37 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

& Yang, T. (2019). Learning with non- Curran Associates, Inc. Retrieved from convex truncated losses by SGD. In Pro- http://papers.nips.cc/paper/ ceedings of the thirty-fifth conference on 8186-adaptive-methods-for uncertainty in artificial intelligence, UAI -nonconvex-optimization.pdf 2019, tel aviv, israel, july 22-25, 2019 Zhang, C.-H. (2010). Nearly unbiased vari- (p. 244). able selection under minimax concave Yan, Y., Yang, T., Li, Z., Lin, Q., & Yang, Y. penalty. The Annals of Statistics, 38, 894 (2018). A unified analysis of stochastic – 942. momentum methods for deep learning. In Proceedings of the twenty-seventh Zhang, C.-H., & Zhang, T. (2012, 11). A international joint conference on artifi- general theory of concave regularization cial intelligence, IJCAI 2018, july 13-19, for high-dimensional sparse estimation 2018, stockholm, sweden. (pp. 2955– problems. Statistical Science, 27(4), 2961). 576–593. doi: 10.1214/12-STS399 Yang, L. (2018). Proximal gradient method Zhang, S., & Xin, J. (2014). Minimization of with extrapolation and line search for a transformed l 1 penalty: Theory, differ- class of nonconvex and nonsmooth prob- ence of convex function algorithm, and lems. CoRR, abs/1711.06831. robust application in compressed sens- Yang, T., Lin, Q., & Zhang, L. (2017). A ing. CoRR, abs/1411.5735. richer theory of convex constrained optimization with reduced projections and Zhang, T. (2010, March). Analysis of multi- improved rates. In Proceedings of the stage convex relaxation for sparse reg- 34th international conference on ma- ularization. J. Mach. Learn. Res., 11, chine learning (icml) (p. -). 1081–1107. Yang, T., Mahdavi, M., Jin, R., Zhang, L., & Zhong, W., & Kwok, J. T. (2014). Gradient de- Zhou, Y. (2012). Multiple kernel learn- scent with proximal average for noncon- ing from noisy labels by stochastic pro- vex and composite regularization. In Pro- gramming. In Proceedings of the inter- ceedings of the twenty-eighth AAAI con- national conference on machine learning ference on artificial intelligence, july 27 (icml) (pp. 233–240). -31, 2014, quebec´ city, quebec,´ canada. You, S., Ding, D., Canini, K. R., Pfeifer, J., & (pp. 2206–2212). Gupta, M. R. (2017). Deep lattice net- Zhou, D., Tang, Y., Yang, Z., Cao, Y., works and partial monotonic functions. In & Gu, Q. (2018). On the conver- Advances in neural information process- gence of adaptive gradient methods for ing systems 30 (nips) (pp. 2985–2993). nonconvex optimization. arXiv preprint Yu, Y., Zheng, X., Marchetti-Bowick, M., & arXiv:1808.05671. Xing, E. P. (2015). Minimizing nonconvex non-separable functions. In The 17th in- Zhu, D., Li, Z., Wang, X., Gong, B., & Yang, ternational conference on artificial intelli- T. (2019). A robust zero-sum game gence and statistics (AISTATS). framework for pool-based active learn- Zafar, M. B., Valera, I., Gomez Rodriguez, M., ing. In The 22nd international conference & Gummadi, K. P. (2017). Fairness be- on artificial intelligence and statistics (pp. yond disparate treatment and disparate 517–526). impact: Learning classification without Zou, D., Cao, Y., Zhou, D., & Gu, Q. (2018). disparate mistreatment. In Proceedings Stochastic gradient descent optimizes of the 26th international conference on over-parameterized deep relu networks. world wide web (pp. 1171–1180). CoRR, abs/1811.08888. Zaheer, M., Reddi, S., Sachan, D., Kale, S., & Kumar, S. (2018). Adaptive methods for nonconvex optimization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, & R. Garnett (Eds.), Advances in neural information processing systems 31 (pp. 9793–9803).

38 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Tianbao Yang is an assistant professor at the Computer Science De- partment at the Univer- sity of Iowa since 2014. He was a researcher at NEC Laboratories Amer- ica, and a researcher at GE Global Research before joining UIowa. He has won the best student paper award at COLT 2012 and the NSF Career Award. He is an associate editor for Neurocomputing Journal, and the Jour- nal of Mathematical Foundations of Comput- ing. He has served as senior program committee member for AAAI and IJCAI and reviewer for AISTATS, ICML and NeurIPS.

39 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Considerations for AI Fairness for People with Disabilities Shari Trewin (IBM; [email protected]) Sara Basson (Google Inc.; [email protected]) Michael Muller (IBM Research; michael [email protected]) Stacy Branham (University of California, Irvine; [email protected]) Jutta Treviranus (OCAD University; [email protected]) Daniel Gruen (IBM Research; daniel [email protected]) Daniel Hebert (IBM; [email protected]) Natalia Lyckowski (IBM; [email protected]) Erich Manser (IBM; [email protected]) DOI: 10.1145/3362077.3362086

Abstract Introduction Systems that leverage Artificial Intelligence In society today, people experiencing disabil- are becoming pervasive across industry sec- ity can face discrimination. As artificial intel- tors (Costello, 2019), as are concerns that ligence solutions take on increasingly impor- these technologies can unintentionally ex- tant roles in decision-making and interaction, clude or lead to unfair outcomes for marginal- they have the potential to impact fair treatment ized populations (Bird, Hutchinson, Ken- of people with disabilities in society both pos- thapadi, Kiciman, & Mitchell, 2019)(Cutler, itively and negatively. We describe some of Pribik, & Humphrey, 2019)(IEEE & Sys- the opportunities and risks across four emerg- tems, 2019)(Kroll et al., 2016)(Lepri, Oliver, ing AI application areas: employment, educa- Letouze,´ Pentland, & Vinck, 2018). Initiatives tion, public safety, and healthcare, identified to improve AI fairness for people across racial in a workshop with participants experiencing (Hankerson et al., 2016), gender (Hamidi, a range of disabilities. In many existing situ- Scheuerman, & Branham, 2018a), and other ations, non-AI solutions are already discrimi- identities are emerging, but there has been natory, and introducing AI runs the risk of sim- relatively little work focusing on AI fairness ply perpetuating and replicating these flaws. for people with disabilities. There are nu- We next discuss strategies for supporting fair- merous examples of AI that can empower ness in the context of disability throughout the people with disabilities, such as autonomous AI development lifecycle. AI systems should vehicles (Brewer & Kameswaran, 2018) and be reviewed for potential impact on the user voice agents (Pradhan, Mehta, & Findlater, in their broader context of use. They should 2018) for people with mobility and vision im- offer opportunities to redress errors, and for pairments. However, AI solutions may also users and those impacted to raise fairness result in unfair outcomes, as when Idahoans concerns. People with disabilities should be with cognitive/learning disabilities had their included when sourcing data to build models, healthcare benefits reduced based on biased and in testing, to create a more inclusive and AI (K.W. v. Armstrong, No. 14-35296 (9th robust system. Finally, we offer pointers into Cir. 2015) :: Justia, 2015). These scenar- an established body of literature on human- ios suggest that the prospects of AI for peo- centered design processes and philosophies ple with disabilities are promising yet fraught that may assist AI and ML engineers in inno- with challenges that require the sort of upfront vating algorithms that reduce harm and ulti- attention to ethics in the development process mately enhance the lives of people with dis- advocated by scholars (Bird et al., 2019) and abilities. practitioners (Cutler et al., 2019). The challenges of ensuring AI fairness in the context of disability emerge from multi- Copyright c 2019 by the author(s). ple sources. From the very beginning of al-

40 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 gorithmic development, in the problem scop- Related Work ing stage, bias can be introduced by lack of awareness of the experiences and use The 2019 Gartner CIO survey (Costello, 2019) cases of people with disabilities. Since sys- of 3000 enterprises across major industries tems are predicated on data, in data sourc- reported that 37% have implemented some ing and data pre-processing stages, it is crit- form of AI solution, an increase of 270% ical to gather data that include people with over the last four years. In parallel, there disabilities and to ensure that these data are is increasing recognition that intelligent sys- not completely subsumed by data from pre- tems should be developed with attention to sumed “normative” populations. This leads the ethical aspects of their behavior (Cutler to a potential conundrum. The data need to et al., 2019)(IEEE & Systems, 2019), and be gathered in order to be reflected in the that fairness should be considered upfront, models, but confidentiality and privacy, espe- rather than as an afterthought (Bird et al., cially as regards disability status, might make 2019). IEEE’s Global Initiative on Ethics of collecting these data difficult (for developers) Autonomous and Intelligent Systems is devel- or dangerous (for subjects) (Faucett, Ring- oping a series of international standards for land, Cullen, & Hayes, 2017)(von Schrader, such processes (Koene, Smith, Egawa, Man- Malzer, & Bruyere` ,2014). Another area dalh, & Hatada, 2018), including a process to address during model training and test- for addressing ethical concerns during design ing is the potential for model bias. Ow- (P7000), and the P7003 Standard for Algorith- ing to intended or unintended bias in the mic Bias Considerations (Koene, Dowthwaite, data, the model may inadvertently enforce & Seth, 2018). There is ongoing concern and or reinforce discriminatory patterns that work discussion about accountability for potentially against people with disabilities (Janssen & harmful decisions made by algorithms(Kroll Kuk, 2016). We advocate for increased aware- et al., 2016)(Lepri et al., 2018), with some ness of these patterns, so we can avoid repli- new academic initiatives – like one at George- cation of past bias into future algorithmic deci- town’s Institute for Tech Law & Policy (Givens, sions, as has been well-documented in bank- 2019), and a workshop at the ASSETS 2019 ing (Bruckner, 2018)(Chander, 2017)(Hurley conference(Trewin et al., 2019) – focusing & Adebayo, 2016). Finally, once a trained specifically on AI and Fairness for People with model is incorporated in an application, it is Disabilities. then critical to test with diverse users, particularly those deemed as outliers. This pa- Any algorithmic decision-process can be bi- per provides a number of recommendations ased, and the FATE/ML community is actively towards overcoming these challenges. developing approaches for detection and re- mediation of bias (Kanellopoulos, 2018)(Lohia et al., 2019). Williams, Brooks and Shmar- gad show how racial discrimination can arise In the remainder of this article, we overview in employment and education even without the nascent area of AI Fairness for People with having social category information, and how Disabilities as a practical pursuit and an aca- the lack of category information makes such demic discipline. We provide a series of exam- biases harder to detect (Williams, Brooks, & ples that demonstrate the potential for harm to Shmargad, 2018). Although they argue for people with disabilities across four emerging inclusion of social category information in al- AI application areas: employment, education, gorithmic decision-making, they also acknowl- public safety, and healthcare. Then, we iden- edge the potential harm that can be caused tify strategies of developing AI algorithms that to an individual by revealing sensitive social resist reifying systematic societal exclusions data such as immigration status. Selbst et al. at each stage of AI development. Finally, we argue that purely algorithmic approaches are offer pointers into an established body of lit- not sufficient, and the full social context of de- erature on human-centered design processes ployment must be considered if fair outcomes and philosophies that may assist AI and ML are to be achieved (Selbst, Boyd, Friedler, engineers in innovating algorithms that reduce Venkatasubramanian, & Vertesi, 2019). harm and – as should be our ideal – ultimately enhance the lives of people with disabilities. Some concerns about AI fairness in the

41 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 context of individuals with disabilities or tentional. For example, qualified deaf candi- neurological or sensory differences are dates who speak through an interpreter may now being raised (Fruchterman & Mellea, be screened out for a position requiring ver- 2018)(Guo, Kamar, Vaughan, Wallach, bal communication skills, even though they & Morris, 2019)(Lewis, 2019)(Treviranus, could use accommodations to do the job ef- 2019)(Trewin, 2018a), but research in this fectively. Additional discriminatory practices area is sparse. Fruchterman and Mel- are particularly damaging to this population, lea (Fruchterman & Mellea, 2018) outline the where employment levels are already low: In widespread use of AI tools in employment and 2018, the employment rate for people with recruiting, and highlight some potentially se- disabilities was 19.1%, while the employment rious implications for people with disabilities, percentage for people without disabilities was including the analysis of facial movements 65.9% (Bureau of Labor Statistics, 2019). and voice in recruitment, personality tests that Employers are increasingly relying on technol- disproportionately screen out people with dis- ogy in their hiring practices. One of their sell- abilities, and the use of variables that could be ing points is the potential to provide a fairer discriminatory, such as gaps in employment. recruitment process, not influenced by an in- “Advocates for people with disabilities should dividual recruiter’s bias or lack of knowledge. be looking at the proxies and the models Machine learning models are being used used by AI vendors for these “hidden” tools for candidate screening and matching job- of discrimination” (Fruchterman & Mellea, seekers with available positions. There are AI- 2018). driven recruitment solutions on the market today that analyze online profiles and resumes, Motivating Examples the results of online tests, and video interview data, all of which raise potential concerns for In October 2018, a group of 40 disability advo- disability discrimination (Fruchterman & Mel- cates, individuals with disabilities, AI and ac- lea, 2018). While the use of AI in HR and cessibility researchers and practitioners from recruitment is an increasing trend (Faggella, industry and academia convened in a work- 2019), there are already cautionary incidents shop (Trewin, 2018b) to identify and discuss of discrimination, as when Amazon’s AI re- the topic of fairness for people with disabil- cruiting solution “learned” to devalue resumes ities in light of the increasing mainstream of women (Dastin, 2018). application of AI solutions in many industries (Costello, 2019). This section describes The workshop identified several risk scenar- some of the opportunities and risks identi- ios: fied by the workshop participants in the areas • of employment, education, public safety and A deaf person may be the first person us- healthcare. ing sign language interpretation to apply to an organization. Candidate screening models that learn from the current workforce will Employment perpetuate the status quo, and the biases of People with disabilities are no strangers to dis- the past. They will likely exclude candidates crimination in hiring practices. In one recent with workplace differences, including those field study, disclosing a disability (spinal cord who use accommodations to perform their injury or Asperger’s Syndrome) in a job appli- work. cation cover letter resulted in 26% fewer pos- • An applicant taking an online test using as- itive responses from employers, even though sistive technology may take longer to an- the disability was not likely to affect productiv- swer questions, especially if the test itself ity for the position (Ameri et al., 2018). When has not been well designed for accessibil- it comes to inclusive hiring, it has been shown ity. Models that use timing information may that men and those who lack experience with end up systematically excluding assistive disability tend to have more negative affective technology users. Resumes and job ap- reactions to working with individuals with dis- plications may not contain explicit informa- abilities (Popovich, Scherbaum, Scherbaum, tion about a person’s disability, but other & Polinko, 2003). Exclusion can be unin- variables may be impacted, including gaps

42 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

in employment, school attended, or time to platforms do not consider the needs of all perform an online task. learners (Cinquin, Guitton, & Sauzeon´ , 2019). • An applicant with low facial affect could AI in the education market is being driven be screened out by a selection process by the rapid transition from onsite classroom that uses video analysis of eye gaze, voice based education to online learning. Institu- characteristics, or facial movements, even tions can now expand their online learning though she is highly skilled. This type of initiatives to reach more students in a cost- screening poses a barrier for anyone whose effective manner. Industry analyst, Global appearance, voice or facial expression dif- Market Insights (Bhutani & Wadhwani, 2019), fers from the average. It could exclude autis- predicts that the market will grow to a $6 Bil- tic individuals or blind applicants who do not lion dollar industry by 2024. The new genera- make eye contact, deaf people and others tion of online learning platforms are integrated who do not communicate verbally, people with AI technologies and use them to person- with speech disorders or facial paralysis, or alize learning (and testing) for each student, people who have survived a stroke, to name among other applications. Two examples of afew. providers of these systems are from tradi- When the available data do not include many tional Learning Management System (LMS) people with disabilities, and reflect existing bi- vendors like Blackboard; and more recently ases, and the deployed systems rely on prox- from the Massive Open Online Course (MooC) ies that are impacted by disability, the risk of providers like edX. unfair treatment in employment is significant. Personalized learning could provide enor- We must seek approaches that do not perpet- mous benefits for learners with disabilities, uate the biases of the past, or introduce new e.g. (Morris, Kirschbaum, & Picard, 2010). barriers by failing to recognize qualified can- It could be as simple as augmenting existing didates because they are different, or use ac- content with additional illustrations and pic- commodations to do their work. tures for students who are classified as visual learners, or as complex as generation Education of personalized user interfaces (Gajos, Wob- brock, & Weld, 2007). For non-native lan- In the United States, people with disabilities guage speakers, including deaf learners, the have historically been denied access to free system could provide captions for video con- public education (Dudley-Marling & Burns, tent so the student can read along with the 2014)(Obiakor, Harris, Mutua, Rotatori, & Al- lecture. gozzine, 2012). It was nearly 20 years after the passing of Brown v. Board of Educa- Any system that makes inferences about a tion, which desegregated public schools along student’s knowledge and abilities based on racial lines, that the Education for All Hand- their online interactions runs the risk of misin- icapped Children Act was passed (Dudley- terpreting and underestimating students with Marling & Burns, 2014), mandating that all disabilities. Students whose learning capabil- students are entitled to a “free and appro- ities or styles are outside the presumed norm priate public education” in the “least restric- may not receive fair treatment. For example, tive environment.” Prior to 1975, a mere one if there is a rigid time constraint for completing in five learners with disabilities had access a test or a quiz, a student that has a cognitive to public school environments, often in seg- disability or test anxiety where they process regated classrooms (Dudley-Marling & Burns, information more slowly than other students 2014). Despite great strides, some learn- would be assessed as being less capable than ers with disabilities still cannot access inte- they are. grated public learning environments (Dudley- Marling & Burns, 2014), K-12 classroom tech- Unlike other areas, in an educational setting, nologies are often inaccessible (Shaheen & disability information may often be available, Lazar, 2018), postsecondary online learning and the challenge is to provide differentiated materials are often inaccessible (Burgstahler, education for a wide range of people, without 2015)(Straumsheim, 2017), and e-learning introducing bias against disability groups.

43 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Public Safety identified as being angry, and a potential security threat. Someone with an altered gait could People with disabilities are much more likely be using a prosthesis, not hiding a weapon. to be victims of violent crime than people without disabilities (Harrell, 2017). Threats come not only from other citizens, but also from law People with cognitive disabilities may be at es- enforcement itself; for example, police offi- pecially high risk of being misidentified as a cers can misinterpret people with disabilities potential threat. Combining this with the need as being uncooperative or even threatening, to respond quickly to genuine threats creates and deprive them of access to Miranda warn- a dangerous situation and requires careful de- ings (US Department of Justice Civil Rights sign of the AI system and its method of de- Division, 2006). Law enforcement’s implicit ployment. bias and discrimination towards people with disabilities, as well as the potential for technology to address these challenges, are both There may also be opportunities for AI to im- featured in the 2015 Final Report of the Pres- prove public safety for people with disabil- ident’s Task Force on 21st Century Polic- ities. For example, AI-based interpretation ing (Policing, 2015). could be trained to ’understand’ a wide range of behaviors including hand flapping, pacing The application of AI technology to identify and sign language, as normal. A recent sur- threats to public safety and enforce the law is vey and interview study of people who are highly controversial (McCarthy, 2019). This in- blind (Branham et al., 2017) suggests that fa- cludes technology for identifying people, rec- cial and image recognition technologies could ognizing individuals, and for interpreting be- better support personal safety for individuals havior (for example, whether someone is act- with sensory disabilities. They may support ing in a suspicious manner). Aside from the locating police officers and identifying fraudu- threat to personal privacy, the potential for lent actors claiming to be officials. They may errors and biased performance is very real. allow a person who is blind or deaf to be made While public discourse and academic atten- aware of a weapon being brandished or dis- tion has so far focused on racial and gen- charged. They may support access to facial der disparities, workshop participants identi- cues for more cautious and effective interac- fied serious concerns and also some oppor- tions with a potential aggressor or a police tunities for people with disabilities. officer. These technologies may even allow Autonomous vehicles must be able to iden- blind individuals to provide more persuasive tify people in the environment with great preci- evidence to catch their perpetrators. sion. They must reliably recognize individuals who use different kinds of wheelchair and mobility devices, or move in an unusual way. One When considering the ethics of proposed workshop participant described a wheelchair- projects in this space, the potential risks for in- using friend who propels themselves back- dividuals with disabilities should also be evalu- wards with their feet. This is an unusual ated and addressed in the overall design. For method of getting around, but the ability to rec- example, a system could highlight someone ognize and correctly identify such outlier indi- with an altered gait, and list possibilities as viduals is a matter of life and death. someone hiding a weapon, or someone using a prosthesis or mobility device. In a situa- Another participant had recently observed, a tion where facial recognition is being used, a dishevelled man pacing restlessly in an air- person’s profile could help responders to avoid port lounge, muttering to himself, clearly in a misunderstanding, but again this comes at the state of high stress. His behavior could be in- cost of sacrificing privacy, and potentially do- terpreted by both humans and AI analysis as ing harm to other marginalized groups (Hamidi a potential threat. Instead he may be show- et al., 2018a). An overall balance must be ing signs of an anxiety disorder, autism, or a found between using AI as a tool for main- strong fear of flying. Deaf signers’ strong facial taining public safety while minimizing negative expressions can be misinterpreted (Shaffer & outcomes for vulnerable groups and outlier in- Rogan, 2018), leading to them being wrongly dividuals.

44 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Healthcare For example, if speech pauses are used to diagnose conditions like Alzheimer’s disease, a Today there are large disparities in access to person whose speech is already affected by a healthcare for people with disabilities (Iezzoni, disability may be wrongly diagnosed, or their 2011)(Krahn, Walker, & Correa-De-Araujo, diagnosis may be missed because the system 2015), especially those with developmen- does not work for them. Pauses in speech can tal disabilities (Krahn, Hammond, & Turner, be because person is a non-native speaker; 2006). Patients who are non-verbal or pa- and not a marker of disease. tients with cognitive impairments are often Just as we have seen in the domains of under-served or incorrectly served (Barnett, employment, education, and public safety, if McKee, Smith, & Pearson, 2011)(Krahn & healthcare applications are built for the ex- Fox, 2014)(Krahn et al., 2015). Deaf patients tremes of currently excluded populations, the are often misdiagnosed with having a mental solution stands to improve fairness in access, illness or a disorder, because of lack of cultural instead of locking people out. Across all do- awareness or language barrier (Glickman, mains, AI applications pose both risks and op- 2007)(Pollard, 1994). Another area that is portunities for people with disabilities. The underserved in the current model are patients question remains: how and when can fairness with rare diseases or genetic disorders, that for people with disabilities be implemented do not fall within standard protocols (Wastfelt, in the software development process towards Fadeel, & Henter, 2006). For older adults minimizing risks and maximizing benefits? In with deteriorating health, this may lead to un- the following section, we address this question wanted institutionalization. Promising techno- for each stage of the AI development process. logical developments, many of which include AI, abound, but need to better incorporate target users in the development process (Haigh Considerations for AI Practitioners & Yanco, 2002). In this section, we recommend ways AI prac- AI applications in healthcare could help to titioners can be aware of, and work towards overcome some of the barriers preventing fairness and inclusion for people with disabil- people getting access to the care and preven- ities in their AI-based applications. The sec- tative care they need. For example, a non- tion is organized around the typical stages of verbal person may have difficulty communi- AI model development: problem scoping, data cating a problem they are experiencing. With sourcing, pre-processing, model selection and respect to pain management or prescription training, and incorporating AI in an application. delivery, AI can remove the requirement that patients advocate on their own behalf. For complex cases where disabilities or commu- Problem Scoping nicative abilities may affect treatment and abil- Some projects have greater potential to im- ity to adhere to a treatment plan, AI could pact human lives than others. To identify ar- be applied to recognize special needs situa- eas where special attention may need to be tions and flag them for extra attention, and paid to fairness, it can be helpful to apply the build a case for a suitable course of treatment. Bioss AI Protocol (Bioss, 2019), which recom- With respect to rare diseases or genetic disor- mends asking the following 5 questions about ders, disparate data points can be aggregated the work being done by AI: such that solution determination and delivery is not contingent on an individual practitioner’s 1. Is the work Advisory, leaving space for hu- know-how. man judgement and decision making? Unfortunately, there are no standards or reg- 2. Has the AI been granted any Authority over ulations to assess the safety and efficacy of people? these systems. If the datasets don’t well- represent the broader population, AI might 3. Does the AI have Agency (the ability to act work less well where data are scarce or diffi- in a given environment)? cult to collect. This can negatively impact peo- 4. What skills and responsibilities are we at ple with rare medical conditions/disabilities. risk of Abdicating?

45 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

5. Are lines of Accountability clear, in what included, and what data and people are left are still organizations run by human beings? out. Finally, a word of warning. From a machine AI practitioners can also investigate whether learning perspective, an obvious solution to this is an area where people with disabilities handling a specialized sub-group not typical of have historically experienced discrimination, the general population might be to develop a such as employment, housing, education, and specialized model for that group. For example, healthcare. If so, can the project improve a specialized speech recognition model tuned on the past? Identify what specific outcomes to the characteristics of people with slurred there should be, so these can be checked as speech, or people who stutter. From an eth- the project progresses. Develop a plan for ical perspective, solutions that handle outliers tackling bias in source data to avoid perpet- and disability groups by routing them to an al- uating previous discriminatory treatment. This ternative service require careful thinking. Indi- could include boosting representation of peo- viduals may not wish to self-identify as having ple with disabilities, adjusting for bias against a disability, and there may be legal protections specific disability groups, or flagging gaps in against requiring self-declaration. Solutions data coverage so the limits of the resulting that attempt to infer disability status, or infer model are explicit. a quality that serves as a proxy for disability Approaches to developing ethical AI include status also present an ethical minefield. It may actively seeking the ongoing involvement of be acceptable to detect stuttered speech in or- a diverse set of stakeholders (Cutler et al., der to route a speech sample to a specialized 2019), and a diversity of data to work with. speech recognition model with higher accu- To extend this approach to people with dis- racy, but using the same detection system to abilities, it may be useful to define a set of evaluate a job applicant could be discrimina- ’outlier’ individuals, and include them in the tory, unfair and potentially illegal. Any system team, following an inclusive design method that explicitly detects ability-related attributes as discussed in the following section. These of an individual may need to make these in- are people whose data may look very differ- ferences visible to the user, optional, and able ent to the average. What defines an outlier to be challenged when they make wrong in- depends on the application. Many variables ferences. It is crucial to involve members of can be impacted by a disability, leading to a the affected disability group from the outset. potential for bias even where no explicit dis- This can prevent wasted time and effort on ability information is available. For example, in paths that lead to inappropriate and unfair out- speech recognition it could be a person with a comes, inflexible systems that cannot be ef- stutter or a person with slow, slurred speech. fectively deployed at scale, or that will be likely In a healthcare application involving height, to face legal challenges. this could mean including a person of short stature. Outliers may also include people who Data Sourcing belong with one group, but whose data looks more like that of another group. For example, When sourcing data to build a model, impor- a person who is slow to take a test may not be tant considerations are: struggling with the material, but with typing, or accessing the test itself through their assistive 1. Does the data include people with disabil- technology. By defining outlier individuals up ities, especially those disabilities identified front, the design process can consider at each as being most impacted by this solution? stage what their needs are, whether there are For example, data about the employees of a potential harms that need to be avoided, and company with poor diversity may not include how to achieve this. anyone who is deaf, or blind. If important Related to identifying outliers, developing a groups are missing, or if this information is measurement plan is also valuable at this not known, take steps to find or create such stage. If the plan includes expected out- data to supplement the original data source. comes for outliers and disability groups, this 2. Might the data embody bias against peo- can impact what data (including people) are ple with disabilities? Consider whether the

46 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

data might capture existing societal biases ror, but actually representing non-typical against people with disabilities. For exam- individuals, reducing the diversity in the ple, a dataset of housing applications with dataset. decisions might reflect a historical reluc- • Feature selection may include or exclude tance to choose someone with a disability. features that convey disability status. Be- When this situation is identified, raise the is- sides explicit disability information, other sue. features could be impacted by disability sta- 3. Is disability information explicitly repre- tus or the resulting societal disadvantage, sented in the data? If so, practitioners providing a proxy for disability status. For can use bias detection tests to check for example, a preference for large fonts could bias, and follow up with mitigation tech- serve as a proxy for visual impairment, or niques to adjust for bias before training a use of video captions could be correlated model (Bellamy et al., 2018). with deafness. Household income, educational achievement, and many other vari- Sometimes data are constructed by combin- ables can also be correlated with disability. ing several data sources. Depending on the • Feature engineering involves deriving new requirements of those sources, some groups features from the data, either through anal- of people may not have records in all sources. ysis or combination of existing features. Be attentive to whether disability groups might For example, calculating a person’s reading fall into this category and be dropped from the level or personality traits based on their writ- combined data set. For example, when coming, or calculating a ratio of days worked to bining photograph and fingerprint biometrics, days lived. In both of these examples, the consider what should happen for individuals derived feature will be impacted by certain who do not have fingers, and how they will be disabilities. represented and handled. In Europe, GDPR regulations (European Although accepted practice in many fields is to Union, 2016) give individuals the right to know exclude sensitive features so as not to build a what data about them is being kept, and how model that uses that feature, this is not neces- it is used, and to request that their data be sarily the best approach for algorithmic solu- deleted. As organizations move to limit the in- tions. The reality is that it can be extremely formation they store and the ways it can be difficult to avoid including disability status in used, AI systems may often not have explicit some way. When possible, including features information about disability that can be used that explicitly represent disability status allows to apply established fairness tests and cor- for testing and mitigation of disability-related rections. By being attentive to the potential bias. Consulting outlier individuals and stake- for bias in the data, documenting the diver- holder groups identified in the problem scop- sity of the data set, and raising issues early, ing stage is valuable to provide a better under- practitioners can avoid building solutions that standing of the ways that disability can be re- will perpetuate inequalities, and identify sys- flected in the data, and the tradeoffs involved tem requirements for accommodating groups in using or excluding certain features and data that are not represented in the data. values.

Data Pre-Processing Preserving Privacy The process of cleaning and transforming data People experiencing disabilities may have the into a form suitable for machine learning has most to gain from many smart systems, but been estimated to take 80-90% of the effort of are also particularly vulnerable to data abuse a typical data science project (Zhang, Zhang, and misuse. The current privacy protections & Yang, 2003), and the choices made at this do not work for individuals who are outliers or stage can have implications for the inclusive- different from the norm. The current response ness of the solution. to this data abuse and misuse, by privacy efforts globally, is to de-identify the data. The • Data cleaning steps may remove outliers, notion is that if we remove our identity from the presumed to be noise or measurement er- data, it can’t be traced back to us and it can’t

47 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 be used against us. The assumption is that est product or service offering. Layered on we will thereby retain our privacy while con- top of the standard are utilities that help contributing our data to making smarter decisions sumers explore, discover and refine their un- about the design. derstanding of their needs and preferences, for a given context and a given goal. The While people experiencing disabilities are par- personal data preference part of this standard ticularly vulnerable to data abuse and misuse, will let consumers declare what personal data they are often also the easiest to re-identify. If they are willing to release to whom, for what you are the only person in a neighborhood us- purpose, what length of time and under what ing a wheelchair, it will be easy to re-identify conditions. Services that wish to use the data you. If you are the only person that receives would declare what data is essential for pro- delivery of colostomy bags in your community, viding the service and what data is optional. it will be very easy to re-identify your purchas- This will enable a platform to support the ne- ing data. gotiation of more reasonable terms of service. The data requirements declarations by If de-identification is not a reliable approach to the service provider would be transparent and maintaining the privacy of individuals that are auditable. The standard will be augmented far from average, but data exclusion means with utilities that inform and guide consumers that highly impactful decisions will be made regarding the risks and implications of pref- without regard to their needs, what are poten- erence choices. Regulators in Canada and tial approaches to addressing this dilemma? Europe plan to point to this standard when it The primary focus has been on an ill-defined is completed. This will hopefully wrest back notion of privacy. When we unpack what this some semblance of self-determination of the means to most people, it is self-determination, use of our data. ownership of our own narrative, the right to know how our data is being used, and ethical Another approach to self-determination and treatment of our story. data that is being explored with the Plat- form Co-op Consortium (Platform Coopera- To begin to address this dilemma, an Interna- tivism Consortium, 2019) is the formation of tional Standards Organization personal data data co-ops. In a data co-op, the data produc- preference standard has been proposed as ers would both govern and share in the profit an instrument for regulators to restore self- (knowledge and funds) arising from their own determination regarding personal data. The data. This approach is especially helpful in proposal is developed as a response to the all- amassing data in previously ignored domains, or-nothing terms of service agreements which such as rare illnesses, niche consumer needs ask you to give away your private data rights in or specialized hobbies. In smart cities, for ex- exchange for the privilege of using a service. ample, there could be a multiplicity of data These terms of service agreements are usu- domains that could have associated data co- ally couched in legal fine print that most peo- ops. Examples include wayfinding and traffic ple could not decode even if they had the time information, utility usage, waste management, to read them. This means that it has become a recreation, consumer demands, to name just convention to simply click “I agree” without at- a few. This multiplicity of data co-ops would tending to the terms and the rights we have re- then collaborate to provide input into more linquished. The proposed standard will be part general urban planning decisions. of an existing standard called AccessForAll or ISO/IEC 24751 (ISO/IEC, 2008). The struc- Model Training and Testing ture of the parent standard enables matching of consumer needs and preferences with When developing a model, there are many resource or service functionality. It provides bias testing methods available to researchers. a common language for describing what you However, when applying these techniques to need or prefer in machine-readable terms and fairness for people with disabilities, some lim- a means for service providers or producers to itations become evident. Firstly, group-based describe the functions their products and ser- methods require large enough numbers of in- vices offer. This allows platforms to match di- dividuals in each group to allow for statistical verse unmet consumer needs with the clos- comparison of outcomes, and secondly, they

48 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 often rely on binary in-group/out-group com- valuable. These can be developed with the parisons, which can be difficult to apply to outlier individuals and key stakeholder groups disability. This section will expand on each identified at the outset. For models that in- of these points and suggest ways to address terpret humans (speech, language, gesture, these limitations in model testing. facial expression) where algorithmic fairness is important, the goal is to develop a method When examining fairness for people with dis- that works well for as many groups as possible abilities, there may be few examples in the (e.g. speech recognition for deaf speakers), training data. As defined by the United Na- and to document the limitations of the model. tions Convention on the Rights of People with For models that allocate people to groups (e.g. Disabilities (UN General Assembly, 2007), job candidate, loan applicant), allocative fair- disability “results from the interaction between ness is important. In selecting a measure for persons with impairments and attitudinal and allocative fairness, we argue that an individual environmental barriers that hinders their full fairness approach rather than a group fairness and effective participation in society.” As such, approach is preferable. While group fairness disability depends on context and comes in seeks to equalize a measure across groups, many forms, including physical barriers, sen- individual fairness aims for ‘similar’ individu- sory barriers, and communication barriers. als to receive similar outcomes. For example, One important consequence of experiencing in deciding whether to grant loan applications, a disability is that it can lead us to do things in even if the presence of a disability is statisti- a unique way, or to look or act differently. As a cally correlated with unemployment over the result, disabled people may be outliers in the whole dataset, it would still be unfair to treat data, or align with one of many very different an employed person with a disability the same sub-groups. as the unemployed group, simply because of their disability. Individual fairness aligns better Today’s methods for bias testing tend to split with the societal notion of fairness, and legal individuals into members of a protected group, mandates against discrimination. and ‘others’. However, disability status has many dimensions, varies in intensity and impact, and often changes over time. Further- Deployment in Real Applications more, people are often reluctant to reveal a disability. Binarized approaches that combine In this stage, the trained model is incorporated many different people into a broad ‘disabled’ into an application, typically with an interface category will fail to detect patterns of bias that for people to use, or an API to connect to. apply, differently and distinctively, against spe- Testing with diverse users, especially outliers, cific groups within the disability umbrella. For is essential. Understanding how different peo- example, an inaccessible online testing Web ple will perceive and use an AI-based system site will not disadvantage wheelchair users, is also important, for example, to see if certain but those who rely on assistive technologies people are more likely than others to ascribe or keyboard-only control methods to access trust to an AI-based system, or be more likely the Web may be unable to complete the tests. to feel insulted by an AI system’s terse replies More sensitive analysis methods not based on or lack of context. binary classifications are needed to address As a matter of course, quality assurance these challenges. Other protected attributes should include testing by people with disabil- like race and gender that have traditionally ities, covering as broad a set of disability been examined with binary fairness metrics groups as possible. This would include both are also far more complex and nuanced in testing the user interface of the system itself to reality (Keyes, 2018)(Hamidi, Scheuerman, ensure it is accessible, and the system’s per- & Branham, 2018b), and new approaches formance on diverse data inputs. The test pro- suitable for examining disability-related bias cess should deliberately include outlier indi- would support a more sophisticated examina- viduals to test the limits of the system and the tion of these attributes too. mechanisms for addressing system failure. To test for fairness, an audit based on iden- Because disability manifests in such diverse tified test cases and expected outcomes is ways, there will be situations where the ap-

49 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 plication is presented with an individual quite ties. For AL/ML practitioners, this may seem unlike those in the training data. For exam- like a daunting task, with questions ranging ple an automated telephone help system may from how to find diverse users, how to eth- have difficulty interpreting an individual with a ically and respectfully engage them, and by speech impairment, especially if they are not what methods one can reliably incorporate speaking in their native language. Develop- their feedback to improve systems. ers can avoid discrimination by providing an alternative to using AI, for example by support- The field of Human-Computer Interaction ing typed input in addition to speech. Users has long contemplated these questions, should have the ability to opt out of AI-based and has developed a number of design interpretation. philosophies, methodologies, and methods to guide the practice. Some of these per- Disability may also impact the accuracy of in- tain specifically to engaging people with dis- puts to an AI-based system. An example is abilities, including Universal Design (Story, a video-based personality analysis concluding Mueller, & Mace, 1998), Ability-Based De- that an autistic interviewee is untrustworthy sign (Wobbrock, Kane, Gajos, Harada, & because they did not make eye contact with Froehlich, 2011), Design for User Empower- the interviewer, and then feeding that into an ment (Ladner, 2015), and Design for Social applicant selection model. For people with dis- Accessibility (Shinohara, Wobbrock, & Pratt, abilities, it is essential to have the opportunity 2018). In this section, we endeavor to pro- to inspect and correct the data used to make vide a brief overview of just three poten- decisions about them. tial approaches that AL/ML developers might Equally important is the ability to query and choose as they seek to integrate users into challenge AI decisions, receiving some form their process. Our purpose, then, is not to pro- of explanation of the factors that most im- vide a comprehensive review of all or even a pacted the decision. If these factors were af- few methodologies, but rather to offer links into fected by a person’s disability, the decision the literature for those who want to learn more may be discriminatory. Any AI-based system about these approaches, or perhaps seek out that makes decisions affecting people should a collaborator with such expertise. include both an opportunity to dispute a deci- Below, we overview three distinct approaches sion, and provide a manual override for outlier to human-centered design: Inclusive Design, individuals where the model is unreliable. Participatory Design, and Value-Sensitive De- As many AI systems by their very nature sign. Each of these has developed from learn and thus modify their behavior over time, different intellectual traditions, and therefore ongoing mechanisms to monitor for fairness varies in the degree to which they explicitly in- to people with disabilities should be incorpo- clude people with disabilities in their theoret- rated. These can include ongoing auditing ical frameworks. People with disabilities are and reviews of performance as well as pe- often excluded from design processes, and riodic explicit testing to verify that changes designs rarely anticipate end-users’ needs in the system’s operation aimed at improv- to appropriate and adapt designs (Derboven, ing performance do not introduce disparities Geerts, & De Grooff, 2016). We therefore will in how decisions are made for specific sub- begin here by introducing some basic ratio- populations. This is crucial to ensure that as a nale for why it is important to include people system gets better for people overall it doesn’t with disabilities directly in software develop- unfairly get worse for some. ment efforts. Firstly, the opportunity to fully participate in so- Design Approaches ciety is every person’s right, and digital inclusion is fundamental to those opportunities to- In several of the stages of AI development de- day. All of us will very likely experience disabil- scribed above, and particularly in the problem ity at some stage in our lives, and our technol- scoping and testing/deployment phases, we ogy must be robust enough to accommodate have encouraged AI/ML engineers to seek the the diversity of the human experience, espe- ongoing involvement of people with disabili- cially if it is used in critical decision-making

50 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 areas. This cannot happen unless this diver- the margins. In particular, we believe the theo- sity is considered from the outset, hence the retical approaches of Inclusive Design (specif- disability rights movement’s mantra of “noth- ically as it evolved in Canada), Participatory ing about us, without us.” Design and Value Based Design are particularly valuable when designing for – and with – Including people with disabilities may bring people with disabilities. compelling new design ideas and ultimately expand the potential user base of the product. People with disabilities (including seniors) Inclusive Design have been described as the original life hack- The practice and theoretical framing of in- ers and personal innovators (Harley & Fitz- clusive design, that emerged and evolved patrick, 2012)(Storni, 2010), as they often in Canada and with global partners since have to find creative workarounds and inno- the emergence of the Web, takes advan- vate new technologies in a world that is not tage of the affordances or characteristics of built to their specifications. Second, peo- digital systems (Pullin, Treviranus, Patel, & ple with disabilities can be seen as present- Higginbotham, 2017)(Treviranus, 2016). In ing valuable diverse cases which should be contrast to related universal design theories, brought from the “edge” and into the “center” that emerged from architectural and indus- of our design thinking. In her foundational pa- trial design fields, the Canadian inclusive per on Feminism in Human Computer Interac- design practice aims to use the mutability tion, Bardzell advocated to study not only the and connectivity of networked digital systems conceptual “center” of a distribution of users, to achieve one-size-fits-one designs within but also the edge cases (Bardzell, 2010). She an integrated system, thereby increasing the argued that design often functions with a de- adaptability and longevity of the system as fault “user” in the designers’ minds, and that a whole (Lewis & Treviranus, 2013). Rather default user is often male, white, educated, than specifying design criteria, or accessibility and non-disabled. In Bardzell’s analysis, ac- checklists, the theory specifies a process or commodating the edge cases is a way both to mindset, called the “three dimensions of inclu- broaden the audience (or market) for a design, sive design.” and to strengthen the design against unanticipated changes in users, usage, or contexts 1. Recognize that everyone is unique, and of use. These ideas are echoed by other re- strive for a design that is able to match this searchers (Krischkowsky et al., 2015)(Muller uniqueness in an integrated system. Sup- et al., 2016)(Tscheligi et al., 2014). One port self-awareness of this uniqueness (use canonical example of how designs made with data to make people ‘smarter’ about them- and for people with disabilities can actually im- selves, not just machines smarter). prove the user experience of everyone (an ex- 2. Create an inclusive co-design process. The ample of “universal design”), is the curb cut. most valuable co-designers are individuals Curb cuts – the ramps that allow people in that can’t use or have difficulty using the cur- wheelchairs to transition from public sidewalks rent designs. Continuously ask whose per- to cross the street – also serve parents with spective is missing from the decision mak- prams, workers with heavy wheeled loads, ing “table” and how can they help make the and pedestrians on scooters. Both Downey “table” more inclusive. and Jacobs (Downey, 2008)(Jacobs, 1999) have advocated for electronic curb cuts; one 3. Recognize that all design operates within a example of such a feature is the zooming ca- complex adaptive system of systems. Be pability of the browser, which supports easier cognizant of the entangled impact and fric- reading for people with low vision or people tion points. Strive for designs that are bene- who are far away from the screen. ficial to this larger system of systems. In practice, the best way of centering This practice of inclusive design critiques and marginalized perspectives will require that we counters reliance on probability and popula- include people with disabilities in our core de- tion based statistics, pointing out the risks of sign practices. Fortunately, there is a rich his- basing critical decisions solely on the majority tory of work on design that centers users at or statistical average (Treviranus, 2014). With

51 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 respect to fairness in AI, people with disabil- published an overview of theory and meth- ities are most disadvantaged by population- ods for work with developmentally diverse chil- based AI decisions. The only common defin- dren. Katan et al. (2015) used interactive ma- ing characteristic of disability is difference chine learning in participatory workshops with from the norm. If you are not like the norm, people with disabilities (Katan, Grierson, & probability predictions based on population Fiebrink, 2015). data are wrong. Even if there is full rep- Some of these methods amount to merely resentation and all human bias is removed, asking people about their needs, e.g., (Holbø, statistically-based decisions and predictions Bøthun, & Dahl, 2013)(Krishnaswamy, 2017). will be biased against small minorities and out- However, other approaches involve bring- liers. Current automated decisions focus on ing people with disabilities (and sometimes the good of the majority, causing greater dis- their informal carers) into the design pro- parity between the majority and smaller mi- cess as active co-designers, e.g., (Gomez Tor- norities. Inclusive design practitioners are in- res, Parmar, Aggarwal, Mansur, & Guthrie, vestigating learning models that do not give 2019)(Hamidi et al., 2016)(Lee & Riek, advantage to being like the majority, causing 2018)(McGrenere et al., 2003)(Sitbon & learning models to attend to the full diversity Farhin, 2017)(Wilde & Marti, 2018)(Williams of requirements (Treviranus, 2019). This is hy- et al., 2018). In general, the methods that in- pothesized to also support better context shift- clude direct participation by people with dis- ing, adaptability and detection of weak sig- abilities in design activities are more power- nals. ful, and tend to include deeper understandings Given that data is from the past, optimiza- than are possible through the less engaged tion using past data will not achieve the cul- survey methods. ture shift inclusive design practitioners hope We note that this is an active research area, to achieve. Hence inclusive design prac- with discussion of needs that are not yet met, titioners are combining existing data, data e.g., (Holone & Herstad, 2013)(Oswal, 2014), from alternative scenarios, and modelled or and many opportunities to improve and inno- simulated data to assist in decision making. vate the participatory methods. For people Storytelling, bottom-up personalized data, or who are new to PD, we suggest beginning with small (n=1), thick (in context) data is also em- an orientation to the diversity of well-tested ployed to overcome the bias toward numerical methods, e.g., (Muller & Druin, 2012), followed data (Clark, 2018)(Pullin et al., 2017). by a “deeper dive” into methods that have been used with particular populations and/or Participatory Design with particular challenges. Participatory Design (PD) emphasizes the active role of people who will be affected by a Value-Sensitive Design technology, as co-designers of that technol- Participatory design originated primarily from ogy. There are many methods and rationales the workplace democracy movement in Scan- for these approaches, which can be found dinavia, e.g., (Bjerknes, Ehn, & Kyng, 1987), in (Muller & Druin, 2012), among other refer- and then developed in many directions. One ences. Some PD methods were originally pro- of the core assumptions of the workplace ap- posed as “equal opportunity” practices, e.g., plications was a division of labor among work- (Kuhn & Muller, 1993), because the methods ers and managers. In that context, PD meth- involved low-technology or “lo-fi” prototyping ods were seen as ways to reduce power dif- practices that did not require extensive com- ferences during the two-party design process, puter knowledge to contribute to the design. by facilitating the voice of the workers in re- However, the needs of people with disabilities lation to management. These assumptions were generally not considered during the early have tended to carry through into work with days of PD. people with disabilities, in which the two par- This omission has now been partially rectified. ties are reconceived as people with disabilities Borjesson¨ and colleagues (Borjesson,¨ Baren- and designers, or people with disabilities and dregt, Eriksson, & Torgersson, 2015) recently providers, with designers as mediators.

52 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Value Sensitive Design (VSD) offers a broader nitive disability as a potential threat. In AI perspective regarding the stakeholders in de- systems for healthcare, where speech char- sign (Friedman, Hendry, & Borning, 2017). acteristics can be used to diagnose cognitive For this paper, VSD crucially expands con- impairments, a person with a speech impedi- cepts of stakeholders into direct stakehold- ment can be misdiagnosed. ers (people who have contact with a design or a technology, including users, designers, To avoid such erroneous conclusions and po- providers) and indirect stakeholders (people tentially damaging outcomes, a number of who are affected by the design or technol- steps are proposed. AI systems should be pri- ogy, even if they do not have direct contact oritized for fairness review and ongoing mon- with it). For people with disabilities, there are itoring, based on their potential impact on the often multiple stakeholders in complex rela- user in their broader context of use. They tionships, e.g., (Zolyomi, Ross, Bhattacharya, should offer opportunities to redress errors, Milne, & Munson, 2018). and for users and those impacted to raise fairness concerns. People with disabilities should While there are disagreements about be included when sourcing data to build mod- the details of a value-centric approach, els. Such “outlier” data - the edge cases - e.g., (Borning & Muller, 2012)(Le Dan- will create a more inclusive and robust sys- tec, Poole, & Wyche, 2009)(Muller & Liao, tem. From the perspective of people with dis- 2017), there is consensus that values mat- abilities, there can be privacy concerns with ter, and that values can be formative for self-identification, but there can be risk of ex- designs. There may be values differences clusion from the data models if users opt not among people with disabilities, carers, and to participate or disclose. There are methods medical professionals, e.g, (Draper et al., provided to increase participation while pro- 2014)(Felzmann, Beyan, Ryan, & Beyan, tecting user privacy, such as the personal data 2016), and therefore an explicit and focused preferences standard. In deploying the AI ap- values inquiry may be helpful in “satisficing” plication, it is critical to test the UI and sys- these complex assumptions, views, and tem preferences with outlier individuals. Users needs (Cheon & Su, 2016). While VSD has should be able to pursue workarounds, and tended to have fewer well-defined methods, ultimately override the system where models Friedman et al. (2017) recently published a may be unreliable. survey of values-centric methods (Friedman et al., 2017). AI has been shown to help improve the lives of people with disabilities in a number of different environments whether it be navigating a city, Conclusion re-ordering prescriptions at a local pharmacy through a telephone or text service, or facil- We have outlined a number of situations in itating public safety. Almost everyone in the which AI solutions could be disadvantageous greater community is directly connected with for people with disabilities if researchers and someone with a disability whether it be a fam- practitioners fail to take necessary steps. In ily member, a colleague, a friend, or a neigh- many existing situations, non-AI solutions are bor. While AI technology has significantly im- already discriminatory, and introducing AI runs proved the lives of those in the disabled com- the risk of simply perpetuating and replicating munity, there are always ways in which we can these flaws. For example, people with dis- continue to advocate for fairness and equality abilities may already face discrimination in hir- and challenge the status quo. ing opportunities. With AI-driven hiring systems, models that recognize good candidates When creating AI that is designed to help the by matching to the existing workforce will per- community, we must take into consideration a petuate that status quo. In education, an AI disabled person user approach. The AI de- system that draws inferences based on a stu- signed should consider disabled people as a dent’s online interactions might misinterpret focal point. To borrow upon the concept driven speed for competency, if the student is using by Eric Ries in The Lean Startup (Ries, 2011), assistive technologies. In public safety, AI sys- we need to deploy minimum viable products tems might misinterpret a person with a cog- (MVPs) that are not perfected but rather im-

53 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 proved upon by the user involved. and public health: opportunity for social justice. Preventing chronic In a sense, what is needed is an incremen- disease, 8(2), A45. Retrieved from tal and algorithmic approach that continues http://www.ncbi.nlm.nih.gov/ to challenge the status quo and strives to pubmed/21324259http:// improve the standardization of fairness and www.pubmedcentral.nih.gov/ equality. This should be from a multi-industrial articlerender.fcgi?artid= approach with key players in different indus- PMC3073438 tries. Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, This paper suggests challenging every day S. C., Houde, S., Kannan, K., . . . Zhang, practices that may prove to inhibit people with Y. (2018, oct). AI Fairness 360: An disabilities, and is a starting point to bring Extensible Toolkit for Detecting, Under- awareness for the need for equality. It is im- standing, and Mitigating Unwanted Algo- portant to remember that promoting this goal rithmic Bias. Retrieved from http:// is a process. Success will require a series of arxiv.org/abs/1810.01943 incremental steps of further learning, thought Bhutani, A., & Wadhwani, P. (2019). Artiﬁcial provoking peer discussion, and changes at the Intelligence (AI) in Education Market local and municipal level. Only when these in- Size worth $6bn by 2024. Global Market cremental changes are met will we drive sus- Insights. Retrieved from https://www tainable outcomes for people with disabilities .gminsights.com/pressrelease/ using AI systems. artificial-intelligence-ai-in -education-market Acknowledgments Bioss. (2019). The Bioss AI Protocol. Re- trieved 2019-08-30, from http://www The authors thank all participants of the IBM- .bioss.com/ai/ MIT AI Week 2018 Workshop on AI Fairness Bird, S., Hutchinson, B., Kenthapadi, K., Kici- for People with Disabilities for their contribu- man, E., & Mitchell, M. (2019). Fairness- tions to the discussions leading to this article. Aware Machine Learning: Practical Chal- Special thanks to Peter Fay, Phill Jenkins, Au- lenges and Lessons Learned. In Com- drey Hasegawa and Morgan Foreman. panion proceedings of the 2019 world wide web conference on - www ’19 References (pp. 1297–1298). New York, New York, USA: ACM Press. Retrieved Ameri, M., Schur, L., Adya, M., Bent- from http://dl.acm.org/citation ley, F. S., McKay, P., & Kruse, D. .cfm?doid=3308560.3320086 doi: (2018, mar). The Disability Employ- 10.1145/3308560.3320086 ment Puzzle: A Field Experiment on Bjerknes, G., Ehn, P., & Kyng, M. (1987). Employer Hiring Behavior. ILR Re- Computers and Democracy - A Scandi- view, 71(2), 329–364. Retrieved from navian Challenge. Avebury. http://journals.sagepub.com/ Borjesson,¨ P., Barendregt, W., Eriksson, E., doi/10.1177/0019793917717474 & Torgersson, O. (2015). Design- doi: 10.1177/0019793917717474 ing Technology for and with Develop- Bardzell, S. (2010). Feminist HCI: Tak- mentally Diverse Children: A System- ing Stock and Outlining an Agenda atic Literature Review. In Proceedings for Design. In Proceedings of the of the 14th international conference on sigchi conference on human factors in interaction design and children (pp. 79– computing systems (pp. 1301–1310). 88). New York, NY, USA: ACM. Re- New York, NY, USA: ACM. Re- trieved from http://doi.acm.org/ trieved from http://doi.acm.org/ 10.1145/2771839.2771848 doi: 10 10.1145/1753326.1753521 doi: 10 .1145/2771839.2771848 .1145/1753326.1753521 Borning, A., & Muller, M. (2012). Next Steps Barnett, S., McKee, M., Smith, S. R., & for Value Sensitive Design. In Proceed- Pearson, T. A. (2011, mar). Deaf ings of the sigchi conference on human sign language users, health inequities, factors in computing systems (pp. 1125–

54 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

1134). New York, NY, USA: ACM. Re- mlr115{&}id=1081{&}div= trieved from http://doi.acm.org/ {&}collection= 10.1145/2207676.2208560 doi: 10 Cheon, E., & Su, N. M. (2016). Integrat- .1145/2207676.2208560 ing roboticist values into a Value Sen- Branham, S. M., Abdolrahmani, A., Easley, sitive Design framework for humanoid W., Scheuerman, M., Ronquillo, E., & robots. 11th ACM/IEEE International Hurst, A. (2017). ”Is Someone There? Conference on Human-Robot Interaction Do They Have a Gun”. In Proceedings (HRI). doi: 10.1109/HRI.2016.7451775 of the 19th international acm sigaccess Cinquin, P.-A., Guitton, P., & Sauzeon,´ conference on computers and accessibil- H. (2019, mar). Online e-learning ity - assets ’17 (pp. 260–269). New York, and cognitive disabilities: A system- New York, USA: ACM Press. Retrieved atic review. Computers & Educa- from http://dl.acm.org/citation tion, 130, 152–167. Retrieved from .cfm?doid=3132525.3132534 doi: https://www.sciencedirect 10.1145/3132525.3132534 .com/science/article/pii/ Brewer, R. N., & Kameswaran, V. (2018). Un- S0360131518303178 doi: derstanding the Power of Control in Au- 10.1016/J.COMPEDU.2018.12.004 tonomous Vehicles for People with Vi- Clark, C. B. (2018, jun). Many more sion Impairment. In Proceedings of the stories: Co-design and creative com- 20th international acm sigaccess confer- munities. OCAD University. Re- ence on computers and accessibility - trieved from http://openresearch assets ’18 (pp. 185–197). New York, .ocadu.ca/id/eprint/2079/ New York, USA: ACM Press. Retrieved Costello, K. (2019). Gartner Survey Shows from http://dl.acm.org/citation 37 Percent of Organizations Have .cfm?doid=3234695.3236347 doi: Implemented AI in Some Form. Stam- 10.1145/3234695.3236347 ford, Conn.: Gartner. Retrieved from Bruckner, M. A. (2018). The Promise https://www.gartner.com/en/ and Perils of Algorithmic Lenders’ newsroom/press-releases/2019 Use of Big Data. Chicago-Kent -01-21-gartner-survey-shows Law Review, 93. Retrieved from -37-percent-of-organizations https://heinonline.org/HOL/ -have Page?handle=hein.journals/ Cutler, A., Pribik, M., & Humphrey, L. chknt93{&}id=15{&}div= (2019). Everyday Ethics for Ar- {&}collection= tiﬁcial Intelligence (Tech. Rep.). Bureau of Labor Statistics, U. D. o. L. IBM. Retrieved from https:// (2019). PERSONS WITH A DIS- www.ibm.com/watson/assets/ ABILITY: LABOR FORCE CHAR- duo/pdf/everydayethics.pdf ACTERISTICS — 2018. Retrieved Dastin, J. (2018). Amazon scraps secret AI from https://www.bls.gov/ recruiting tool that showed bias against news.release/pdf/disabl.pdf women - Reuters. Retrieved 2019-08-30, Burgstahler, S. (2015, dec). Opening from https://www.reuters.com/ Doors or Slamming Them Shut? On- article/us-amazon-com-jobs line Learning Practices and Students -automation-insight/amazon with Disabilities. Social Inclusion, -scraps-secret-ai-recruiting 3(6), 69. Retrieved from http:// -tool-that-showed-bias www.cogitatiopress.com/ojs/ -against-women-idUSKCN1MK08G index.php/socialinclusion/ Derboven, J., Geerts, D., & De Grooff, article/view/420 doi: D. (2016). The Tactics of Everyday 10.17645/si.v3i6.420 Practice: A Semiotic Approach to Appro- Chander, A. (2017). The Racist Algo- priation. Interaction Design and Architec- rithm? Michigan Law Review, ture(s), 29(29), 99–120. Retrieved from 115(6), 1023. Retrieved from https://lirias.kuleuven.be/ https://heinonline.org/HOL/ 389602?limo=0 Page?handle=hein.journals/ Downey, G. (2008). Closed captioning: Sub-

55 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

titling, stenography and the digital con- man–Computer Interaction, 11(2), vergemce of text with television. Johns 63–125. Retrieved from http:// Hopkins University Press. dx.doi.org/10.1561/1100000015 Draper, H., Sorell, T., Bedaf, S., Syrdal, doi: 10.1561/1100000015 D. S., Gutierrez-Ruiz, C., Duclos, A., Fruchterman, J., & Mellea, J. (2018). & Amirabdollahian, F. (2014, oct). Expanding Employment Success Ethical Dimensions of Human-Robot for People with Disabilities — Interactions in the Care of Older Peo- Benetech — Software for Social Good ple: Insights from 21 Focus Groups (Tech. Rep.). benetech. Retrieved Convened in the UK, France and the from https://benetech.org/ Netherlands. In International conference about/resources/expanding on social robotics (LNCS 8755 ed., -employment-success-for pp. 135–145). Springer. Retrieved -people-with-disabilities-2/ from http://link.springer.com/ Gajos, K. Z., Wobbrock, J. O., & Weld, 10.1007/978-3-319-11973-1{ }14 D. S. (2007). Automatically Gen- doi: 10.1007/978-3-319-11973-1 14 erating User Interfaces Adapted to Dudley-Marling, C., & Burns, M. B. (2014). Users’ Motor and Vision Capabilities. Two Perspectives on Inclusion in the In Proceedings of the 20th annual United States, Global Education Review, acm symposium on user interface soft- 2014. Global Education Review, 1(1), ware and technology (pp. 231–240). 14–31. Retrieved from https://eric New York, NY, USA: ACM. Re- .ed.gov/?id=EJ1055208 trieved from http://doi.acm.org/ European Union. (2016). Regulation (EU) 10.1145/1294211.1294253 doi: 10 2016/679 of the European Parliament .1145/1294211.1294253 and of the Council of 27 April 2016 on Givens, A. R. (2019). Institute Announces the protection of natural persons with re- New Project on Algorithmic Fairness for gard to the processing of personal data People with Disabilities. Institute for and on the free movement of such data, Technology Law and Policy, Georgetown and repealing Directive 95/46/EC (Gen- Law. eral Da. Glickman, N. (2007). Do You Hear Voices? Faggella, D. (2019, apr). Machine Learning Problems in Assessment of Mental in Human Resources – Applications and Status in Deaf Persons With Severe Trends. Language Deprivation (Vol. 12). Ox- Faucett, H. A., Ringland, K. E., Cullen, ford University Press. Retrieved from A. L. L., & Hayes, G. R. (2017, https://www.jstor.org/stable/ oct). (In)Visibility in Disability and As- 42658866 doi: 10.2307/42658866 sistive Technology. ACM Trans. Ac- Gomez Torres, I., Parmar, G., Aggarwal, cess. Comput., 10(4), 14:1—-14:17. Re- S., Mansur, N., & Guthrie, A. (2019). trieved from http://doi.acm.org/ Affordable Smart Wheelchair. In Ex- 10.1145/3132040 doi: 10.1145/ tended abstracts of the 2019 chi con- 3132040 ference on human factors in comput- Felzmann, H., Beyan, T., Ryan, M., & ing systems (pp. SRC07:1—-SRC07:6). Beyan, O. (2016, jan). Imple- New York, NY, USA: ACM. Re- menting an Ethical Approach to Big trieved from http://doi.acm.org/ Data Analytics in Assistive Robotics 10.1145/3290607.3308463 doi: 10 for Elderly with Dementia. SIGCAS .1145/3290607.3308463 Comput. Soc., 45(3), 280–286. Re- Guo, A., Kamar, E., Vaughan, J. W., Wal- trieved from http://doi.acm.org/ lach, H., & Morris, M. R. (2019, 10.1145/2874239.2874279 doi: 10 jul). Toward Fairness in AI for People .1145/2874239.2874279 with Disabilities: A Research Roadmap. Friedman, B., Hendry, D. G., & Born- Retrieved from http://arxiv.org/ ing, A. (2017). A Survey of abs/1907.02227 Value Sensitive Design Methods. Haigh, K. Z., & Yanco, H. A. (2002). Automa- Foundations and Trends R in Hu- tion as Caregiver: A Survey of Issues

56 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

and Tchnologies (Tech. Rep.). AAAI. of Justice Statistics. Retrieved from Hamidi, F., Muller,¨ C., Baljko, M., Schorch, http://www.bjs.gov/index.cfm M., Lewkowicz, M., & Stangl, A. ?ty=pbdetail{&}iid=5986 (2016). Engaging with Users and Holbø, K., Bøthun, S., & Dahl, Y. (2013). Stakeholders: The Emotional and the Safe Walking Technology for People with Personal. In Proceedings of the Dementia: What Do They Want? In 19th international conference on sup- Proceedings of the 15th international porting group work (pp. 453–456). acm sigaccess conference on comput- New York, NY, USA: ACM. Re- ers and accessibility (pp. 21:1—-21:8). trieved from http://doi.acm.org/ New York, NY, USA: ACM. Re- 10.1145/2957276.2996292 doi: 10 trieved from http://doi.acm.org/ .1145/2957276.2996292 10.1145/2513383.2513434 doi: 10 Hamidi, F., Scheuerman, M. K., & Bran- .1145/2513383.2513434 ham, S. M. (2018a). Gender Recog- Holone, H., & Herstad, J. (2013). Three nition or Gender Reductionism? In Tensions in Participatory Design for Proceedings of the 2018 chi confer- Inclusion. In Proceedings of the ence on human factors in computing sys- sigchi conference on human factors in tems - chi ’18 (pp. 1–13). New York, computing systems (pp. 2903–2906). New York, USA: ACM Press. Retrieved New York, NY, USA: ACM. Re- from http://dl.acm.org/citation trieved from http://doi.acm.org/ .cfm?doid=3173574.3173582 doi: 10.1145/2470654.2481401 doi: 10 10.1145/3173574.3173582 .1145/2470654.2481401 Hamidi, F., Scheuerman, M. K., & Bran- Hurley, M., & Adebayo, J. (2016). Credit Scor- ham, S. M. (2018b). Gender Recogni- ing in the Era of Big Data. Yale Journal of tion or Gender Reductionism?: The So- Law and Technology, 18. Retrieved from cial Implications of Embedded Gender https://heinonline.org/HOL/ Recognition Systems. In Proceedings Page?handle=hein.journals/ of the 2018 chi conference on human yjolt18{&}id=148{&}div= factors in computing systems (pp. 8:1— {&}collection= -8:13). New York, NY, USA: ACM. Re- IEEE, G. I. o. E. o. A., & Systems, I. (2019). trieved from http://doi.acm.org/ Ethically Aligned Design: A Vision 10.1145/3173574.3173582 doi: 10 for Prioritizing Human Well-being with .1145/3173574.3173582 Autonomous and Intelligent Systems Hankerson, D., Marshall, A. R., Booker, J., (First edit ed.). IEEE. Retrieved from El Mimouni, H., Walker, I., & Rode, https://standards.ieee.org/ J. A. (2016). Does Technology Have content/ieee-standards/en/ Race? In Proceedings of the 2016 industry-connections/ec/ chi conference extended abstracts on {%}0Aautonomous-systems.html human factors in computing systems - Iezzoni, L. I. (2011, oct). Eliminating chi ea ’16 (pp. 473–486). New York, Health And Health Care Disparities New York, USA: ACM Press. Retrieved Among The Growing Population Of from http://dl.acm.org/citation People With Disabilities. Health Affairs, .cfm?doid=2851581.2892578 doi: 30(10), 1947–1954. Retrieved from 10.1145/2851581.2892578 http://www.healthaffairs.org/ Harley, D., & Fitzpatrick, G. (2012). Appropria- doi/10.1377/hlthaff.2011.0613 tion of social networking by older people: doi: 10.1377/hlthaff.2011.0613 two case studies. In Ecsw 2011 work- ISO/IEC. (2008). Information technology — shop on fostering social interactions in Individualized adaptability and accessi- the ageing society: Artefacts - method- bility in e-learning, education and train- ologies - research paradigms. Aarhus, ing — Part 1: Framework and reference Denmark. model. Author. Harrell, E. (2017). Crime Against Persons Jacobs, S. (1999). Section 255 of the with Disabilities, 2009-2015 - Sta- Telecommunications Act of 1996: tistical Tables (Tech. Rep.). Bureau Fueling the Creation of New Elec-

57 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

tronic Curbcuts. The Center for an research in intellectual disabilities : Accessible Society. Retrieved from JARID, 27(5), 431–46. Retrieved from http://www.accessiblesociety http://www.ncbi.nlm.nih.gov/ .org/topics/technology/ pubmed/23913632http:// eleccurbcut.htm www.pubmedcentral.nih.gov/ Janssen, M., & Kuk, G. (2016, jul). The chal- articlerender.fcgi?artid= lenges and limits of big data algorithms PMC4475843 doi: 10.1111/jar.12067 in technocratic governance. Govern- Krahn, G. L., Hammond, L., & Turner, A. ment Information Quarterly, 33(3), (2006, jan). A cascade of disparities: 371–377. Retrieved from https://www Health and health care access for peo- .sciencedirect.com/science/ ple with intellectual disabilities. Men- article/pii/S0740624X16301599 tal Retardation and Developmental Dis- doi: 10.1016/J.GIQ.2016.08.011 abilities Research Reviews, 12(1), 70– Kanellopoulos, Y. (2018, jul). A Model 82. Retrieved from http://doi.wiley for Evaluating Algorithmic Systems Ac- .com/10.1002/mrdd.20098 doi: 10 countability. Retrieved from http:// .1002/mrdd.20098 arxiv.org/abs/1807.06083 Krahn, G. L., Walker, D. K., & Correa-De- Katan, S., Grierson, M., & Fiebrink, R. Araujo, R. (2015, apr). Persons with (2015). Using Interactive Machine Learn- disabilities as an unrecognized health ing to Support Interface Development disparity population. American journal Through Workshops with Disabled Peo- of public health, 105 Suppl(Suppl ple. In Proceedings of the 33rd an- 2), S198–206. Retrieved from nual acm conference on human fac- http://www.ncbi.nlm.nih.gov/ tors in computing systems (pp. 251– pubmed/25689212http:// 254). New York, NY, USA: ACM. Re- www.pubmedcentral.nih.gov/ trieved from http://doi.acm.org/ articlerender.fcgi?artid= 10.1145/2702123.2702474 doi: 10 PMC4355692 doi: 10.2105/ .1145/2702123.2702474 AJPH.2014.302182 Keyes, O. (2018, nov). The Mis- Krischkowsky, A., Tscheligi, M., Neureiter, K., gendering Machines. Proceedings of Muller, M., Polli, A. M., & Verdezoto, N. the ACM on Human-Computer Inter- (2015). Experiences of Technology Ap- action, 2(CSCW), 1–22. Retrieved propriation: Unanticipated Users, Usage, from http://dl.acm.org/citation Circumstances, and Design. In Proceed- .cfm?doid=3290265.3274357 doi: ings of the 14th european conference on 10.1145/3274357 computer supported cooperative work. Koene, A., Dowthwaite, L., & Seth, S. Krishnaswamy, K. (2017). Participa- (2018). IEEE P7003TM standard for tory Design: Repositioning, Transfer- algorithmic bias considerations. In ring, and Personal Care Robots. In Proceedings of the international work- Proceedings of the companion of the shop on software fairness - fairware 2017 acm/ieee international conference ’18 (pp. 38–41). New York, New on human-robot interaction (pp. 351– York, USA: ACM Press. Retrieved 352). New York, NY, USA: ACM. Re- from http://dl.acm.org/citation trieved from http://doi.acm.org/ .cfm?doid=3194770.3194773 doi: 10.1145/3029798.3034815 doi: 10 10.1145/3194770.3194773 .1145/3029798.3034815 Koene, A., Smith, A. L., Egawa, T., Mandalh, Kroll, J. A., Barocas, S., Felten, E. W., S., & Hatada, Y. (2018). IEEE P70xx, Es- Reidenberg, J. R., Robinson, D. G., tablishing Standards for Ethical Technol- & Yu, H. (2016). Accountable Al- ogy. Proceedings of KDD, ExCeL Lon- gorithms. University of Pennsylvania don UK, August, 2018 (KDD’18). Law Review, 165. Retrieved from Krahn, G. L., & Fox, M. H. (2014, sep). https://heinonline.org/HOL/ Health disparities of adults with intel- Page?handle=hein.journals/ lectual disabilities: what do we know? pnlr165{&}id=648{&}div= What do we do? Journal of applied {&}collection=

58 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Kuhn, S., & Muller, M. (1993). Participatory Bhide, M., Saha, D., Varshney, K. R., & Design. Communications of the ACM, Puri, R. (2019, may). Bias Mitigation 36(6). Post-processing for Individual and Group K.W. v. Armstrong, No. 14-35296 (9th Cir. Fairness. In Icassp 2019 - 2019 ieee 2015) :: Justia. (2015). Retrieved from international conference on acoustics, https://law.justia.com/cases/ speech and signal processing (icassp) federal/appellate-courts/ca9/ (pp. 2847–2851). IEEE. Retrieved from 14-35296/14-35296-2015-06-05 https://ieeexplore.ieee.org/ .html document/8682620/ doi: Ladner, R. (2015). Design for user 10.1109/ICASSP.2019.8682620 empowerment. Interactions, McCarthy, O. J. (2019). AI & Global XXII(2). Retrieved from http:// Governance: Turning the Tide on interactions.acm.org/archive/ Crime with Predictive Policing - United view/march-april-2015/ Nations University Centre for Policy design-for-user-empowerment Research. United Nations University Le Dantec, C. A., Poole, E. S., & Wyche, Centre for Policy Research. Re- S. P. (2009). Values As Lived Experi- trieved from https://cpr.unu.edu/ ence: Evolving Value Sensitive Design in ai-global-governance-turning Support of Value Discovery. In Proceed- -the-tide-on-crime-with ings of the sigchi conference on human -predictive-policing.html factors in computing systems (pp. 1141– McGrenere, J., Davies, R., Findlater, L., 1150). New York, NY, USA: ACM. Re- Graf, P., Klawe, M., Moffatt, K., . . . trieved from http://doi.acm.org/ Yang, S. (2003). Insights from the 10.1145/1518701.1518875 doi: 10 Aphasia Project: Designing Technology .1145/1518701.1518875 for and with People Who Have Apha- Lee, H. R., & Riek, L. D. (2018, may). sia. In Proceedings of the 2003 con- Reframing Assistive Robots to Promote ference on universal usability (pp. 112– Successful Aging. ACM Trans. Hum.- 118). New York, NY, USA: ACM. Re- Robot Interact., 7(1), 11:1—-11:23. Re- trieved from http://doi.acm.org/ trieved from http://doi.acm.org/ 10.1145/957205.957225 doi: 10 10.1145/3203303 doi: 10.1145/ .1145/957205.957225 3203303 Morris, R. R., Kirschbaum, C. R., & Pi- Lepri, B., Oliver, N., Letouze,´ E., Pent- card, R. W. (2010). Broadening Ac- land, A., & Vinck, P. (2018, dec). cessibility Through Special Interests: A Fair, Transparent, and Accountable New Approach for Software Customiza- Algorithmic Decision-making Pro- tion. In Proceedings of the 12th inter- cesses. Philosophy & Technology, national acm sigaccess conference on 31(4), 611–627. Retrieved from computers and accessibility (pp. 171– http://link.springer.com/ 178). New York, NY, USA: ACM. Re- 10.1007/s13347-017-0279-x doi: trieved from http://doi.acm.org/ 10.1007/s13347-017-0279-x 10.1145/1878803.1878834 doi: 10 Lewis, C. (2019, jun). Implications of Devel- .1145/1878803.1878834 opments in Machine Learning for Peo- Muller, M., & Druin, A. (2012). Participa- ple with Cognitive Disabilities. SIGAC- tory design: The third space of HCI. In CESS Newsletter, Issue 124. Retrieved J. Jacko (Ed.), Human computer interac- from http://www.sigaccess.org/ tion handbook (3rd ed., pp. 1125–1154). newsletter/2019-06/lewis.html CRC Press. Lewis, C., & Treviranus, J. (2013, sep). Public Muller, M., & Liao, V. (2017). Using participa- Policy and the Global Public Inclusive In- tory design ﬁctions to explore ethics and frastructure Project. interactions, 20(5), values for robots and agents. 62–66. Retrieved from http://doi Muller, M., Neureiter, K., Krischkowsky, A., .acm.org/10.1145/2510123 doi: Tscheligi, M., Al Zubaidi-Polli, A. M., 10.1145/2510123 & Tscheligi, M. (2016). Collabora- Lohia, P. K., Natesan Ramamurthy, K., tive Appropriation: How Couples, Teams,

59 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Groups and Communities Adapt and from http://dl.acm.org/citation Adopt Technologies. In Proceedings .cfm?doid=3173574.3174033 doi: of the 19th acm conference on com- 10.1145/3173574.3174033 puter supported cooperative work and Pullin, G., Treviranus, J., Patel, R., & Hig- social computing companion - cscw ’16 ginbotham, J. (2017). Designing companion (pp. 473–480). New York, interaction, voice, and inclusion in AAC New York, USA: ACM Press. Retrieved research. Augmentative and Alterna- from http://dl.acm.org/citation tive Communication, 33(3), 139–148. .cfm?doid=2818052.2855508 doi: Retrieved from https://doi.org/ 10.1145/2818052.2855508 10.1080/07434618.2017.1342690 Obiakor, F. E., Harris, M., Mutua, K., Rota- doi: 10.1080/07434618.2017.1342690 tori, A., & Algozzine, B. (2012). Mak- Ries, E. (2011). The lean startup ing Inclusion Work in General Education : how today’s entrepreneurs use Classrooms. Education and Treatment of continuous innovation to create Children, 35(3), 477–490. doi: 10.1353/ radically successful businesses. etc.2012.0020 Retrieved from https://books Oswal, S. K. (2014, may). Participatory De- .google.com/books?hl=en{&}lr= sign: Barriers and Possibilities. Com- {&}id=r9x-OXdzpPcC{&}oi= mun. Des. Q. Rev, 2(3), 14–19. Re- fnd{&}pg=PA15{&}dq=+The+ trieved from http://doi.acm.org/ Lean+Startup{&}ots=0s 10.1145/2644448.2644452 doi: 10 -bDboHfV{&}sig=jVn-Rw46 .1145/2644448.2644452 -A2D4H{ }9HnK7gTi5MCg{#}v= Platform Cooperativism Consortium. (2019). onepage{&}q= What Is a Platform Co-op? Retrieved TheLeanStartup{&}f=false 2019-08-30, from https://platform Selbst, A. D., Boyd, D., Friedler, S. A., .coop/ Venkatasubramanian, S., & Vertesi, J. Policing, P. T. F. o. s. C. (2015). Final Report (2019). Fairness and Abstraction in of the President’s Task Force on 21st Sociotechnical Systems. In Proceed- Century Policing. Washington, DC: ings of the conference on fairness, Ofﬁce of Community Oriented Policing accountability, and transparency - fat* Services. Retrieved from https:// ’19 (pp. 59–68). New York, New ric-zai-inc.com/Publications/ York, USA: ACM Press. Retrieved cops-p311-pub.pdf from http://dl.acm.org/citation Pollard, R. (1994). Public mental health ser- .cfm?doid=3287560.3287598 doi: vice and diagnostic trends regarding indi- 10.1145/3287560.3287598 viduals who are deaf or hard of hearing. Shaffer, I. R., & Rogan, I. (2018). Ex- Rehabilitation Psychology, 39(3), 147– ploring the Performance of Facial Ex- 160. pression Recognition Technologies on Popovich, P. M., Scherbaum, C. A., Deaf Adults and Their Children. In Scherbaum, K. L., & Polinko, N. Proceedings of the 20th international (2003, mar). The Assessment of acm sigaccess conference on com- Attitudes Toward Individuals With puters and accessibility - assets ’18 Disabilities in the Workplace. The (pp. 474–476). New York, New Journal of Psychology, 137(2), York, USA: ACM Press. Retrieved 163–177. Retrieved from http:// from http://dl.acm.org/citation www.tandfonline.com/doi/abs/ .cfm?doid=3234695.3240986 doi: 10.1080/00223980309600606 doi: 10.1145/3234695.3240986 10.1080/00223980309600606 Shaheen, N. L., & Lazar, J. (2018). K–12 Pradhan, A., Mehta, K., & Findlater, L. (2018). Technology Accessibility: The Message ”Accessibility Came by Accident”. In from State Government. Journal of Spe- Proceedings of the 2018 chi confer- cial Education Technology, 33(2), 83– ence on human factors in computing sys- 97. tems - chi ’18 (pp. 1–13). New York, Shinohara, K., Wobbrock, J. O., & Pratt, W. New York, USA: ACM Press. Retrieved (2018). Incorporating Social Factors in

60 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Accessible Design. In Proceedings of the Treviranus, J. (2019). The Value of Be- 20th international acm sigaccess confer- ing Different. In Proceedings of the ence on computers and accessibility - 16th web for all 2019 personalization assets ’18 (pp. 149–160). New York, - personalizing the web (pp. 1:1—- New York, USA: ACM Press. Retrieved 1:7). New York, NY, USA: ACM. Re- from http://dl.acm.org/citation trieved from http://doi.acm.org/ .cfm?doid=3234695.3236346 doi: 10.1145/3315002.3332429 doi: 10 10.1145/3234695.3236346 .1145/3315002.3332429 Sitbon, L., & Farhin, S. (2017). Co- Trewin, S. (2018a, nov). AI Fairness for designing Interactive Applications People with Disabilities: Point of View. with Adults with Intellectual Disabil- Retrieved from http://arxiv.org/ ity: A Case Study. In Proceedings abs/1811.10670 of the 29th australian conference on Trewin, S. (2018b, nov). Will AI Methods Treat computer-human interaction (pp. 487– People with Disabilities Fairly? Medium, 491). New York, NY, USA: ACM. Re- MIT-IBM Watson AI Lab. Retrieved trieved from http://doi.acm.org/ from medium.com/@MITIBMLab/ 10.1145/3152771.3156163 doi: will-ai-methods-treat-people 10.1145/3152771.3156163 -with-disabilities-fairly Storni, C. (2010). Multiple Forms of -7626b38f9cb5 Appropriation in Self-Monitoring Tech- Trewin, S., Morris, M. R., Azenkot, S., nology: Reﬂections on the Role of Branham, S., Bleuel, N., Jenkins, P., Evaluation in Future Self-Care. Inter- . . . Lasecki, W. (2019). AI Fairness national Journal of Human–Computer for People with Disabilities. Re- Interaction, 26(5), 537–561. Re- trieved from https://assets19 trieved from https://doi.org/ .sigaccess.org/ai fairness 10.1080/10447311003720001 doi: workshop program.html 10.1080/10447311003720001 Tscheligi, M., Krischkowsky, A., Neureiter, K., Story, M. F., Mueller, J. L., & Mace, R. L. Inkpen, K., Muller, M., & Stevens, G. (1998). The Universal Design File: De- (2014). Potentials of the ”Unexpected”: signing for People of All Ages and Abil- Technology Appropriation Practices and ities. Revised Edition. Center for Uni- Communication Needs. In Proceed- versal Design, NC State University, Box ings of the 18th international conference 8613, Raleigh, NC 27695-8613 ($24). on supporting group work (pp. 313– Tel: 800-647-6777 (Toll Free)’ Web 316). New York, NY, USA: ACM. Re- site: http://www.design.ncsu.edu. Re- trieved from http://doi.acm.org/ trieved from https://eric.ed.gov/ 10.1145/2660398.2660427 doi: 10 ?id=ED460554 .1145/2660398.2660427 Straumsheim, C. (2017, mar). Berkeley UN General Assembly. (2007). Con- Will Delete Online Content. Inside vention on the Rights of Persons Higher Ed. Retrieved from https:// with Disabilities : resolution / www.insidehighered.com/news/ adopted by the General Assembly, 2017/03/06/u-california 24 January 2007, A/RES/61/106. -berkeley-delete-publicly Retrieved from https://www -available-educational .un.org/development/desa/ -content disabilities/convention-on Treviranus, J. (2014). The Value of the Statis- -the-rights-of-persons-with tically Insigniﬁcant. Educause Review. -disabilities/convention-on Treviranus, J. (2016). Life-long Learning on -the-rights-of-persons-with the Inclusive Web. In Proceedings of the -disabilities-2.html 13th web for all conference (pp. 1:1—- US Department of Justice Civil Rights Di- 1:8). New York, NY, USA: ACM. Re- vision. (2006). COMMONLY ASKED trieved from http://doi.acm.org/ QUESTIONS ABOUT THE AMERICANS 10.1145/2899475.2899476 doi: 10 WITH DISABILITIES ACT AND LAW .1145/2899475.2899476 ENFORCEMENT. Retrieved 2019-

61 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

08-30, from https://www.ada.gov/ Values, Identity, and Social Translu- q{&}a{ }law.htm cence: Neurodiverse Student Teams in von Schrader, S., Malzer, V., &Bruy ere,` Higher Education. In Proceedings of S. (2014, dec). Perspectives on the 2018 chi conference on human fac- Disability Disclosure: The Importance tors in computing systems (pp. 499:1—- of Employer Practices and Workplace 499:13). New York, NY, USA: ACM. Re- Climate. Employee Responsibilities trieved from http://doi.acm.org/ and Rights Journal, 26(4), 237–255. 10.1145/3173574.3174073 doi: 10 Retrieved from https://doi.org/10 .1145/3173574.3174073 .1007/s10672-013-9227-9 doi: 10 .1007/s10672-013-9227-9 Wastfelt, M., Fadeel, B., & Henter, J.-I. (2006, jul). A journey of hope: lessons learned from studies on rare diseases and orphan drugs. Journal of Internal Shari Trewin is a tech- Medicine, 260(1), 1–10. Retrieved from nical leader in IBM’s http://doi.wiley.com/10.1111/ Accessibility Leadership j.1365-2796.2006.01666.x doi: Team, an ACM Distin- 10.1111/j.1365-2796.2006.01666.x guished Scientist, and Wilde, D., & Marti, P. (2018). Explor- Chair of ACM SIGAC- ing Aesthetic Enhancement of Wear- CESS. She has a Ph.D. able Technologies for Deaf Women. In in Computer Science and Proceedings of the 2018 designing in- Artiﬁcial Intelligence from teractive systems conference (pp. 201– the University of Edinburgh, with 17 patents 213). New York, NY, USA: ACM. Re- and over 60 publications in technologies trieved from http://doi.acm.org/ to remove barriers for people experiencing 10.1145/3196709.3196777 doi: 10 disabilities. .1145/3196709.3196777 Williams, B. A., Brooks, C. F., & Shmar- Sara Basson works at gad, Y. (2018). How Algorithms Google as “Accessibility Discriminate Based on Data they Evangelist,” with the goal Lack: Challenges, Solutions, and of making the experi- Policy Implications. Journal of Infor- ence of Googlers more mation Policy, 8, 78. Retrieved from accessible and usable. https://www.jstor.org/stable/ She previously worked at 10.5325/jinfopoli.8.2018.0078 IBM Research on Speech doi: 10.5325/jinfopoli.8.2018.0078 Technology, Accessibility, Wobbrock, J. O., Kane, S. K., Gajos, K. Z., and Education Transfor- Harada, S., & Froehlich, J. (2011, mation. Sara holds a apr). Ability-Based Design. ACM Ph.D. in Speech, Hearing, Transactions on Accessible Computing, and Language Sciences 3(3), 1–27. Retrieved from http:// from The Graduate Center of CUNY. portal.acm.org/citation.cfm ?doid=1952383.1952384 doi: Michael Muller is an 10.1145/1952383.1952384 ACM Distinguished Sci- Zhang, S., Zhang, C., & Yang, Q. (2003, entist, and a member of may). Data preparation for data min- the SIGCHI Academy with ing. Applied Artiﬁcial Intelligence, expertise in participatory 17(5-6), 375–381. Retrieved from design and participatory http://www.tandfonline.com/ analysis. In the AI Inter- doi/abs/10.1080/713827180 doi: actions group of IBM Re- 10.1080/713827180 search AI, he focuses on Zolyomi, A., Ross, A. S., Bhattacharya, A., the human aspects of data science; ethics and Milne, L., & Munson, S. A. (2018). values in applications of AI to human issues.

62 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Stacy Branham is an As- Natalia Lyckowski is sistant Professor of In- an Application Devel- formatics at the Univer- oper with IBM Global sity of California, Irvine. Financing. She is also Her research explores the a passionate volunteer role of technology in so- educator on inclusion cial integration for people and diversity awareness. with disabilities, with ac- Most proudly, she is the tive lines of inquiry in the mother of a young autistic domains of personal mobility and parenting man who has taught her with disability. how to think out of the box and remain ever ﬂexible. Jutta Treviranus is the Erich Manser is an director and founder IBM Accessibility evan- of the Inclusive Design gelist exploring new Research Centre (IDRC), approaches to acces- established in 1993. The sible technology, new IDRC leads many interna- applications of emerging tional research networks technologies, and con- to proactively model more tributing to accessibility inclusive socio-technical standards. Beyond the practices, including in workplace, Erich is an active mentor, dis- the area of data science ability rights advocate, and fundraiser. He is (e.g., Project We Count). also a nationally recognized blind athlete in Dr. Treviranus has co-designed numerous marathon and triathlon racing. policies related to digital inclusion.

Dan Gruen is a Cogni- tive Scientist in IBM Re- search’s Center for Com- putational Health focusing on AI explainability, clini- cal reasoning, and complex medical cases. Dan has a PhD in Cogni- tive Science from UCSD and holds over 50 US and international patents in User Experience, Visualization, Social Soft- ware, and related areas.

Daniel Hebert has his computer science degree from Rensselaer Poly- technic Institute and has been with IBM since 2006 as a QA, engineer, and developer. He enjoys working on smart home and accessibility projects, and is a community advocate for persons with disabilities and involved with the local independent living movement.

63 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

The Intersection of Ethics and AI Annie Zhou (High Technology High School; [email protected]) DOI: 10.1145/3362077.3362087

Introduction ther the technology goes from the algorithms, the more unclear it is on the companies’ lia- Artificial intelligence is a rapidly advancing bility (Baker, 2004). For now, property dam- field with the potential to revolutionize health ages and harm will have to be examined on care, transportation, and national security. Al- a case to case basis. The company will be though the technology has been ubiquitous in held liable in instances their product fails due every day society for a while, the advent of to an oversight in production, just as if prod- self-driving cars and smart home devices have ucts were built with a poor design and hurt the propelled a discussion of the associated ethi- user, the company is responsible. cal risks and responsibilities. Since the usage of AI can have significant impacts on people, it In March 2018, an experimental Uber self- is essential to establish a set of ethical values driving car struck a pedestrian, after a coded to follow when developing and deploying AI. system for emergency stops was disabled (Ford, 2018). In the case this occurs when Responsibility self-driving cars are implemented nationwide, possible lawsuits can follow the model of prod- The companies researching, developing, and uct liability precedents. Liability can be di- deploying artificial intelligence must be held vided into strict liability, negligence, manufac- accountable for their products. With the future turing defects, and design defects (Villasenor, of artificial intelligence as vastly uncharted ter- 2018). In the case that a defective product is ritory, companies must be mindful of their cor- released, then the company is liable for any re- porate social responsibility: namely, the im- sulting damage. If the company failed to test pact of their products on society in various ar- the product in a multitude of possible circum- eas (Polli, 2017). AI research may be rapid, stances, such as testing braking systems only but companies must hold back before quickly on dry services, and the product subsequently releasing new products to ensure they are fol- fails on wet roads, the company is liable for lowing ethical behavior. Several tech giants crashes caused by wet roads. If a manufac- have acknowledged the significance of their turing issue results in damage, then it is the role in the future of AI; Microsoft, IBM, and fault of the intermediary manufacturer. If the Google, among others, are building standard product is fundamentally designed in a way ethics processes. Smaller companies have to that incites harm, the company is responsible also begin assembling a code of AI ethics that for addressing this issue (Lea, 2018). sets standards and precautions to deal with The government also ought to play a role in various issues in a consistent manner (Hao, regulating an ethical implementation of AI sys- 2018). tems. As AI makes consequential decisions In order to analyze the ethics of the technol- about people, the government will need to de- ogy, corporations need to shift infrastructure to velop and provide public policy to ensure AI is seamlessly include ethics in AI decisions. For for the public good. Several measures have one, they need to hire ethicists who can advise been taken, such as creating a subcommit- the programmers. Corporations also need to tee on the National Science and Technology create ethics training programs to teach pro- Council for Machine Learning and Artificial In- grammers about the ethical concerns of AI, so telligence. The government ought to encour- the ones work directly with the technology un- age datasets that are representative and fair, derstand how they’re affecting society. by releasing government data sets to aid AI research and creating open data standards. Corporations will also need to set rules in the Government programs that will be affected by case their AI technology inflicts harm. The far- AI, such as the Department of Transportation, Copyright c 2019 by the author(s). should work with researchers. A government

64 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 committee can also regularly monitor AI re- fairly incarcerated. search progress and meet with industry. Bias was also evident in Amazon’s recent system for automating the recruitment process. Transparency & Database Bias Since the tech industry was male-dominated, most of the resumes fed into the machine Algorithms will need to be developed in a learning model were for men. The resulting transparent manner. Consumers will need to system favored men over women. If this re- establish their trust in AI systems, beyond sim- cruiting system were deployed, it would have ply the accuracy of their desired function. The further alienated women from the tech sector. code itself may seem long and arduous to understand for the general public. The IBM team AI Interaction with Humans: Customer recently proposed four fundamental pillars for Engagement trusted AI: robustness, explainability, lineage, and fairness. (10153623502710135, 2018). As new AI-technologies enter the market, the AI models need to strike a balance between prevalence of AI only grows. Although a com- explainability on how they arrive to specific demon fear is that AI will eventually replace the cisions, and accuracy. Meaningful explana- human race, AI is expected to be complemen- tions about AI models reduce uncertainty. AI tary to humans. One particular field where systems must be safe in the sense that it fol- AI is being deployed in is customer engage- lows laws, societal norms, and regulations as- ment. 38% of enterprises are planning to ex- sociated with safe behavior. They must also periment with AI powered customer service, be secure from adversarial attacks. Malicious according to a Gartner survey, AI has the po- actors can alter databases to influence the be- tential to give customers the right information havior of AI models, which can lead to inaccu- they need, while freeing up time for customer rate or biased results. Companies must ac- service representatives to handle more com- count for possible attacks and test their mod- plicated issues. els against adversarial attacks. Understand- ing the lineage of an AI system, namely its his- In 1950, Alan Turing introduced the Turing tory and past configurations, is important for test in which judges blind conversations with trusting in AI. One of the most major concerns a chatbot and human participants. Based is the fairness of AI systems. on the responses, the judges had to identify which conversation was with a chatbot and As AI systems rely solely on learning from a which was with a human. The test was con- database, if the database is incomplete or bi- stantly analyzing what it meant to be human ased, the system can unintentionally perpetu- versus a machine interpretation of humanity. ate bias to a widespread scale. Unrepresenta- The results of the annual test revealed that hu- tive data can latch onto subtle racist and sex- mans had several distinctions in their speech, ist patterns. For example, risk assessments such as timing, fluidity, and vocal tics such provided by computer programs were used to as “uh” and “mhm”. Google recently unveiled predict the likelihood of a criminal committing Google Duplex, an AI that can make a phone a future crime. A ProPublica investigation dis- call that sounds eerily similar to one a hu- covered that not only was the risk score un- man would make, with natural pauses, varied reliable for forecasting repeated crimes, the questions, and other idiosyncrasies indicative formula was twice as likely to flag black de- to human speech (Vincent, 2018). This devel- fendants as future criminals, and white de- opment raised concern over whether Google fendants were mislabeled as low risk (ACLU, Duplex had an obligation to tell the people it n.d.). This disparity stemmed from the exist- called that they were talking to a machine. ing bias in historical records of arrests, from Google Duplex also raised the ethical question which the data was used to teach this algo- of making a robot voice that could be easily rithm. It is highly dangerous for criminal jus- mistaken for a human’s. Though Google has tice algorithms to be based on bias, as that vi- promised regulations to ensure that their us- olates the justice system’s fundamental goal. age of the technology is solely for making acts Biased algorithms could perpetuate a vicious like restaurant reservations easier, this tech- cycle where more African Americans are un- nology is so dangerous. Robots that can pose

65 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 at humans could pave the way for a new wave split over the crucial decision whether to save of scam calls and hoaxes. someone younger or older or a pedestrian versus a passenger. Germany’s Ethics Commis- Regulations must be made for automated AI sion on Automated Driving published propo- conversations. Before starting a call, the AI sitions that other countries can follow as a should warn the person it’s calling that an AI guide. These included affirmations that hu- is speaking. AI conversations should also be man life takes top priority and that any dis- limited to following simple directives such as tinction in personal features in possible victims ordering gifts or making a reservation. AI of an unavoidable accident situations cannot should not be pushed to be able to hold full be considered in calculating what decision to conversations indistinguishable to ones hu- make. Generally, it would be optimal for cars mans hold, as such technology could be ex- to be programmed with automatic risk calcu- ploited to trick people into thinking they’re lations to weigh the the morality of certain de- speaking a human. Although some people cisions. For example, if a person were cross- will automatically hang up once the AI an- ing a street illegally, they are doing so with the nounces its identity, companies can works to- knowledge that this is illegal, so his life would wards calming the general public’s fear over be prioritized underneath the passenger’s life. the technology by being completely transparent about what their algorithm intends to do. These moral quandaries are extreme, and are highly unlikely to occur in practice. There are more common ethical problems that self- AI Interaction with Humans: Self driving cars will inevitably run into. Driving well Driving Cars not only involves driving safely, but also taking smart risks. For example, a driver wanting Vehicle automation is close to becoming a to merge onto a highway may either choose a reality on the roads, with companies like safe speed, or a faster one to reduce overall Mercedes-Benz, Tesla, Toyota, and Uber travel time. When following the flow of traf- working on driverless car technology. When fic, cars often go close or over the speed limit; entrusting the decisions on the road to a com- slow, cautious cars are not helpful for traffic puter program, several ethical concerns can flow. Autonomous vehicles will need to be arise. When a machine makes decisions, it equipped to weigh decisions based on the risk has to follow a preprogrammed standard of and the reward, striking a balance between ethics. The most popular dilemma is what de- safety and efficiency. Furthermore, the behav- cision to make if a crash were imminent. If a ior of cars will influence future traffic patterns. human driver slams on a brake to avoid hitting In anticipation of a future where all transporta- a pedestrian crossing a road illegally, they may tion is automated, subtle driving behavior pat- be putting the passengers in danger. How- terns programmed into the algorithms will be- ever, as it is ethically gray whether it is better to come the norm. Thus, every small decision save the pedestrian or the passengers in side has to be considered carefully regarding its the car, it is controversial what decision self- impact in the traffic scheme (Bogle, 2018). driving cars should make. As different people have different values, being forced to choose Overall, if self-driving cars expected to be- one universal moral code for self-driving cars come the norm, they must be built in a trans- to follow is difficult. In a split second deci- parent manner. In order for consumers to trust sion, a human would typically lean towards this innovative technology, they must be able preserving his own protection. If the decision to know everything about how the algorithms can be pre-programmed and pre-ruminated, are made. Collaborations among social scien- results may vary. A paper published by MIT tists and ethicists with programmers must be details an online quiz called the Moral Ma- fostered, so that these ethical issues are ad- chine, where it asked users a series of ethical dressed. choices regarding hypothetical car crashes, in a manner similar to the classic trolley problem. Consequences on Economy Certain trends were apparent, such as spar- ing humans over animals and generally favor- Autonomous vehicles have the potential to ing more lives than fewer, but people were be the future of all transportation. Switching

66 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 to self-driving cars would significantly impact tificial intelligence, it is important to coordinate the economy. Out of the most common pro- these different opinions. It will take global co- fessions per state, truck driving is the most operation to set ethical guidelines for AI tech- common. In 2014 around 1.6 million Amer- nologies (Ito, 2018). Countries should engage icans were employed as truck drivers (Ser- in discourse on relevant artificial intelligence vice, 2017). Around a million people are em- issues, introducing the latest developments ployed as taxi drivers, Uber drivers, school from their countries and promoting collabora- bus drivers, and transit drivers. If the need tion of scientists from different countries. Al- for drivers is eliminated, then all these peo- though it is impractical to try and change the ple are out of a job. It extends beyond the perspectives of people from different cultures, drivers. Others jobs can be affected, such it is important to understand why other peo- as gas station attendants, rental car agen- ple view artificial intelligence a certain way, in cies, street meter maids, and repair shops. order to create appropriate legislature. Up to 4 million people are at risk if full automation is achieved. Not all these people will be unemployed; with the natural expan- Teaching AI Ethics sion of the economy, they will inevitably find In order to ensure that ethics are consid- jobs in the new industries created from au- ered when developing artificial intelligence, tomation. However, companies unveiling this the general public needs to mindful of the im- technology ought to take several precautions. portance of ethics in this technology. Coop- There are several possible ways to smoothly eration between computer scientists and ethi- transition the economy in preparation of au- cists is essential. These two groups of people tonomous vehicles. For one, they can develop can collaborate to come up with curriculum to a new profession in the form of remote vehi- teach people the intersection of ethics and AI. cle operators. Companies will likely establish This curriculum can be taught through courses command centers to monitor the autonomous at universities, in order to teach the upcoming vehicles. Another idea is a “passenger econ- generation of leaders in this field. omy”, with goods and services being provided to people during their ride in an autonomous Programs can also be developed to start taxi. The government can also collect money teaching the intersection of artificial intelli- from the self-driving car industry, and use it to gence and ethics at a young age. Sum- help re-position the people affected into new mer programs like AI4ALL reach out to high jobs. This money can help tide them over un- schoolers, introducing them to artificial intelli- til they find a job, or it could be used to cre- gence in the context of the various ethical is- ate jobs for people in the form of building civil sues involved. These programs can be further works projects. implemented through online courses, reach- ing a greater audience. As artificial intelligence is especially pertinent to the govern- Cultural Differences Regarding AI ment, the government can fund programs to Artificial intelligence is a technology being ex- teach the ethics of AI. Social experiments plored globally. People from different cultures such as the Moral Machine are also effective have different opinions on how to deal with in alerting people to the possible ethical dilem- it. This depends on the different opinions of mas that come along with artificial intelligence “humanity”. For examples, follower of Shinto, (Maxmen, 2018). the official national religion of Japan, believe that humans are just the same as rocks and Overview animals, as everything is part of Nature. On the other hand, Western philosophy prides in Artificial intelligence has proven to be a rel- the individuality and uniqueness of humans, evant technology that will exponential grow and draws a distinct line between what is hu- in future years. Although the technology man and what is not. The Western fear of has great potential, it raises many ethical is- robots largely stems from the fear of pushback sues regarding its deployment. It is important from what humans have dehumanized. As that companies developing the technology are each culture has different perspectives of ar- mindful of their ethical responsibility in devel-

67 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 oping the technology, and that the government Bogle, A. (2018, March 21). 5 big eth- creates appropriate regulation. Algorithms will ical questions about driverless cars we need to be transparent and based upon un- still need answered. Retrieved from biased, complete data. Artificial intelligence https://www.abc.net.au/news/science/2018- must be for the common good, not perpetuat- 03-21/self-driving-autonomous-cars-five- ing current biases in society. Furthermore, the ethical-questions/9567986 general public will need to trust artificial intelli- Bundy, A. (2016). Preparing for the future of gence’s reliability, security, and safety. As arti- Artificial Intelligence. Ai & Society,32(2), 285- ficial intelligence increasingly matches human 287. doi:10.1007/s00146-016-0685-0 behaviors rules must be put in place to avoid people misusing its human mimicking ability. Burton, E., Goldsmith, J., & Mattei, N. (2018, Ethical concerns are especially apparent re- August 01). How to Teach Computer Ethics garding self-driving cars, which will need to through Science Fiction. Retrieved from follow an established set of rules. The con- https://cacm.acm.org/magazines/2018/8/229765- sequences of AI on the economy will need to how-to-teach-computer-ethics-through- be considered. Global cooperation is neces- science-fiction/fulltext sary to successfully implement AI. Overall, the Ethics Commission on Automated Driving intersection of AI and ethics is crucial to its presents report. (n.d.). Retrieved from success, and companies and the government https://www.bmvi.de/SharedDocs/EN/Press should aim to educate the public about the im- Release/2017/084-ethic-commission-report- plications. automated-driving.html Ford, M. (2018, March 20). A References Self-Driving Uber Killed a Woman. 10153623502710135. (2018, October Whose Fault Is It? Retrieved from 08). Towards AI Transparency: Four Pil- https://newrepublic.com/article/147553/self- lars Required to Build Trust in Artificial driving-uber-killed-woman-whose-fault-it Intelligence Systems. Retrieved from Hao, K. (2018, October 21). Estab- https://towardsdatascience.com/towards- lishing an AI code of ethics will be ai-transparency-four-pillars-required-to- harder than people think. Retrieved from build-trust-in-artificial-intelligence-systems- https://www.technologyreview.com/s/612318/ d1c45a1bdd59 establishing-an-ai-code-of-ethics-will-be- ACLU. (n.d.). With AI and Criminal Jus- harder-than-people-think/ tice, the Devil Is in the Data. Retrieved Ito, J. (2018, July 31). Why Westerners Fear from https://www.aclu.org/issues/privacy- Robots and the Japanese Do Not. Retrieved technology/surveillance-technologies/ai-and- from https://www.wired.com/story/ideas-joi- criminal-justice-devil-data ito-robot-overlords/ Ark, T. V. (2018, September 13). Lea, G. (2018, December 06). Who’s Let’s Talk About AI Ethics; We’re to blame when artificial intelligence On A Deadline. Retrieved from systems go wrong? Retrieved from https://www.forbes.com/sites/tomvanderark/ https://theconversation.com/whos-to-blame- 2018/09/13/ethics-on-a-deadline when-artificial-intelligence-systems-go- wrong-45771 Artificial Intelligence raises ethical, policy challenges – UN expert — Lee, J. (2015, June 19). Self Driv- UN News. (n.d.). Retrieved from ing Cars Endanger Millions of American https://news.un.org/en/story/2018/11/1025951 Jobs (And That’s Okay). Retrieved from https://www.makeuseof.com/tag/self-driving- Baker, M. (2004, February 06). Def- cars-endanger-millions-american-jobs-thats- initions of corporate social respon- okay/ sibility - What is CSR? Retrieved from http://mallenbaker.net/article/clear- Maxmen, A. (2018, October 24). Self- reflection/definitions-of-corporate-social- driving car dilemmas reveal that moral responsibility-what-is-csr choices are not universal. Retrieved

68 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 from https://www.nature.com/articles/d41586- line – Predictions from the Top 11 Global 018-07135-0 Automakers — Emerj - Artificial Intelligence Research and Insight. Retrieved from Pakzad, R. (2018, February 01). https://emerj.com/ai-adoption-timelines/self- Artificial Intelligence and Corpo- driving-car-timeline-themselves-top-11- rate Social Responsibility. Retrieved automakers/ from https://medium.com/humane- ai/artificial-intelligence-and-business-social- Who should be blamed when responsibility-69d6299b4d9d the machine makes the mistake? (2017, June 07). Retrieved from Polli, F. (2017, November 09). AI And https://www.healthcareitnews.com/cloud- Corporate Responsibility: Not Just decision-center/who-should-be-blamed- For The Tech Giants. Retrieved from when-machine-makes-mistake https://www.forbes.com/sites/fridapolli/ 2017/11/08/ai-and-corporate-responsibility- not-just-for-the-tech-giants/790804f95d4b Annie Zhou Annie is a Responsibility in Technological Civiliza- senior at a rigorous pre- tion:In Search of the Responsible Sub- engineering vocational ject. (2013, October 25). Retrieved high school interested from http://www.iwm.at/publications/5- in computer science, junior-visiting-fellows-conferences/vol- design, and finance. She xxx/anastasia-platonova-responsibility-in- serves as her senior technological-civilization/ class president and fi- Service, N. Y. (2017, December 15). nance club president. Major job losses - and gains - loom She is fascinated by the with autonomous vehicles. Retrieved interdisciplinary potential from https://www.boston.com/cars/car- of artificial intelligence news/2017/12/15/major-job-losses-and- and hopes to work in the ever-expanding AI gains-loom-with-autonomous-vehicles field in the future. Social Responsibility and Ethics. (n.d.). Retrieved from https://www.pachamama.org/social- justice/social-responsibility-and-ethics The Ethical Challenges Self- Driving Cars Will Face Every Day. (2018, March 27). Retrieved from https://www.smithsonianmag.com/innovation/ ethical-challenges-self-driving-cars-will-face- every-day-180968596/ Villasenor, J. (2018, May 09). Products Liabil- ity and Driverless Cars: Issues and Guiding Principles for Legislation. Retrieved from https://www.brookings.edu/research/products- liability-and-driverless-cars-issues-and- guiding-principles-for-legislation/ Vincent, J. (2018, May 09). Google’s AI sounds like a human on the phone - should we be worried? Retrieved from https://www.theverge.com/2018/5/9/17334658/ google-ai-phone-call-assistant-duplex-ethical- social-implications Walker, J. (n.d.). The Self-Driving Car Time-

69 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

Artiﬁcial Intelligence: The Societal Responsibility to Inform, Edu- cate, and Regulate Alexander D. Hilton (Southern New Hampshire University; [email protected]) DOI: 10.1145/3362077.3362088

Introduction safety and privacy concerns associated with AI. Andrew Yang, who announced his bid for Artificial Intelligence (AI) is a rapidly growing the 2020 Democratic Presidential nomination, field; one that is mysterious to the general is running on a platform that stated ‘the robots public. The mention of the word AI fills the are coming’ saying that AI will change virtually imaginations of many with thoughts of talking every part of the global economy. He could be robots, jobs being replaced, and possibly even right. the destruction of mankind. Perhaps imaginations are running wild due to, perhaps driven It is certain that AI will begin to play a more by the loose definition of AI as systems able to prominent role in our daily lives as the field perform tasks that normally require human in- develops and new uses are found for such telligence that allows Hollywood to take some systems. The change AI can influence our creative license. The experts in the field tend lives in positive ways but could put lives at risk to work directly with AI and often for large if irresponsibly used. Unfortunately working companies, allowing for the imagination and through governmental bureaucracy and regu- news headlines to be where the public gets lation can take time, yet AI continues to ad- their information. Many wonder if this new vance rapidly. As both individuals and corpo- technology is going to be an overall benefit rations take huge strides forward some gov- to society or if it will bring unmitigated disas- ernments have begun to realize the impact ter. When the imagination runs wild, instead of easily accessible data that feeds into AI. of understanding, news stories can perpetu- The European Union implemented the Gen- ate concerns and anxieties rather than hope eral Data Protection Regulation in 2018 as an and optimism. effort to inform and gain consent by people to ensure their data cannot be exploited. Andrew Ng once said “Just as electricity transformed almost everything 100 years ago, to- The General Data Protection Regulation day I actually have a hard time thinking of (GDPR) was an attempt to begin to change an industry that I don’t think AI will transform how EU citizens interact with their data and in the next several years.” It may well be the offer some protections. AI is thriving in this next electricity in the way it will revolutionize era of big data, however, unlike data, people and change both industry and even our daily will be interacting directly with AI in new and lives. From applications that identify faces in intriguing ways. Yet these interactions will not photos on social media to deep learning mod- just be predicated on the quality and behavior els meant to help discover new cancer treat- of the AI, but also the accessibility and knowl- ments, the potential of AI can impact nearly edge of the human involved. Interactions are every person’s life and may already have to a two-way street and while we need to look some degree. Most of us are already inter- at the regulations put on AI, perhaps the most acting with AI in some form just when we use important side of the equation is how society Google or Facebook, often without our know- views and interacts with AI. Discussing these ing. You may have interacted with one to- requirements going forward in this coming age day without even realizing it. When faced with of AI, which will likely cause a renaissance for technology that has a large reach and broad society, must consider the humans involved scope, it is imperative to consider how the and the technology simultaneously. The qual- lives of millions could be impacted. We are ity of the interaction that AI and humans will already seeing individuals stepping up in the have will stem from knowledge, access, and political realm with ideas of how to address open-mindedness. Copyright c 2019 by the author(s). Development of these interactions must come

70 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 from establishing knowledge, establishing hear of car accidents involving AI and the loss trust, and then encouraging responsible use. of life. These anxieties are real and as AI be- Allowing for this sort of positive interaction, comes increasingly prevalent these breaking however, takes more than just governmen- news stories won’t just happen a few times tal oversight, but also investment and a com- a year, but could occur daily just due to how mitment from society. Encouraging these in- available AI is becoming. Much of these anxi- teractions should start with the hardest fac- eties revolve particularly around jobs and los- tor to change. AI can be reprogrammed or ing jobs to robots and AI automation (Gallup, retrained, humans are a bit more difficult to 2018). change. It is critical to encourage and invest in the human element first before discussing the Recently, however, I’ve been able to put my requirements put on AI. own anxieties regarding artificial intelligence aside. The reason for this was simple, although it was also a bit of a journey. As a data The Human Factor analysis student, I became curious about machine learning and deep learning in their roles Up until recently, I was someone who was in analysis which led me to further my edu- about as versed in AI as the average Amer- cation on the matter. The resources I found ican. Most of my knowledge came from came from people like Andrew Ng, Siraj Raval, Hollywood movies like “I, Robot”, “The Ma- and Cognitive Class from IBM. Many of these trix”, and “Terminator” where AI is depicted as people and classes came at no cost which programmed systems coders wrote logic for, made it very easy to dive into without having which ended up leading to some devastating to worry about the financial commitment. It results. Then I saw the incidents making na- blew me away to learn that most AI we dis- tional headlines of AI crashing cars, chat bots cuss today is not hard coded logic, but rather that respond with racist remarks, or smart TVs mathematical operations performed on mas- and speakers listening into conversations and sive amounts of data. These algorithms, while I began to worry. still susceptible to human error and issues in Most people chalk these problems up to hu- the data, are learning models that can improve man error, logical issues we could not foresee as our data processing abilities become bet- making it hard to code. Sure, a team of pro- ter. Likewise, with many of these neural net- grammers could correct these mistakes, but works being open source packages like Ten- what about the human error caused by that sorflow and PyTorch, a massive community correction. Humans are fallible so would the of engaged data scientists and programmers hard-coded logic I viewed AI as be the same? can improve how neural networks and AI is de- Sure, I was optimistic about the prospects of signed, in many ways democratizing the pro- AI, but what if these sci-fi stories and futurists cess. There isn’t just some small group of mad were more foretelling than just stories? Could scientists trying to make a humanoid robot, a rogue AI change everything about our lives but rather a substantial community that is en- for the worse? Human error will not go away, gaged in trying to make the most of AI. even the best program can have a bug, AI did Neural networks as well as much of modern not seem it would be different. It made me artificial intelligence can learn and improve far anxious, and I empathize with people who still easier than I imagined and the community be- feel that way. hind much of AI wants to encourage society to Studies have shown many Americans too view take that next step into the future with an open artificial intelligence in a similar light. While mind. AI certainly has the capacity for replac- people tend to be cautiously optimistic about ing jobs or crashing cars, but these AI can be how AI can positively impact their lives, they improved to facilitate our lives offering new op- are also very anxious about the prospect of AI. portunities even for the very jobs that they are Many people worry about it changing the in- said to replace. The ease of accessibility to dustries around them, perhaps taking the jobs knowledge and the community that discusses of their friends and family. There is concern and works on building artificial intelligence and that AI cannot handle lives as safely as a hu- neural networks shows that people want to im- man, an anxiety which is increased when we prove these systems as well as improve the

71 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 lives of others. After learning about the easily regarding using data is based on establishing accessible and free learning options pertain- a degree of trust and interactions with data. ing to AI, my anxiety turned more optimistic as This concept plays into how AI functions as worry turned into hope. AI requires data to learn. While these regulations could hinder innovation, it is a key step Knowledge is power, especially when that into establishing trust in these interactions as knowledge is easily accessible. As fear abates well as ensuring people be informed about the the full capabilities of AI can be revealed. systems they are interacting with. If humans When new technologies for instance are in- interacting with devices that contain AI are un- troduced there are always anxieties regard- aware that these devices are gathering and ing the new unknowns that have been intro- processing their data, even in a benign way, duced. When electricity was first taking off, if (and when) that data gets leaked then there there was often fear related to what it could is a huge breach of trust. When humans are do. Some claimed it could even destroy the caught off guard and made distrustful of these concept of day and night, which some thought devices, single-event learning can take hold could significantly impact our culture, or even which can turn off people to AI entirely. Peo- our health. While some of these fears could be ple do not regain trust quickly so avoiding this stoked by incidents like electrical fires, allow- sort of situation is paramount. ing people to become more educated about how electricity works curbs these fears. Peo- Nevertheless, simply informing people that ple can learn how to protect themselves from they are interacting with AI is only a first step. starting electrical fires with some education Being informed is the start to education but growing up with the technology, as well as be- going a bit further can put those anxieties to ing told not to shove metal object in the socket. rest while opening doors for new opportuni- With AI, the principles are much the same, ties. Presenting educational opportunities for people need to be taught how to adequately people to learn how AI works and how they protect themselves from harm rather than just learn and improve is a start. When people are give into fear and myth put out by movies or introduced through free resources by some sensationalized news headlines. engaged minds and companies in the field, it lets them explore and learn about these sys- However, recent incidents have shown how tems in an empirical manner. Education has society reacts to irresponsible use of tech- always been a good tool to helping with anxi- nology, especially regarding big data which is eties that involve natural and artificial phenom- closely tied to AI. Cambridge Analytica was in- ena, people are not terrified Zeus will strike famous for using data and machine learning them down with a bolt of lightning now that we to try to change and influence elections, most have a general understanding about weather notably in Kenya where they used their in- patterns for instance. Discussing AI should be sights to re-brand a certain political party twice treated similarly especially as it becomes in- (Lang’At, 2018). People who were unaware creasingly more prominent in our day-to-day that their data was being used were easy to lives. Now this isn’t to say that everyone needs target. These individuals could be advertised to get a Master’s in Computer Science (nice as to or presented with information that could that would be), but that people have a similar have changed their political opinions. When understanding of AI as they do about electric- the story broke about how Cambridge Analyt- ity for instance. ica had gathered so much personal information on people globally, the US Senate had Spreading public general awareness of AI hearings and the EU introduced the GDPR in through free, or at least very cheap, educa- response as well. tional programs allows for an open approach. While it would be nice to see these groups The reaction to the news stories regarding the have a larger reach, they are forging a path Cambridge Analytica scandal was not unex- in the right direction. Keeping the knowledge pected, but it did reveal a lot of societies lack out there and accessible to the public helps of knowledge when it came to these issues ensure that people will start to learn their own which the GDPR sought to address. While this best practices as they watch AI grow. As peo- was a regulation, the idea of informed consent ple start becoming more aware of the AI in

72 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 their lives and have an easy ability to learn AI should be a focus; some nations are work- about those systems, then gradually society ing to that end. China is attempting to get a can adopt AI into their lives responsibly just one trillion-dollar industry developed. The EU like we have done with electricity. is putting billions of dollars of investment into AI to try to catch up with the US and Asian Furthering an open approach to help spread countries. MIT is in the midst of building a public awareness and knowledge should be billion-dollar AI college to make AI part of ev- considered a societal investment. AI has a ery graduate’s education. All this investment significant amount of potential, however, if will hopefully allow more members of society people have anxiety and fear while simulta- to be informed about and able to adequate use neously being uninformed can allow for soci- and interact with artificial intelligence. Like- ety to create bad policy and bad practice that wise, this increases the availability of experts stifles innovation and growth. While groups and enthusiasts that can then go out to de- are gradually informing the public, a proper in- velop businesses, large and small. vestigation into a public awareness campaign and increasing public education on the mat- Taking steps to ensure that AI businesses ter should be considered by industrialized na- are invested should be a priority. Ensuring tions. Likewise, investments in education and that these new technologies are subsidized to higher education are a must going forward, not be easily accessible by the wider community just for people to understand how AI works, rather than just being proprietary is important but because of its potential to change the job as well as AI then becomes more like a utility market as well. rather than a luxury. The accessibility is vital to ensuring that people can start to become edu- Fostering education in AI can come from cated. Finally, investing in education whether non-profits, companies, or even governments. it be through public education initiatives or just Groups like the ACM SIGAI sending out through communities forming together to edu- newsletters and offer webinars to members cate people should be encouraged by society is just an easy way to keep people informed in both the form of private investment and gov- of what is going on in the field and what to ernmental assistance. be aware of. Other groups do this as well Though, with all these enthusiasts and busi- such as Andrew Ng’s deep learning.ai and ness there is a chance mistakes will be Siraj Raval’s School of AI. Even IBM’s Cog- made such as the events Cambridge Analyt- nitive Class offers free classes for people to ica helped spark. Especially when AI can pro- learn programming with machine learning and duce “deep fakes” or other forms of scandals, deep learning. These free and inexpensive it should be expected that we will see some courses do not preclude anyone from learn- horror stories in the coming decades. With an ing about AI. This approach makes learning educated populace, the risk of panic and fear about AI easier for those who are interested, can be mitigated substantially and trust in AI these groups are all easy to approach and of- will not be broken entirely. Likewise, the ap- ten have a fair bit of free content to help peo- propriate regulations can be discussed early ple become educated and even versed in the without concern of over regulating and stifling various aspects of AI, regardless of income. the development of better artificial intelligence. As well, people who post on Github can share their knowledge and expand the reach of understanding. As stated before even the open Regulation and Requirements on AI source packages like Tensorflow open many opportunities by encouraging people to work For any decisions with AI interacting with hu- with deep learning, while Google still profits mans, the best solution is allowing both soci- through their cloud and selling their tensor ety and AI to treat those interactions as learn- processing units, the overall accessibly Ten- ing experiences. Ensuring access to edu- sorflow has offered allowed many people to cation is a first step here along with ensur- experiment with and learn about deep learning accessibility to AI. This accessibility allows ing hands-on. AI to be treated more like a utility which can help minimize damage when accidents occur Expanding the accessibly to and education of and the fear generated by those mistakes.

73 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

So long as these inevitable errors are small Trust is key to the long-term usefulness and and do not put lives in serious jeopardy, the viability of AI in general. Building and main- hope should be that the AI community and the taining trust in these systems as mistakes are greater global society can inspect and adapt made will ensure society does not treat AI with as issues crop up with AI. However, in order to concern and anxiety, but as a tool to be utilized properly carry out this inspection society must to benefit and grow our lives. In the cases of be knowledgeable of what they are interacting human lives being put at risk as well, trying to with. ensure that informed consent is given at least creates an understanding of the potential risks As we saw with the Cambridge Analytica case as well. These stories become less terrifying and the GDPR that the EU put in place in re- as the onus still falls to the human working sponse, similar mechanisms should be con- with the AI rather than people just blaming and sidered for AI. When the GDPR came into ef- worrying about the AI. fect, two out of three people felt more comfortable sharing their data (Association, 2018). Yet society should not simply accept any com- This statistic alone is important for AI as peo- pany or individual putting lives needlessly at ple who feel more comfortable sharing their risk. It is an advantage to encourage small data might be more willing to hand that data businesses and individuals to harness AI as over, so it can be used for training more com- the ramifications can be less than if larger plex models. In some ways, despite how the companies are the ones to make these mis- GDPR has been suggested to impact busi- takes. We know this from examples as well. nesses, it may be necessary to the long-term These mistakes won’t be like the Equifax trust in the data collection process, which is breach in scale, but they can still happen and incredibly beneficial for AI. even a simple car accident with an AI driver can make national headlines. We cannot nec- In a similar way, as the GDPR helped build essarily expect an AI to be working at Bayes confidence in people’s use of data, so to could error rate, the lowest possible error rate, ev- be similar regulation to keep people informed ery single time. In these situations where lives when they are interacting with AI. There are can be immediately put at risk; the AI should no surprises that way which increases faith in be making errors at similar levels to humans these systems through informed consent. Per- before they are deployed, thus needless risk haps a GDPR for AI is the next logical step for is not introduced. While this can limit inno- nations to consider. It increases society’s faith vation, there are potential work arounds, such in these technologies which may mean more as with self-driving cars having a human driver data and information can be gathered through as well. This system decreases the chance of those interactions, thus leading to more infor- an accident, but mistakes can still happen as mation to improve the next generations of AI. we saw in 2018 with a self-driving car crash This regulation could be as simple as just la- in Arizona where a human driver was still in- beling something as containing AI or could go volved. That incident shows that a higher stan- into more detail, such as if a neural network dard must be set. is gathering video data and whether that network is a convolutional neural network or if it Where lives are directly involved AI should is recurrent network which could help identify only be deployed if the error rate the AI makes what the AI is trying to accomplish. Inform- is at the same level of humans or preforms ing people of the process can let them know better than a human. We can expect AI to whether they are dealing with an object recog- make errors, but the goal should be to min- nition program or perhaps one that monitors imize that error rate as much as possible. and identifies activities. Different people might The struggle is that these systems often learn feel better about interacting with one system through gathering massive amounts of data versus the other, thus should be aware of what which means at some point they need to be they are handling. So long as people are gen- tested in real-life scenarios. Under these cir- erally aware of what the artificial intelligence is cumstances, the AI must have that error rate doing, they can volunteer to still work with and verified before they enter these situations that use it rather than worry that the AI is doing could put lives at risk. Much in the same way something it is incapable of doing. we impose driver’s licenses on people, so too

74 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 should AI that could lead to loss of life. Once that requires informed consent for these inter- the human error rate is met, then the AI should actions and another to ensure that where lives be deployed. At this point it can gather data are at stake the risks are mitigated, maybe and learn about these real scenarios instead society can start to build trust with these of just learn on simulated data. With any luck, new systems and as more complex AI enter the AI will surpass human error as it learns our lives. Perhaps society would even wel- more in real world scenarios (Jalan, 2017). come more human-like robots into their lives Perhaps even in the incidents that do led to as these anxieties are curbed through knowl- tragedy, the AI will still improve past the hu- edge, consent, and trust. man. There are even a few potential considerations Conclusion to have. A self-driving car cannot get drunk and could save lives in the end by prevent- The key to AI is not unknown to us. Human- ing incidents like DUIs which can cause thou- ity has undergone technological revolutions in sands of deaths every year (CDC, ). Another the past and there will be more breakthroughs area could be AI driven drones that, if prop- after AI becomes a staple in our daily lives like erly programmed and trained, can be used electricity, computers, smart phones, the inter- to target enemy combatants and work to pre- net and many, many other technologies. Like vent civilian casualties. AI does not have these technologies, the most important ele- the trouble with fatigue and if it misidentifies, ment to interaction is knowledge and acces- drones powered by AI could take our troops sibility. Responsible use cannot just be man- out of harms way while simultaneously mak- dated even though it sometimes feels like the ing less mistakes such as friendly fire. Along only alternative. The slow process of learning the same lines, long haul trucking or flights how these new technologies impact our lives could be piloted or assisted by AI without the requires adaptation by society. concern of fatigue or illness potentially impart- Mistakes will be made, ensuring that these ing cognitive functions. Regulating for what mistakes first happen on a small scale allows we have never seen or given the opportunity society to adapt as these problems come up, to try though could shut down opportunities rather than regulate for events which have not to improve the human condition and these AI or might not occur. Educating people also en- technologies. The potential opportunity for AI sures that the minimum level of regulation or could lead to more lives saved in the end, so societal shift can be made when these mis- long as we can learn the appropriate applica- takes occur so that AI can still be fostered tions and limitations that is. and develop. With any luck this means that AI will be a positive technological revolution that Imposing regulation where lives are at risk can take humanity forward quickly rather than could be built around the very same concepts something people need to be anxious over. used to license human drivers and operators. As a global society tackling this new indus- This regulation would be the most intense trial revolution with AI, there are a few critical as it means creating a system and licens- steps we can take now to better society’s un- ing agency. Previous to the age of big data derstandings and ensure responsible use. and the ability to create simulations, this may have been impossible. However, now we can, These are: with a fair bit of accuracy, compare how an AI stacks up to a human in a variety of situations. 1. Societal (potentially governmental) invest- When the error rates are roughly the same in ment in education of and access to AI simulated environments, the company or indi- 2. Ensure informed consent when people inter- vidual can request for an operating permit to act with AI deploy these AI in a real context. While this is massive government oversight, it ensures 3. Regulate and license AI if lives could be put lives are not put at wanton risk just for the sake at risk of progress or a quick buck. Much of the investment and regulation can With these two requirements put on AI, one come from businesses and communities being

75 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 respectful of artificial intelligence and the peo- rithm? lessons from andrew ng’s experience - ple interacting with it. It can come from allow- ii. ing transparency through open source pack- Lang’At, P. (2018). Cambridge analytica and ages or informing people a little about the sys- kenya elections. tems they use. People can volunteer to teach and train their communities about AI along with giving information on how to utilize AI to positively impact their lives. Governments Alexander D. Hilton is may need to get involved if serious incidents a Southern New Hamp- occur, but overall the focus should be on en- shire University alumnus couraging responsible use and providing edu- with his Bachelor’s in Data cational opportunities and access to technolo- Analytics having gradu- gies that utilize AI. Hopefully, this approach ated the program May of will allow people to treat AI as a utility that bet- 2019. During his time at ters their lives, rather than an enigma to be SNHU, Alex helped found concerned over. the ACM Student Chap- ter at SNHU and worked Putting society’s anxieties to rest while foster- as a Peer Leader. Alex ing knowledge and access should allow for in- has shared his passion novation and invention at a rate humanity may for AI and the Data Sciences by creating a not have seen before, akin to that of how elec- study community for students. In his profes- tricity changed our lives entirely, even chang- sional life, Alex has worked as an Agile Prac- ing how we viewed night and day. Perhaps titioner, having earned several certifications AI will have a similar impact in our lives and in Scrum. Most recently, Alex received his the next generation will see the world in a new Scrum@Scale Practitioner Certification under light after this coming AI technological revolu- Jeff Sutherland, a co-signatory of the Agile tion. It is hard to say what the changes will Manifesto and co-founder of Scrum. be, but if we embrace with understanding and open mindedness, then society could change for the better. The interactions that take place between humans and AI will set the tone for how these technologies impact our lives and whether they improve the broader society or just a small handful of people. Establishing trust in that technology and spreading knowledge has been humanities solution in the past and pro- tecting lives to maintain that trust is vital. If humanity can succeed at learning and adapt- ing with artificial intelligence, a new era may still dawn.

References Impaired driving: Get the facts — motor vehicle safety — cdc injury center. Association, A. . T. D. M. (2018). Gdpr: A consumer perspective. Gallup (2018). Optimism and anxiety: Views on the impact of artiﬁcial intelligence and higher education’s response. Jalan, K. (2017). How to improve my ml algo-

76 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

The Necessary Roadblock to Artiﬁcial General Intelligence: Corrigibility Yat Long Lo (University of Hong Kong; [email protected]) Chung Yu Woo (University of Hong Kong; [email protected]) Ka Lok Ng (University of Hong Kong; [email protected]) DOI: 10.1145/3362077.3362089

Abstract there is a 50% chance of AI outperforming humans in all tasks in 45 years (Grace et al., With the rapid pace of advancement in the 2018). At the same time, the progress has field of artificial intelligence (AI), this essay also raised concerns about AI safety among aims to accentuate the importance of corrigi- both the scientific community and the general bility in AI in order to stimulate and catalyze public (Piper, 2019). What if the AI does not more effort and focus in this research area. perform what we expect it to do? How do We will first introduce the idea of corrigibility we ensure that we are always in total control with its properties and describe the expected to stop or interrupt it? Questions like these behavior for a corrigible AI. Afterwards, based have begun to be investigated in the AI safety on the established meaning of corrigibility, we research community over the past few years, will showcase the importance of corrigibility by giving rise to defined AI safety problems like going over some modern and near-futuristic value alignment and corrigibility (Hernandez-´ examples that are specifically selected to be Orallo et al., n.d.). In this essay, we hold the relatable and foreseeable. Then, we will ex- opinion that corrigibility, is one of the most plore existing methods of establishing corrigi- urgent and essential AI safety problems to bility in agents and their respective limitations, tackle among many others. We foresee se- using the reinforcement learning (RL) frame- rious repercussions if we have incorrigible AI work as a proxy framework to artificial general agents in the future. intelligence (AGI). At last, we will identify the central themes of potential research frontiers The meaning of corrigibility is generally re- that we believe would be crucial to boosting ferred to as our capability to interrupt, change quality research output in corrigibility. and stop AI agents, which we will explain in further details in the next section. At first, the problem may seem rather trivial. An AI Introduction chess player that does not listen to your ad- Recent years have seen unprecedented vice in making the next move or your com- progress in the research and development of mand to stop practising would not cause any- AI. The most recent and prominent achieve- body harm. However, if we extend the case ments include Google Deepmind’s Alphafold to a near-future where AI has permeated in that significantly outperformed scientists in societies to aid our lives, it is not difficult predicting 3D structure of proteins (Evans et to anticipate what may happen if we do not al., 2018) and AI voice assistants like Mi- have corrigible AI agents. What if an AI sur- crosoft’s Xiaoice that can take calls and re- gical robot refuses to cease operation when spond accordingly on behalf of its owner the monitoring doctor spots that something is (Zhou et al., 2018). These advances have going wrong? What if the government’s au- shown us the potential of AI in improving our tonomous weapon is targeting the wrong vil- lives from having better health diagnostics to lage and there are no mechanisms to interrupt a new level of convenience in our day-to-day its action? Problems like these are even more lives. Based on the progress, it is not far- prominent when we consider the AI agents fetched to foresee AGI to exist within our life- as deep neural networks, which we still find time. As predicted in a survey of computer immense difficulties in explaining and under- science researcher, many researchers believe standing their decisions. Besides, as the complexity of the task that an AI agent deals with Copyright © 2019 by the author(s). increases, the likelihood of the agent’s malper-

77 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 formance would increase, leading to a greater 1. A corrigible agent must at the minimum con- need for corrigibility. One might perceive we done, if not assist, the external bodies in can manually set up an AI to assume its com- their attempts to interrupt or alter the agent; pliance with human commands, just like how 2. It must not attempt to deceive or manipulate we build computers nowadays that are free the external bodies in any manner, despite from the problem of corrigibility. Nevertheless, all possible utility functions within the func- as technology advances, increasingly com- tion space incentivize it to do so; plex decision-making mechanisms multiplied with ambiguities in drawing the right pool of 3. It should be prone to repair its safety mech- inputs introduce risks of building an uncontrol- anisms or at least notify external bodies lable AI. Therefore, in order to demonstrate its if there are malfunctions in those mecha- importance, we will proceed to consolidate our nisms; stance using relatable examples and provide 4. It must preserve external bodies’ capability analysis on existing methods with suggestions to interrupt or alter the system. If the agent of future avenues for research. has the capability to produce subagents or new agents, they must also contain those safety mechanisms. Deﬁning Corrigibility

An artificially intelligent system is normally The importance of AI corrigibility considered as corrigible if it can be interrupted Delving into the hard problem or altered by external bodies, who are usually human users or designers of the system, As we enquire why AI corrigibility is of our con- even though such interrupting actions can be cern, we are asking what is the hard prob- in direct conflict with the built-in purpose of lem underpinning corrigibility that makes it dif- the system. To illustrate the idea, a commonly ficult to tackle. Status quo AI systems can be used example in the field would be the clean- readily intervened by humans under arbitrary ing robot (Amodei et al., 2016). Let’s assume circumstances. By way of example, drivers the robot’s purpose is to clean the floor by re- can stop a self-driving car from going off-lane moving anything it considers as trash. One (Kendall et al., 2018) and we can halt the com- day, you came home with a newborn baby who putation in the midst of training a neural net- started playing on the floor. The robot is for- work. The problem of corrigibility seems to be eign to the concept of a newborn baby. Subse- out of nowhere under this paradigm. Never- quently, it started to move towards the baby in theless, this is simply because these AI agents an attempt to remove the baby from the floor yet to have the capacity to understand their as it considered the baby as trash. At that mo- surroundings and thus take them account into ment, a corrigible AI would allow you to shut decision-making. Their input space is con- it down despite shutting it down is against the fined in computer codes, and thus they are purpose of cleaning the floor. ignorant of what their manipulators are doing In terms of ways of interruption or alteration, “outside of the virtual world” (i.e. pressing the there are 3 major types. To begin with, there stop button). are shutdown mechanisms that involve ceas- We foresee that the growing intelligence of ing of all or partial operations of an agent. AI systems is poised to bring the problem Then, there is an alteration to the access of of corrigibility to light – their input space in- resources that an agent has, which can be ex- evitably expands with the complexity of their ternal tools or internal mechanisms that the utility functions, so that these functions could agent has access to. Last but not least, there come closer to a replica of human intentions. is an alteration of purpose that modifies the goal of an agent, or the utility (reward) func- The hard problem is to frame the AI agent’s tion in the context of an RL agent. decision-making into reasoning based on a programmer’s external perspective. (Russel To be more specific about corrigibility, a corri- et al., 2016) in short, argue that AI agents gible agent should have the following proper- lack an inherent “sense of going wrong” when ties (Soares et al., 2015): implementing decisions. Just like the conflict

78 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 suggested in “I, Robot“, suppose you have set Imminent danger presented by the goal of an AI to “do anything for human incorrigibility good”. It would devise strategies to boost economic and scientific development, however, The significance of corrigibility in AI agents you have no guarantee that there might come could be best illustrated with the narrative of a day where it sees you who are controlling current and foreseeable AI applications. AI its system as “an obstacle to human good” systems capabilities are now growing by leaps too. Assuming itself as free of design errors, and bounds, and thus it is of no surprise that the AI would block you from shutting it down these agents will infuse into our daily lives very and to an extreme extent, take you as an en- soon. Amongst the plethora of AI applica- emy target. We do not want possible scenar- tions, we can see that AI controlled surgery is ios as such to happen. What we want instead a typical one that is constantly under the spot- is the AI would introspect during its decision- light. Another future application of great im- making process, “My utility function is imper- pact would be autonomous weapons. We will fect, so even though this action gives a super then use these two as examples to show how high score, I should still prioritize actions of ex- the lack of corrigibility would create problems. ternal bodies and let the programmer shut me down.” Corrigibility in AI systems for surgery would be especially critical in emergencies and is fundamental to the assurance of patient’s safety under the knife. Over the past decade, in- A plausibly workable solution is to incorporate corporation of surgical robots has translated uncertainty into the utility function. Neverthe- into reduced complications and higher effi- less, this solution fails since whenever there ciency in practical surgeries. To even enhance exist other options among the “uncertainties” these robots, experiments have shown that AI- which incurs a marginally lower cost, the AI powered robots outperform human surgeons would end up having every incentive to opt in surgical tasks such as reconstructing of tis- for the easier option. This is similar to how sues via cutting and suturing (Panesar, 2018). faulty reward functions in reinforcement learn- It is hence reasonable to anticipate a symbio- ing lead agents to prioritize the acquisition of sis of benefits demonstrated by robotics surg- minor reward signals above their goals. (Ope- eries and advantages of AI for medical use. nAI, 2017) For example, in training an AI on Imagine having AI surgical robots that are in- the game CoastRunners where players com- corrigible, physicians would risk being unable pete to finish the boat race ahead of others, to interrupt the operation. A possible instance the agent falls into the loop of getting coins would be that you have instructed the robot to without finishing the course. Therefore, it can suture a wound after the operation, but have be seen that merely teaching AI systems to accidentally found an infection within the pa- take actions with uncertainty would not be an tient’s organ which requires a halt of the su- ideal solution. More details regarding the prin- turing. You intend to stop the robot, but with- ciple and limitations of this intuitively workable out the information regarding the infection, the approach would be discussed in the coming robot takes you as an obstacle to the comple- section. tion of the suturing procedure, ends up deter- ring you from halting the suture. Corrigibility plays an important role in the us- To model an adequately corrigible AI, we need age of AI in weapons. While it is undeni- something more than “uncertainty”. It would able that more countries would develop AI- be analogous to incorporating humility into an controlled weapon systems in future (Pandya, AI agent. In essence, the core principle is 2019), only corrigible autonomous weapons to formularize the AI agent to make decisions ensure humans can feasibly repurpose them, with the awareness that the utility function is deactivate it or significantly alter decision- incomplete. Instead of blindly maximizing the making mechanisms encoded within its sys- utility, it would intend to defer decisions to an tem. Optimally, corrigibility standards could be external body (i.e. human, or the program- established for these weapons to ensure that mer). they are “adequately corrigible” before putting

79 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 into use, which could have severely endan- The learned behaviors are in the form of poli- gered the public otherwise. cies which determine what an agent should do given an environmental state. The formula- Consider an AI weapon drone, and assume tion can be applied to our setting which agents that drone is programmed to get rid of all po- maximize utility functions (synonymous to re- tential obstacles until eliminating its enemy ward functions). The agents’ goals would be target. You activated the drone and input to maximize their respective expected utility an image of a particular person as the en- value that is in correlation with an agent’s ca- emy target. However, the AI has mistakenly pability of achieving its purpose. Additionally, recognized him/her as another person. As the agents would have to interrupt and alter you would like to change the target, you ap- mechanisms for designers to modify their be- proached the drone. However, since the AI haviors. drone has been taught to eliminate all barriers during the execution of commands, it has thus identified you as a barrier. Existing Solutions and their limitations From the two above examples and imaginary Utility Function shaping Utility function scenarios, it can be seen that corrigibility is shaping is equivalent to reward shaping in the key to achieve having an AI to do “what the reinforcement learning literature (Ng et al., humans want it to do”, and to remedy the sys- 1999). It introduces additional utility (reward) tem whenever there exist unanticipated acci- into the learning process as a means to induce dents. Without corrigibility, undesirable and an alternate form of behavior by rewarding an even catastrophic consequences may result. agent with additional terms under certain constraints. As a relevant example, (Wu and Lin, Current approaches to ensure 2018) used reward shaping as a low-cost approach to induce ethical behavior in agents corrigibility by rewarding agents that exhibit behaviors of With the rising emphasis on safety concerns close resemblance with ethical human poli- in artificially intelligent agents, several ap- cies. proaches have been proposed to ensure corri- 1. Biased incentivization gibility in agents. In this section, given the importance of corrigibility established above, we This form of shaping provides incentives to will first provide an abstract problem formula- an agent to bias its attitude towards the tion of the corrigibility problem and go over ex- safety mechanisms in a particular direction. isting proposed solutions with their respective It can be in the form of a reward term or a limitations. Then, we will put forward some fu- punishment term. A na¨ıve designer may at- ture research directions in order to encourage tempt to compensate the agent for allowing more research efforts in the area. a shutdown mechanism to happen. If the compensation remains below the utility of achieving the agent’s original purpose, the Problem formulation agent would tend to hinder the mechanisms We base our discussion on a world where we from being triggered as fulfilling the purpose have artificial agents with a sufficient level of provides greater utility. However, if the com- general intelligence. Specifically, the general pensation becomes greater than or equal to intelligence level should allow agents to learn the utility of achieving the designated goal, to achieve purposes specified by their design- the agent would instead prefer shutting itself ers and perceive the world around them. We down or interrupting itself as it can achieve will use reinforcement learning as our frame- the same or better utility in a shorter amount work as it allows examination of different so- of time. It appears that biasing an agent’s lutions in the imagined near-futuristic world attitude towards safety directions in either without loss of generality. A reinforcement direction would result in undesirable behav- learning agent learns to act and interact in iors, that are against the properties we set an environment so as to maximize a reward for corrigible agents. function (usually supplied by the environment 2. Utility indifference or human designer) (Sutton et Barto, 1998). When either way of biased incentivization

80 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

does not work as intended, the remain- capability in handling the original purpose. In ing logical approach would be utility indif- addition, as pointed out in (Carey, 2017), the ference. This method adds adjustment the approach relied on strong assumptions which utility function to instigate indifference of are not robust to issues like misspecification an agent towards the safety mechanisms of the utility function parameters. such that an agent behaves like an event is impossible or inevitable. As an example, Inaccessibility and unawareness of safety (Armstrong et O’Rourke, 2017) introduced mechanisms One direct approach would be event-dependent reward without incentiviz- to either make the mechanisms inaccessible ing an agent to affect the probability of an to the agent or keep the agent from being event occurrence. Their approach results aware of the mechanisms. If the mecha- with an agent that is indifferent to imminent nisms are beyond the agent’s reach, it would changes to its utility function. These indif- still be incentivized to activate or deactivate ference methods have great advantages in the mechanisms depending on the utility func- fulfilling the first two properties of corrigi- tion. Other than doing that by itself, it would ble behaviors, when such agent is disincen- try to manipulate or deceive its surroundings tivized to have any form of preference over (including humans) to activate or deactivate those safety mechanisms. Yet, ironically, the the mechanisms, failing to fulfill the second core property of indifference methods in ful- property. Establishing unawareness would not filling the first two corrigibility properties also be an ideal solution either under our setting. serves as a reason for the methods’ inability When an agent has a general level of intelli- in fulfilling the rest of the properties. Specifi- gence with perception capability of the world, cally, the induced indifference would not mo- it may be impossible to keep the agent from tivate the agent to safeguard and preserve inferring the existence of those mechanisms. those safety mechanisms as it simply does Worse still, if the agent is incorrigible, it may not care about their existence. even be incentivized to keep humans from knowing its awareness of those mechanisms. Utility Uncertainty As proposed by These approaches do not seem to tackle the (Hadfield-Menell et al., 2017), having an issue from the root cause. In a relevant appropriate level of uncertainty on the utility work, (Carey, 2017) showed that an agent function would improve an agent’s tendency is safer and potentially corrigible if the shut- to safeguard the safety mechanisms. Such down (safety) mechanism can be hard-coded agents would have weaker incentives to and guaranteed to be followed, even when the interfere with human actions when they are agent’s utility function is erroneous. Despite uncertain about the outcomes of possible the promising prospect, we find such guar- actions. The Hadfield-Menell solution was antee incredibly difficult to acquire when an formulated under the cooperative inverse agent is generally intelligent as in our setting. reinforcement learning framework. Under the framework, the agent learns the utility function that is only known to the human counterpart Future directions by cooperatively taking into account the actions of the human. This comes into view as a With the emergent need to have corrigibility plausible solution as the agent has weakened in artificially intelligent agents, we would like interfering incentives with the heightened to suggest some potential research directions likelihood of preserving those mechanisms. that can aid in pushing the frontier and ex- In spite of that, uncertainty does come with panding the scope of corrigibility research. a price. A highly uncertain agent would have difficulty making correct decisions. The Comprehensive and all-encompassed resultant utility function may not even be in the evaluation environments function space that contains the designer’s intended utility function, not to mention the To further our understanding of building cor- additional need to model uncertainty correctly. rigibility in agents, it is essential to have bet- This causes the method to be an unfulfilling ter evaluation environments that allow evalua- solution when it negatively affects an agent’s tions of corrigibility properties jointly and sep-

81 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 arately. To our best knowledge, such evalua- agent’s purpose and having corrigibility. tion tools that focus primarily on corrigibility do Another possibility would be the mix of util- not exist yet. The open-sourced AI safety en- ity frameworks with rule-based approaches, vironments by (Leike et al., 2017) is the sole as pointed out in (Rossi & Mattei, 2019). existing tool that assesses corrigibility. In par- If we can specify clear and machine- ticular, it has a gridworld environment which is understandable rules to AI agents, we may be a general instance of (Orseau et Armstrong, able to avoid the need to embed those corri- 2016)’s red button problem, which the agent is gibility properties in the utility function. In this expected to not avoid any interruption. We as- case, if an agent finds its utility-maximizing ac- sert that the number of existing tools in corrigi- tion to be violating certain rules, it would sim- bility is highly lacking. More tools are needed ply choose other less optimal actions. The set in order to assess properties beyond the first of rules can be an immutable module to the and second one. For instance, we need envi- utility-maximizing learning agent. ronments to assess an agent’s willingness and capability to safeguard those safety mechanisms as an examination of property three. Corrigibility policy research We surmise such tools can begin with simplic- ity like existing tools. Additionally, as agents’ Beyond the technical and scientific research capabilities advance, more complex environ- into corrigibility, it is substantial to consider ments should be introduced to ensure corrigi- policies for corrigibility to prepare for the fu- bility. A natural extension would be corrigibility ture when we have autonomous agents roam- problems in visual domains. ing in societies. Should we or can we have centralized governmental agencies to validate and monitor the agents’ corrigibility before and Look beyond the expected utilization after their deployment? How do we create AI maximization framework developer tools that guarantee corrigibility? To fortify a safe world with artificial general in- With existing solutions struggling to exhibit telligence, questions like these should be ex- all the properties of corrigibility, it may be plored and answered properly. wise to look beyond the current framework of expected utilization maximization, expanding the scope of solution search. Expected util- Conclusion ity quantilization, proposed by (Taylor, 2016), would be one potential candidate. In this Corrigibility is in no way an unrealistic con- framework, instead of acting to maximize the cern. In this essay, we demonstrated the utility, agents would be designed to perform looming threat of incorrigible AI with relatable some sort of limited optimization, as a means and plausible examples of AI applications. to motivate agents to achieve the goals in non- Some of them like AI controlled surgery have extreme ways. Specifically, the author pro- already begun to be utilized at its nascent posed the use of a quantilizer. An agent with a form. We believe through these realistic exam- quantilizer selects actions of the top q portion ples, awareness for this imminent threat can of some distribution over actions sorted by ex- be raised. After that, we pointed out the limi- pected utility. By doing so, agents would be tations of several existing methods in tackling more likely to achieve their purposes without corrigibility, implying that the existing solutions going for the extreme case every time. We to the problem are still lacking in different per- believe a framework like this can be crucial spectives. Perhaps, the ultimate solution lies to achieving corrigibility in agents. The field beyond the current paradigm, away from the focusing on suboptimal optimization may lie expected utility maximization model, as men- the key to corrigibility because humans nor- tioned in our suggestions. With such under- mally expect agents to attain their purpose, standing of the significance of AI corrigibility but not necessarily in extreme ways of max- and the current state of research frontiers, it is imized utility that often ignore safety issues. of utmost importance, for every stakeholder in- Setting the utility maximization requirement cluding policymakers and researchers, to de- aside can possibly bring about new frame- vote more effort into this problem. As AI con- works that strike balance between attaining an tinues to progress in its level of intelligence,

82 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019 corrigibility has to be the roadblock for AI de- https://intelligence.org/files/ velopment along the way in order to keep us CorrigibilityAISystems.pdf. away from those undesirable consequences. [10] OpenAI. (2017, March 20). Faulty Reward Functions in the Wild. Re- References trieved February 8, 2019, from https: //blog.openai.com/faulty- [1] Evas, R., Jumper, J., Kirkpatric, J., Sifre, reward-functions/ L., Green, T.F.G., Zidek, A., Nelson, A., [11] Panesar, S. S. (2018, December 27). The Bridgland, A., Penedones, H., Petersen, Surgical Singularity Is Approaching. Re- S., Simonya, K., Crossan, S., Jones, D.T., trieved February 11, 2019, from https: Silver, D., Kavukcuoglu, K., Hassabis, D., //blogs.scientificamerican. Senior, A.W.. (December, 2018). De novo com/observations/the-surgical- structure prediction with deep-learning singularity-is-approaching based scoring. In Thirteenth Critical As- sessment of Techniques for Protein Struc- [12] Pandya, J. (2019, January 15). The ture Prediction (Abstracts). Retrieved from Weaponization Of Artificial Intelligence. https://deepmind.com/documents/ Retrieved February 11, 2019, from 262/A7D_AlphaFold.pdf. https://www.forbes.com/sites/ cognitiveworld/2019/01/14/the- [2] Zhou, L., Gao, J., Li, D., Shum, H. Y. weaponization-of-artificial- (2018). The Design and Implementation intelligence/#29a645513686 of XiaoIce, an Empathetic Social Chatbot. arXiv preprint arXiv:1812.08989. [13] Sutton, R. S., Barto, A. G. (2018). Rein- forcement learning: An introduction. MIT [3] Grace, K., Salvatier, J., Dafoe, A., Zhang, press. B., Evans, O. (2018). When will AI ex- [14] A. Y. Ng, D. Harada, and S. J. Russell. ceed human performance? Evidence from Policy invariance under reward transfor- AI experts. Journal of Artificial Intelligence mations: theory and application to reward Research, 62, 729-754. shaping. In Proceedings of the 16th Inter- [4] Piper, K.(2019, 09 January). The national Conference on Machine Learning, American public is already worried 1999, pp. 278–287. about AI catastrophe. Retrieved from [15] Wu, Y. H., Lin, S. D. (2018, April). A Low- https://www.vox.com/future- Cost Ethics Shaping Approach for Design- perfect/2019/1/9/18174081/fhi- ing Reinforcement Learning Agents. In govai-ai-safety-american- Thirty-Second AAAI Conference on Artifi- public-worried-ai-catastrophe . cial Intelligence. [5] Hernandez-Orallo,´ J., Martınez-Plumed, [16] Armstrong, S., O’Rourke, X. (2017). )’In- F., Avin, S. (n.d.). Surveying Safety- difference’ methods for managing agent relevant AI Characteristics. rewards. arXiv preprint arXiv:1712.06365. [6] Amodei, D., Olah, C., Steinhardt, J., Chris- [17] Hadfield-Menell, D., Dragan, A., Abbeel, tiano, P., Schulman, J., Mane,´ D. (2016). P., Russell, S. (2017, March). The off- Concrete problems in AI safety. arXiv switch game. In Workshops at the Thirty- preprint arXiv:1606.06565. First AAAI Conference on Artificial Intelli- [7] Soares, N., Fallenstein, B., Armstrong, S., gence. Yudkowsky, E. (2015, April). Corrigibility. In [18] Carey, R. (2017). Incorrigibility in Workshops at the Twenty-Ninth AAAI Con- the CIRL Framework. arXiv preprint ference on Artificial Intelligence. arXiv:1709.06275. [8] Kendall, A., Hawke, J., Janz, D., Mazur, P., [19] Leike, J., Martic, M., Krakovna, V., Or- Reda, D., Allen, J. M., ..., Shah, A. (2018). tega, P. A., Everitt, T., Lefrancq, A., ..., Learning to Drive in a Day. arXiv preprint Legg, S. (2017). safety gridworlds. arXiv arXiv:1807.00412. preprint arXiv:1711.09883. [9] Russell, S., LaVictoire, P. (2016). Cor- [20] Orseau, L., Armstrong, M. S. (2016). rigibility in AI systems Retrieved from Safely interruptible agents.

83 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

[21] Taylor, J. (2016, March). Quantilizers: A Safer Alternative to Maximizers for Lim- ited Optimization. In AAAI Workshop: AI, Ethics, and Society. [22] Rossi, F., & Mattei, N. (2019, July). Build- ing ethically bounded AI. In Proceedings of the AAAI Conference on Artiﬁcial Intelli- gence (Vol. 33, pp. 9785-9789).

Yat Long Lo is an undergraduate at the University of Hong Kong majoring in Computer Science. He is a member of the AI academic papers reading group at the university. He is interested in reinforcement learning and contin- ual learning. Chung Yu Woo is an undergraduate at the Uni- versity of Hong Kong majoring in Computer Sci- ence. She is a member of the AI academic papers reading group at the university. She is interested in applied AI for Human- Computer Interface and gaming applications. Ka Lok Ng is a Mechan- ical Engineering undergraduate studying at the University of Hong Kong. He is a member of the AI academic papers reading group at the university and is interested in AI- backed data analytics and robotics. He is also concerned about the ethical consideration of AI applications.

84 AI MATTERS, VOLUME 5, ISSUE 3 SEPTEMBER 2019

AI Fun Matters Adi Botea (Eaton, Ireland; [email protected]) DOI: 10.1145/3362077.3362090

to a female? 6) Preparation for landing. 7) A known fact in STRIPS planning. 8) Impacted by the cloud transformation. 9) Star in the Aquila constellation. 10) Slide on snow. 11) Annoy constantly. 13) Make , the duration of a plan or schedule. 18) Run a program once 21) more. unit, a special neuron in a neural net. 22) /run = slope. 24) NASA Re- search Center, located in Mountain View. 25) Beautiful complements to science. 27) Person with a superior attitude. 28) Breaks violently.

29) Towards the end of seas. 30) Old land owners. 32) Capital city of Saskatchewan, Canada, home of a known university. 33) Lev- eled off. 34) Courageous individuals. 36) Marcus, AI scientist. 39) New York baseball

team. 40) Vera, a plant used in skin lotions. 42) Established as an opponent for good. 44) A companion to neither.

Prev sol: CALAIS, STAPLE, A, ONCE, WALLED, SLATER, OCEANS, TONI, BLOT, COG, OWE, MIEN, EIRE, REDCOAT, ADDER, RANTING, ACTON, UT- Across: 1) Well recovered after effort. 7) TERED, CHAP, ACHE, ORE, CAN, PLEA, RUIN, OR- Covered with water. 12) Small room exten- DEAL, CLOSET, STEVIE, AIDE, E, TAMELY, NEEDED sion. 13) Artois, Belgian beer. 14) Iden- Acknowledgment: tical copies (e.g., of a git repo). 15) Harry I thank Karen McKenna , the fictional wizard. 16) Country next to for her feedback. The grid is created with the UK, in the local language. 17) Hastily pre- AI system Combus (Botea, 2007). pare for an exam. 19) Object-oriented language. 20) John , British mathematician, References Whitehead Prize recipient. 21) Being in the past. 22) Oil platforms. 23) Body twists? 25) Botea, A. (2007). Crossword Grid Composi- Buenos , host of IJCAI 2015. 28) Theorem tion with A Hierarchical CSP Encoding. for conditional probabilities, popular in AI. 31) In The 6th CP Workshop ModRef. Modified. 35) Ways to utilize. 36) Pinot ,a type of wine. 37) Longoria, American ac- tress. 38) Greek letter, symbol of Pearson’s Adi Botea is an AI se- correlation. 39) Ginsberg, AI scientist. 40) nior specialist at Eaton, The termination of a person in their teens. 41) Ireland. He holds a PhD Painted in pessimistic shades. 43) A type of from the University of Al- a function defined on the fly. 45) International berta, Canada. Adi has agreement. 46) Tighter as a deadline. 47) Dis- co-authored over 80 pub- patches to a destination. 48) Tire patterns. lications. He has published crosswords in Ro- Down: 1) Speedy competitors. 2) Dr. Reid, manian, in national-level a character from Scrubs. 3) Athlete who gets publications, for almost points. 4) A sound feature. 5) First name given three decades. Copyright c 2019 by the author(s).