Everyday Ethics for Artificial Intelligence © Copyright IBM Corp. 2014, 2015, 2016, 2017, 2018, 2019

IBM, the IBM logo and ibm.com are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/us/en/copytrade.shtml

This document is current as of the initial date of publication and may be changed by IBM at any time. Not all offerings are available in every country in which IBM operates.

The information in this document is provided “as-is” without any warranty, express or implied, including without any warranties of merchantability, fitness for a particular purpose and any warranty or condition of non- infringement. IBM products are warranted according to the terms and conditions of the agreements under which they are provided.

This report is intended for general guidance only. It is not intended to be a substitute for detailed research or the exercise of professional judgment. IBM shall not be responsible for any loss whatsoever sustained by any organization or person who relies on this publication. Acknowledgements Everyday Francesca Rossi AI Ethics Global Leader, Distinguished Research Staff Member, Ethics IBM Research Anna Sekaran for Artificial IBM Program Lead, Cognitive Computing, Academic Initiatives & Public Affairs Intelligence

Jim Spohrer Director of Cognitive OpenTech at IBM

Ryan Caruthers IBM Design Principal, Visual Design

IBM Design Program Office If you have questions, comments or suggestions please email Table of [email protected] to contribute to this effort. Contents

Adam Cutler IBM Distinguished Designer, IBM Design for AI

Milena Pribić Advisory Designer, IBM Design for AI

Lawrence Humphrey Designer, IBM Design for AI Using this Document 06 Introduction 08

Five Areas of Ethical Focus 10 Accountability 14 Value Alignment 20 Explainability 26 Fairness 32 User Data Rights 40

Closing 46 References 48

04 05 “You can use an eraser on the drafting table or a sledgehammer on the construction site.”

– Frank Lloyd Wright

This document represents the So please experiment, play, use, beginning of a conversation and break what you find here defining Everyday Ethics for AI. and send us your feedback. Ethics must be embedded in the design and development process Designers and developers of from the very beginning of AI AI systems are encouraged creation. to be aware of these concepts and seize opportunities to Rather than strive for perfection intentionally put these ideas into first, we’re releasing this to practice. As you work with your allow all who read and use this team and others, please share to comment, critique and par- this guide with them. ticipate in all future iterations.

06 07 This guide provides discussion Introduction points concerning: – Specific virtues that AI systems should possess. – Guidance for designers and developers building and training AI.

Ethical decision-making is not to do, usually in terms of rights, ethical AI systems will ultimately obligations, benefits to society, depend on the industry and use just another form of technical fairness, or specific virtues.1 To case they operate within. We hope create and foster trust between this document serves as a central humans and machines, you must source that helps teams establish problem solving. understand the ethical resources best practices. Designers and and standards available for developers should never work in a As AI designers and developers, we around improving the capabilities reference during the designing, vacuum and must stay in tune with hold a vast share of the collective of an intelligent system doesn’t building, and maintenance of AI. users’ needs and concerns. influence. We are creating systems sufficiently consider human needs. that will impact millions of people. An ethical, human-centric AI must The large-scale focus on AI ethics Constant improvement and assess- Artificial intelligence technology is be designed and developed in a by groups like the IEEE Global ment is key to ensuring that design rapidly growing in capability, impact manner that is aligned with the Initiative on Ethics of Autonomous and development teams address and influence. As designers and values and ethical principles of and Intelligent Systems,2 which users’ concerns. This document developers of AI systems, it is an a society or community it affects. will be referenced throughout this provides teams with a starting imperative to understand the ethical Ethics is based on well-founded document, should be mirrored in point and will surely evolve as AI considerations of our work. A tech- standards of right and wrong that businesses and working groups of capabilities continue to grow. centric focus that solely revolves prescribe what humans ought all sizes. The criteria and metrics for

08 09 It’s our collective responsibility Five Areas of to understand and evolve these ethical focus areas as AI cap- Ethical Focus abilities increase over time. These focal areas provide an intentional framework for establishing an ethical foundation for building and using AI systems.

Mike Monteiro’s direct, unambi- 05 Accountability guous approach in discussing Data Protection and Data Safety: designers’ code of ethics was General Data Protection Regulation Value Alignment influential to this guide.3 Designers and the importance of data in and developers of AI may want building and maintaining AI systems.4 Explainability to delve deeper, exploring topics covered in an IEEE course called In addition, an IBM Research Fairness “Artificial Intelligence and Ethics in team has proposed a Supplier’s Design,” including: Declaration of Conformity (SDoC, or factsheet, for short) to be com- User Data Rights 01 pleted and voluntarily released by Responsible Innovation in the Age AI service developers and providers of AI: Philosophical foundation and to increase the transparency of companies using AI for profits and their services and engender trust societal purpose. in them.5 Like nutrition labels for foods or information sheets 02 for appliances, factsheets for AI The Economic Advantage of Ethical services would provide information Design for Business: Intelligent about the product’s important systems, ethics, and government characteristics. We hope to further policies. develop this proposal alongside our focus areas in order to usher in 03 the era of trusted AI systems and Values by Design in the Algorithmic bootstrapping their broader adoption. Era: Identify, analyze, and practice moral, societal, and legal values.

04 The Nature of Nudging: AI ability to influence people can be used for good or bad.

10 Five Areas of Ethical Focus 11 Running Example

A hotel chain wants to embed artificial intelligence into an in- room virtual assistant/concierge to augment and personalize their users’ stays. We’ll use the project team in charge of this effort as an example throughout the document. This conversational agent will include capabilities such as:

– Agentive-style assistance.

– Introduction to their room and services in their preferred language.

– Control of room facilities through natural language.

– Sending a request directly to the service team through the in-room virtual assistant.

12 Five Areas of Ethical Focus 13 “ Nearly 50% of the surveyed Accountability developers believe that the humans creating AI should be responsible for considering the ramifications of the technology. Not the bosses. Not the middle managers. The coders.”

– Mark Wilson Fast Company6 on Stack Overflow’s Developer Survey Results 20187

AI designers and developers are responsible for considering AI design, development, decision processes, and outcomes.

Human judgment plays a role system’s outcomes. Every person throughout a seemingly objective involved in the creation of AI at any system of logical decisions. It is step is accountable for considering humans who write algorithms, who the system’s impact in the world, define success or failure, who make as are the companies invested in its decisions about the uses of systems development. and who may be affected by a

14 Five Areas of Ethical Focus Accountability 15 Recommended actions to take

01 03 To consider Questions for your team Make company policies clear and Keep detailed records of your design Understand the workings of your How does accountability change accessible to design and develop- processes and decision making. AI even if you’re not personally according to the levels of user ment teams from day one so that Determine a strategy for keeping developing and monitoring influence over an AI system? no one is confused about issues of records during the design and its algorithms. responsibility or accountability. As development process to encourage Is the AI to be embedded in a an AI designer or developer, it is best practices and iteration. Refer to secondary research by human decision-making process, your responsibility to know. sociologists, linguists, behaviorists, is it making decisions on its own, and other professionals to under- or is it a hybrid? stand ethical issues in a holistic 02 04 context. How will our team keep records Understand where the responsibility Adhere to your company’s business of our process? of the company/software ends. conduct guidelines. Also, understand You may not have control over how national and international laws, How do we keep track of ethical data or a tool will be used by a user, regulations, and guidelines8 that your design choices and considerations client, or other external source. AI may have to work within. You can after the launch of the AI? find other related resources in the IEEE Ethically Aligned Design document.9 Will others new to our effort be able to understand our records?

16 Five Areas of Ethical Focus Accountability 17 Accountability Example

– The team utilizes design researchers to contact real guests in the hotels to understand their wants and needs through face-to-face user interviews.

– The team considers their own responsibility when the hotel assistant’s feedback does not meet the needs or expectations of guests. They have implemented a feedback learning loop to better understand preferences and have highlighted the ability for a guest to turn off the AI at any point during their stay.

18 Five Areas of Ethical Focus Accountability 19 “ If machines engage in human Value communities as autonomous agents, then those agents will Alignment be expected to follow the community’s social and moral norms.

A necessary step in enabling machines to do so is to identify these norms. But whose norms?”

– The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems10 AI should be designed to align with the norms and values of your user group in mind.

AI works alongside diverse, human the job of designers and developers interests. People make decisions to collaborate with each other in based on any number of contextual order to ensure consideration of factors, including their experiences, existing values. Care is required memories, upbringing, and cultural to ensure sensitivity to a wide norms. These factors allow us to range of cultural norms and values. have a fundamental understanding As daunting as it may seem to take of “right and wrong” in a wide range value systems into account, the of contexts, at home, in the office, common core of universal principles or elsewhere. This is second nature is that they are a cooperative for humans, as we have a wealth of phenomenon. Successful teams experiences to draw upon. Today’s already understand that cooperation AI systems do not have these types and collaboration leads to the best of experiences to draw upon, so it is outcomes.

20 Five Areas of Ethical Focus Value Alignment 21 Recommended actions to take

01 03 To consider Questions for your team Consider the culture that estab- Consider mapping out your under- If you need somewhere to start, Which group’s values are expressed lishes the value systems you’re standing of your users’ values and consider IBM’s Standards of by our AI and why? designing within. Whenever aligning the AI’s actions accordingly Corporate Responsibility13 or possible, bring in policymakers and with an Ethics Canvas.12 Values will use your company’s standards How do we agree on which values academics that can help your team be specific to certain use cases and documentation. to consider as a team? (For more articulate relevant perspectives. affected communities. Alignment reading on moral alignment, check will allow users to better understand Values are subjective and differ here.14) your AI’s actions and intents. globally. Global companies must take into account language barriers How do we change or adjust the 02 and cultural differences. values reflected by our AI as our Work with design researchers to values evolve over time? understand and reflect your users’ Well-meaning values can create values. You can find out more about unintended consequences. e.g. a this process here.11 tailored political newsfeed provides users with news that aligns with their beliefs but does not holistically represent the gestalt.

22 Five Areas of Ethical Focus Value Alignment 23 Value Alignment Example

– The team understands that for a voice-activated assistant to work properly, it must be “always listening” for a wake word. The team makes it clear to guests that the AI hotel assistant is designed to not keep any data, or monitor guests, in both cases without their knowledge, even if it is listening for a wake word.

– The audio collected while listening for a wake word is auto- deleted every 5 seconds. Even if a guest opts in, the AI does not actively listen in on guests unless it is called upon.

– The team knows that this agent will be used in hotels across the world, which will require different languages and customs. They consult with linguists to ensure the AI will be able to speak in guests’ respective languages and respect applicable customs.

24 Five Areas of Ethical Focus Value Alignment 25 “ IBM supports transparency and Explainability data governance policies that will ensure people understand how an AI system came to a given conclusion or recommendation. Companies must be able to explain what went into their algorithm’s recommendations. If they can’t, then their systems shouldn’t be on the market.”

– Data Responsibility at IBM16

AI should be designed for humans to easily perceive, detect, and understand its decision process.

In general, we don’t blindly trust Explainability is key for users those who can’t explain their interacting with AI to understand reasoning. The same goes for AI, the AI’s conclusions and recom- perhaps even more so.15 As an mendations. Your users should AI increases in capabilities and always be aware that they are achieves a greater range of impact, interacting with an AI. Good design its decision-making process should does not sacrifice transparency in be explainable in terms people can creating a seamless experience. understand. Imperceptible AI is not ethical AI.

26 Five Areas of Ethical Focus Explainability 27 Recommended actions to take

01 03 To consider Questions for your team Allow for questions. A user should When an AI is assisting users Explainability is needed to build How do we build explainability into be able to ask why an AI is doing with making any highly sensitive public confidence in disruptive our experience without detracting what it’s doing on an ongoing basis. decisions, the AI must be able technology, to promote safer from user experience or distracting This should be clear and up front in to provide them with a sufficient practices, and to facilitate broader from the task at hand? the user interface at all times. explanation of recommendations, societal adoption. the data used, and the reasoning Do certain processes or pieces of behind the recommendations. There are situations where users information need to be hidden from may not have access to the full users for security or IP reasons? decision process that an AI might How is this explained to users? 02 04 go through, e.g., financial investment Decision-making processes must Teams should have and maintain algorithms. Which segments of our AI decision be reviewable, especially if the AI access to a record of an AI’s processes can be articulated for is working with highly sensitive decision processes and be amenable Ensure an AI system’s level of users in an easily digestible and personal information data like to verification of those decision transparency is clear. Users should explainable fashion? personally identifiable information, processes. stay generally informed on the AI’s protected health information, and/or intent even when they can’t access biometric data. a breakdown of the AI’s process.

28 Five Areas of Ethical Focus Explainabiity 29 Explainability Example

– Per GDPR (General Data Protection Regulation17), a guest must explicitly opt in to use the hotel room assistant. Additionally, they will be provided with a transparent UI to show how the AI makes its recommendations and suggestions.

– A researcher on the team, through interviews with hotel guests, understands that the guests want a way to opt into having their personal information stored. The team enables a way for the AI to provide guests (through voice or graphic UI) with options and the ability for the system to gather pieces of information with consent.

– With permission, the AI offers recommendations for places to visit during their stay. Guests can ask why these recommendations are made and which set of data is being utilized to make them.

30 Five Areas of Ethical Focus Explainability 31 “ By progressing new ethical Fairness frameworks for AI and thinking critically about the quality of our datasets and how humans perceive and work with AI, we can accelerate the [AI] field in a way that will benefit everyone. IBM believes that [AI] actually holds the keys to mitigating bias out of AI systems – and offers an unprecedented opportunity to shed light on the existing biases we hold as humans.”

– Bias in AI: How we Build Fair AI AI must be designed to Systems and Less-Biased Humans18 minimize bias and promote inclusive representation.

AI provides deeper insight into our to be embedded in the systems we personal lives when interacting create. It is the role of a responsible with our sensitive data. As humans team to minimize algorithmic bias are inherently vulnerable to biases, through ongoing research and data and are responsible for building AI, collection which is representative of there are chances for human bias a diverse population.

32 Five Areas of Ethical Focus Fairness 33 Recommended actions to take

01 03 To consider Questions for your team Real-time analysis of AI brings to Instill a feedback mechanism or Diverse teams help to represent How can we identify and audit un- light both intentional and unin- open dialogue with users to raise a wider variation of experiences intentional biases that we run into tentional biases. When bias in data awareness of user-identified biases to minimize bias. Embrace team during the design and development becomes apparent, the team must or issues. e.g., Woebot asks “Let members of different ages, of our AI? investigate and understand where me know what you think,” after ethnicities, genders, educational it originated and how it can be suggesting a link.19 disciplines, and cultural The status quo changes over time. mitigated. perspectives. How do we instill methods to reflect that change in our ongoing data Your AI may be susceptible to collection? different types of bias based on 02 Design and develop without inten- the type of data it ingests. Monitor How do we best collect feedback tional biases and schedule team training and results in order to from users in order to correct reviews to avoid unintentional quickly respond to issues. Test early unintentional bias in design or biases. Unintentional biases can and often. decision-making? include stereotyping, confirmation bias, and sunk cost bias. (see pages 38 and 39)

34 Five Areas of Ethical Focus Fairness 35 Fairness Example

– After sitting down with members of the hotel’s global management, the team uncovers that diversity and inclusiveness are important elements to the hotel’s values. The team ensures that the data collected about a user’s race, gender, etc. in combination with their usage of the AI, will not be used to market to or exclude certain demographics.

– The team inherited a set of data about guests from the hotel. After analyzing this data and implementing it into a build of the agent, they realize that it has a degree of algorithmic bias from the data. The team proceeds to take the time to train the model further on a bigger, more diverse set of data.

36 Five Areas of Ethical Focus Fairness 37 Unconscious Shortcut Biases Impartiality Biases Self-Interest Biases “ I don’t have the time or energy to “ I know I’m wrong sometimes, but “We contributed the most. They Bias Definitions think about this.” I’m right about this.” weren’t very cooperative.”

The average knowledge worker Availability Bias Anchoring Bias Ingroup / Outgroup Bias is unaware of the many different Overestimating events with To rely too much on one trait or The tendency or patter of favoring types of biases. While this list is not greater “availability“ in memory — piece of information when making members of one’s ingroup over all-encompassing, these biases are influenced by how recent, unusual, decisions (usually the first piece of outgroup members. some of the more common types or emotionally charged the mem- information that we acquire on that to be consciously aware of when ories may be. subject). Sunk Cost Bias designing and developing for AI. The tendency to justify past Base Rate Fallacy Bandwagon Bias choices, even though they no The tendency to ignore general The tendency to do or believe things longer seem valid. information and focus on specific because many other people do information (a certain case). (groupthink). Status Quo Bias The tendency to maintain the Congruence Bias Bias Blind Spot current situation — even when The tendency to test hypotheses The tendency to see oneself as less better alternatives exist. exclusively through direct testing, biased than others, or to be able instead of testing alternative to identify more cognitive biases in Not Invented Here Bias hypotheses. others than in oneself. Aversion to contact with or use of products, research, standards, or Empathy Gap Bias Confirmation Bias knowledge developed outside a The tendency to underestimate the The tendency to search for, inter- group. influence or strength of feelings, in pret, or focus on information in a either ones’ self or others. way that confirms one’s Self-Serving Bias preconceptions. The tendency to focus on Stereotyping strengths/achievements and Expecting a member of a group to Halo Effect overlook faults/failures. To take have certain characteristics without The tendency of an overall im- more responsibility for their having actual information about that pression to influence the observer. group’s work that they give to individual. Positive feelings in one area causes other groups. ambiguous or neutral traits to be viewed positively.

38 Five Areas of Ethical Focus Fairness 39 “ Individuals require mechanisms User Data Rights to help curate their unique identity and personal data in conjunction with policies and practices that make them explicitly aware of consequences resulting from the bundling or resale of their personal information.”

– The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems23

AI must be designed to protect user data and preserve the user’s power over access and uses.

It is your team’s responsibility to share information about them keep users empowered with control without their permission.21 These over their interactions. percentages will rise as AI is further used to either amplify our privacy Pew Research recently found or undermine it. Your company that being in control of our own should be fully compliant with the information is “very important” applicable portions of EU’s General to 74% of Americans.20 The Data Protection Regulation22 and European Commission found any comparable regulations in that 71% of EU citizens find it other countries, to make sure users unacceptable for companies to understand that AI is working in their best interests.22

40 Five Areas of Ethical Focus User Data Rights 41 Recommended actions to take

01 04 To consider Questions for your team Users should always maintain Allow users to deny service or Employ security practices in- What types of sensitive personal control over what data is being used data by having the AI ask for cluding encryption, access control data does the AI utilize and how and in what context. They can deny permission before an interaction methodologies, and proprietary will this data be protected? access to personal data that they or providing the option during consent management modules may find compromising or unfit for an interaction. Privacy settings to restrict access to authorized What contractual agreements are an AI to know or use. and permissions should be clear, users and to de-identify data in necessary for data usage and what findable, and adjustable. accordance with user preferences. are the local and international laws that are applicable to our AI? It is your responsibility to work with your team to address any lack of How do we create the best user 02 05 Users’ data should be protected Forbid use of another company’s these practices. experience with the minimum from theft, misuse, or data data without permission when amount of required user data? corruption. creating a new AI service.

03 06 Provide full disclosure on how the Recognize and adhere to applicable personal information is being used national and international rights or shared. laws when designing for an AI’s acceptable user data access permissions.24

42 Five Areas of Ethical Focus User Data Rights 43 User Data Rights Example

– The hotel provides guests with a consent agreement to utilize the AI hotel assistant before they begin using the AI’s services. This agreement clearly outlines to guests that the hotel does not own their data and the guests have the right to purge this data from the system at any time, even after checkout.

– During user interviews, the design researchers find that the guests feel they should be provided with a summary of the information that was acquired from them during their stay. At checkout, they can instruct the hotel to remove this information from the system if they wish.

44 Five Areas of Ethical Focus User Data Rights 45 Closing

Designers and developers of AI can help mitigate bias and disenfranchisement by practicing within these five areas of ethical considerations.

AI systems must remain flexible for all of us, individuals and groups enough to undergo constant will need to further define criteria maintenance and improvement as and metrics for evaluation to ethical challenges are discovered better allow for the detection and and remediated. mitigation of any issues.

By adopting and practicing This is an ongoing project: we the five focal areas covered in welcome and encourage feedback this document, designers and so the guide can develop and developers can become more mature over time. We hope it ethically aware, mitigate biases contributes to the dialogue and within these systems, and instill debate about the implications of responsibility and accountability these technologies for humanity in those who work with AI. As and allows designers and much of what we do related to developers to embed ethics into artificial intelligence is new territory the AI solutions they work on.

46 47 References

01 www.scu.edu/ethics/ethics-resources/ethical-decision-making/what-is-ethics/ 02 ethicsinaction.ieee.org/ 03 https://muledesign.com/2017/07/a-designers-code-of-ethics 04 https://forms1.ieee.org/AI--Ethics-in-Design-Program.html?LT=EA_LG_IAW 05 arxiv.org/abs/1808.07261 06 www.fastcodesign.com/90164226/what-developers-really-think-about-ai-and-bias 07 insights.stackoverflow.com/survey/2018/ 08 www.ohchr.org/EN/pages/home.aspx 09 ethicsinaction.ieee.org 10 ethicsinaction.ieee.org 11 www.ibm.com/design/research/ 12 www.ethicscanvas.org 13 https://www.ibm.com/ibm/responsibility/2017/ 14 faculty.mtsac.edu/cmcgruder/moraljudgements.html 15 www.ibm.com/watson/advantage-reports/future-of-artificial-intelligence/building-trust-in-ai.html 16 www.ibm.com/blogs/policy/dataresponsibility-at-ibm/ 17 martechtoday.com/guide/gdpr-the-general-data-protection-regulation 18 www.ibm.com/blogs/policy/bias-in-ai/ 19 woebot.io 20 www.pewresearch.org/fact-tank/2016/09/21/the-state-of-privacy-in-america/ 21 ibm.biz/eucprivacy 22 www.eugdpr.org/ 23 ethicsinaction.ieee.org/ 24 www.ohchr.org/EN/ProfessionalInterest/Pages/InternationalLaw.aspx

48 IBM Design for AI Download the latest version here: ibm.biz/everydayethics