Demystifying AI: “A” is for Augmented
Betsy Greytok, IBM Senior Counsel April 12, 2018 “Artificial Intelligence”
Inigo Montoya, The Princess Bride You can’t be what you can’t see. Ginni Rometty, Chairman, President and Chief Executive Officer of IBM
Preferred Terminology:
• Augmented Intelligence: Augment human intelligence by reviewing more resources faster. Intelligence moving at the speed of data.
• Cognitive computing is the simulation of human thought processes; it doesn’t replace human thought. To build expertise, people follow certain steps
Study Apprentice
Practice Master Cognitive computing follows a similar path
Study Ingest
Apprentice Learn Practice Test Master Experience Cognitive computing brings together cutting edge areas of computer science
Natural language Machine learning processing
Question answering High performance technology computing
Knowledge representation Unstructured information and reasoning management Visual Recognition Answering questions is not about searching for documents
Search engine vs. IBM Watson Has question Asks a question
Distills to 2-3 keywords Understands question
Finds documents containing keywords Produces possible answers and evidence
Delivers documents based on popularity Analyzes evidence
Reads documents Computes confidence
Finds answers Delivers response, evidence and confidence
Finds and analyzes evidence Considers answers and evidence These cognitive systems are neither autonomous nor sentient, but they form a new kind of intelligence that has nothing artificial about it. They augment our capacity to understand what is happening in the complex world around us.
Leaders of every industry and institution are sprinting to become digital, adopting digital products, operations and business models. But once everything becomes digital, who will win?
The answer is clear: It will be the companies and the products that make the best use of data. Data is the great new natural resource of our time, and cognitive systems are the only way to get value from all its volume, variety and velocity. Having ingested a fair amount of data myself, I offer this rule of thumb: If it’s digital today, it will be cognitive tomorrow.
Ginni Rometty, Wall Street Journal, 10/18/16 8
Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights
Executive Office of the President May 2016
1
Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights
Contents PREFACE ...... 4 The Assumption: Big Data is Objective ...... 6 Challenge 1: Inputs to an Algorithm ...... 7 Challenge 2: The Design of Algorithmic Systems and Machine Learning ...... 8 CASE STUDIES IN THE USE OF BIG DATA ...... 10 Big Data and Access to Credit ...... 11 The Problem: Many Americans lack access to affordable credit due to thin or non-existent credit files. .. 11 The Big Data Opportunity: Use of big data in lending can increase access to credit for the financially underserved...... 11 The Big Data Challenge: Expanding access to affordable credit while preserving consumer rights that protect against discrimination in credit eligibility decisions...... 12 Big Data and Employment ...... 13 The Problem: Traditional hiring practices may unnecessarily filter out applicants whose skills match the job opening...... 13 The Big Data Opportunity: Big data can be used to uncover or possibly reduce employment discrimination...... 14 The Big Data Challenge: Promoting fairness, ethics, and mechanisms for mitigating discrimination in employment opportunity...... 15 Big Data and Higher Education ...... 16 The Problem: Students often face challenges accessing higher education, finding information to help choose the right college, and staying enrolled...... 16 The Big Data Opportunity: Using big data can increase educational opportunities for the students who most need them...... 17 The Big Data Challenge: Administrators must be careful to address the possibility of discrimination in higher education admissions decisions...... 18 Big Data and Criminal Justice ...... 19 The Problem: In a rapidly evolving world, law enforcement officials are looking for smart ways to use new technologies to increase community safety and trust...... 19
2
The Big Data Opportunity: Data and algorithms can potentially help law enforcement become more transparent, effective, and efficient...... 19 The Big Data Challenge: The law enforcement community can use new technologies to enhance trust and public safety in the community, especially through measures that promote transparency and accountability and mitigate risks of disparities in treatment and outcomes based on individual characteristics...... 21 LOOKING TO THE FUTURE ...... 22
3
PREFACE
Big data and associated technologies have enormous potential for positive impact in the United States, from augmenting the sophistication of online interactions to enhancing understanding of climate change to making advances in healthcare. These efforts, as well as the technological trends of always-on networked devices, ubiquitous data collection, cheap storage, sensors, and computing power, will spur broader use of big data. Our challenge is to support growth in the beneficial use of big data while ensuring that it does not create unintended discriminatory consequences.
The Obama Administration’s Big Data Working Group released reports on May 1, 20141 and February 5, 2015.2 These reports surveyed the use of data in the public and private sectors and analyzed opportunities for technological innovation as well as privacy challenges. One important social justice concern the 2014 report highlighted was “the potential of encoding discrimination in automated decisions”—that is, that discrimination may “be the inadvertent outcome of the way big data technologies are structured and used.” Building on these prior reports and the 2014 study conducted by the President’s Council of Advisors on Science and Technology (PCAST), the Administration is further examining how big data is used in the public and private sectors.3 Specifically, we are examining case studies involving credit, employment, education, and criminal justice to shed light on how using big data to expand opportunity has the potential to introduce bias inadvertently that could affect individuals or groups. As discussed in a report released by the Federal Trade Commission (FTC) earlier this year, big data provides opportunities for innovations that reduce discrimination and promote fairness and opportunity, including expanding access to credit in low-income communities, removing subconscious human bias from hiring decisions and classrooms, and providing extra resources to at-risk students.4 However, the FTC also emphasized the need to prevent such technologies from being used to deny low-income communities credit, perpetuate long-standing biases in employment, or exclude underserved communities from other benefits and opportunities.5
This report examines several case studies from the spheres of credit and lending, hiring and employment, higher education, and criminal justice to provide snapshots of opportunities and dangers, as well as ways that government policies can work to harness the power of big data and avoid discriminatory outcomes. These are issues that strike at the heart of American values, which we must work to advance in the face of emerging, innovative technologies.
CECILIA MUÑOZ MEGAN SMITH DJ PATIL Director U.S. Chief Technology Officer Deputy Chief Technology Domestic Policy Council Office of Science and Officer for Data Policy and Technology Policy Chief Data Scientist Office of Science and Technology Policy 4
THE BIG DATA REVIEW AND RECENT PROGRESS
Civil rights legislation of the last century responded to the reality that some Americans were being denied access to fundamental building blocks of opportunity and security, such as employment, housing, access to financial products, and admission to universities, on the basis of race, color, national origin, religion, sex, gender, sexual orientation, disability, or family status. Today, in the United States, anti-discrimination laws help enforce the tenet that all people are to be treated equally. These safeguards are important to protect all Americans against discrimination. Big data techniques have the potential to enhance our ability to detect and prevent discriminatory harm. But, if these technologies are not implemented with care, they can also perpetuate, exacerbate, or mask harmful discrimination. 6 In May 2014, a federal working group led by then-Counselor to the President John Podesta released a report on the emerging role of big data in our changing world and the opportunities and challenges that it presents. 7 This was followed by an update on February 5, 2015.8 These reports highlight how big data tools are already being used to improve lives, make the economy work better, and save taxpayer dollars. In our increasingly networked world, the building blocks of big data are everywhere. We upload messages and photos over social media to stay connected to our friends; our phones transmit our specific locations to transportation apps; and information about who we are and what we are interested in is collected by a wide variety of retail, advertising, and analytics companies. Supplying data to these services enables a greater degree of improvement and customization, but this sharing also creates opportunities for additional uses of our data that may be unexpected, invasive, or discriminatory. As data-driven services become increasingly ubiquitous, and as we come to depend on them more and more, we must address concerns about intentional or implicit biases that may emerge from both the data and the algorithms used as well as the impact they may have on the user and society. Questions of transparency arise when companies, institutions, and organizations use algorithmic systems and automated processes to inform decisions that affect our lives, such as whether or not we qualify for credit or employment opportunities, or which financial, employment and housing advertisements we see. 9 Ideally, data systems will contribute to removing inappropriate human bias where it has previously existed. We must pay ongoing and careful attention to ensure that the use of big data does not contribute to systematically disadvantaging certain groups. To avoid exacerbating biases by encoding them into technological systems, we need to develop a principle of “equal opportunity by design”—designing data systems that promote fairness and safeguard against
5
discrimination from the first step of the engineering process and continuing throughout their lifespan. The 2014 report focused on the emerging role of big data in our changing world and the opportunities and challenges posed by such technology.10 The report made a series of recommendations to the Federal government, among them to advance several pieces of key legislation that would further individual privacy in the digital economy. The Administration, Congress, and the public have made progress in acting on these recommendations; in 2015 we issued an Interim Progress Report illustrating the steps that have been taken thus far.11 Further, the Council of Economic Advisers issued a report on price discrimination, exploring the processes that companies use to harvest information to in essence charge different consumers different prices for products and services rendered, and the Federal Trade Commission (FTC) included price discrimination in its analysis of issues relating to big data.12 The purpose of this report is to advance our understanding of the problems that we can address with big data and algorithmic systems, along with the challenges that exist in deploying them. This new set of practices has great potential to increase access to opportunity and help overcome discrimination, if fairness in computational systems and ethical approaches to data analytics are the norm. At the same time, there are great risks that the very same innovations could perpetuate discrimination and unequal access to opportunity as the use of data expands. This report examines instances where big data methods and systems are being used in the public and private sectors in order to illustrate the potential for positive and negative outcomes and the extent to which “equal opportunity by design” safeguards may help address harms. These examples are meant to be a snapshot of the problems that data analytics can help to solve and the potential issues that its use might create, rather than an exhaustive look or set of recommendations on avoiding discrimination as big data becomes more central to the work of government and business.
OPPORTUNITIES AND CHALLENGES IN BIG DATA The Assumption: Big Data is Objective
It is often assumed that big data techniques are unbiased because of the scale of the data and because the techniques are implemented through algorithmic systems. However, it is a mistake to assume they are objective simply because they are data-driven.13 The challenges of promoting fairness and overcoming the discriminatory effects of data can be grouped into the following two categories:
1) Challenges relating to data used as inputs to an algorithm; and 6
2) Challenges related to the inner workings of the algorithm itself.
Challenge 1: Inputs to an Algorithm
The algorithmic systems of big data employ sophisticated processes. These processes need inputs. Consider how you might use a smart phone or GPS device to get the fastest route to a particular destination. To begin, the device needs at least three inputs: (1) where you are, (2) where you want to go, and (3) a map of the area. It then employs algorithms to calculate the fastest route from (1) to (2) and generate directions for how to proceed. At its most basic, this algorithmic system might simply draw upon map data for its inputs, such as the names, locations, and length of the roads in a given city. To compute the fastest route, the algorithmic system would then calculate which path on the map involved the fewest number of roads with the shortest road length. A more advanced routing system could include maximum speed limits for each road or even information about traffic congestion such as data on the speed of other drivers collected from those drivers’ mobile devices. For these more complex algorithmic systems, the inputs extend beyond a simple street map to potentially include many others, such as weather updates, historical traffic patterns, and the presence of disruptive events occurring nearby. The decision to use certain inputs and not others can result in discriminatory outputs. Some of the technical themes that can cause discriminatory outputs include: Poorly selected data, where the designers of the algorithmic system decide that certain data are important to the decision but not others. In the “fastest route” example, the architect of the system might only include information about roads but not public transportation schedules or bike routes, thereby disadvantaging individuals who do not own a vehicle. Such issues can be regarded as qualitative errors, where human choices in the selection of certain datasets as algorithmic inputs over others are ill-advised. Careless choices of input might lead to biased results—in the “fastest route” example, results that might favor routes for cars, discourage use of public transport, and create transit deserts. Relatedly, designers might select data that is of too much or too little granularity, resulting in potentially discriminatory effects.
Incomplete, incorrect, or outdated data, where there may be a lack of technical rigor and comprehensiveness to data collection, or where inaccuracies or gaps may exist in the data collected. In the “fastest route” example, this could occur if, for instance, the algorithmic system does not update bus or train schedules regularly. Even if the system works perfectly in other respects, the resulting directions could again discourage use of 7
public transportation and disadvantage those who have no viable alternatives, such as many lower-income commuters and residents.
Selection bias, where the set of data inputs to a model is not representative of a population and thereby results in conclusions that could favor certain groups over others. In the “fastest route” example, if speed data is collected only from the individuals that own smartphones, then the system’s results may be more accurate for wealthier populations with higher concentrations of smart phones and less accurate in poorer areas where smart-phone concentrations are lower.14
Unintentional perpetuation and promotion of historical biases, where a feedback loop causes bias in inputs or results of the past to replicate itself in the outputs of an algorithmic system. For instance, when companies emphasize “hiring for culture fit” in their employment practices, they may inadvertently perpetuate past hiring patterns if their current workplace culture is primarily based on a specific and narrow set of experiences. In a workplace populated primarily with young white men, for example, an algorithmic system designed primarily to hire for culture fit (without taking into account other hiring goals, such as diversity of experience and perspective) might disproportionally recommend hiring more white men because they score best on fitting in with the culture.
Each of these issues is critical to take into account in designing systems to deliver services effectively, fairly, and ethically to consumers and community members, or to influence processes like credit-granting, hiring, housing allocation, and admissions.15 Transparency, accountability, and due process mechanisms are important components of ensuring that the inputs to an algorithmic system are accurate and appropriate.
Challenge 2: The Design of Algorithmic Systems and Machine Learning
For those who are not directly involved in the technical development of algorithms for large scale data systems, the end product of such as system can feel like a “black box”—an opaque machine that takes inputs, carries out some inscrutable process, and delivers unexplained outputs based on that process.16 The technical processes involved in algorithmic systems are typically unknown to a consumer, potential student, job candidate, defendant, or the public as they are often treated as confidential or proprietary to the entities that use them.17 Some systems even “passively” pre-screen potential candidates without notice as a preemptive effort to streamline decision-making processes at a later date.18 This lack of transparency means that 8
affected individuals—such as those who receive word that they will not receive a job offer, were denied admission to their college of choice, or will be denied a line of credit or lease— have limited ability to learn the reasons why such decisions were made and limited ability to detect and seek correction of any errors or bias if they do occur. It may even mean that certain individuals will be entirely excluded from certain opportunities—for instance, seeing particular advertisements for jobs, financial products, or educational opportunities and never discover that they were denied these opportunities.19 Such situations can be complex and difficult to address, especially if the outputs are relied upon again in subsequent determinations.20 At a minimum, it is important to encourage transparency, accountability, and due process mechanisms wherever possible in the use of big data. Without these safeguards, hard-to-detect flaws could proliferate. Such flaws include: Poorly designed matching systems, which are intended to help find information, resources, or services. For example, search engines, social media platforms, and applications rely on matching systems to determine search results, what advertisements to display, and which businesses to recommend. These matching systems may result in discriminatory outcomes if the system designs are not kept current or do not take into account historical biases or blind spots within the data or the algorithms used.21 Personalization and recommendation services that narrow instead of expand user options, where detailed information about individual users might be collected and analyzed to infer their preferences, interests, and beliefs in order to point them to opportunities such as new music to download, videos to watch, price discounts, or products to purchase. Academic studies have shown that the algorithms used to recommend such content may inadvertently restrict the flow of information to certain groups, leaving them without the same opportunities for economic access and inclusion as others.22 Decision-making systems that assume correlation necessarily implies causation, whereby a programmer or the algorithmic system itself may assume that because two factors frequently occur together (e.g., having a certain income level and being of a particular ethnicity), there is necessarily a causal relationship between the two. Assuming a causal relationship in these circumstances can lead to discrimination. Data sets that lack information or disproportionately represent certain populations, resulting in skewed algorithmic systems that effectively encode discrimination because of the flawed nature of the initial inputs. Data availability, access to technology, and participation in the digital ecosystem vary considerably, due to economic, linguistic, structural or socioeconomic barriers, among others. Unaddressed, this systemic flaw can
9
reinforce existing patterns of discrimination by over-representing some populations and under representing others. An additional area that presents challenges for further study is a genre of computer science known as machine learning—the “science of getting computers to act without being explicitly programmed.”23 Complex and often inscrutable even at times to their programmers, machine learning models are starting to be used in areas such as credit offers, entrepreneurial funding, or hiring. As these methods continue to advance, it may become more difficult to explain or account for the decisions machines make through this process unless mechanisms are built into their designs to ensure accountability.24 Using the principle of “equal opportunity by design” and grounding engineering with sound ethical and professional best practices will also help mitigate discriminatory results over time and increase inclusion. Just as in other areas, programmers and data scientists may inadvertently or unconsciously design, train, or deploy big data systems with biases. Therefore, an important factor in implementing the “equal opportunity by design” principle is engaging with the field of “bias mitigation” to avoid building in the designers’ biases that are an inevitable product of their own culture and experiences in life.25 Research-based methods are emerging that can help reduce biases in decision-making around hiring, promotions, classroom grading, funding, social engagement, and more.26 Use of these methods can help stop biased big data systems from becoming the norm, instead of the exception. As improvements in the uses of big data and machine learning continue, it will remain important not to place too much reliance on these new systems without questioning and continuously testing the inputs and mechanics behind them and the results they produce. “Data fundamentalism”—the belief that numbers cannot lie and always represent objective truth—can present serious and obfuscated bias problems that negatively impact people’s lives.27 As we work to address challenges related to data inputs and the inner workings of algorithms, we must also pay attention to how to the products of these algorithmic systems are used, with an eye for ensuring that information about places, people, preferences, and more is used legally, ethically, and to advance democratic principles, such as equality and opportunity.
CASE STUDIES IN THE USE OF BIG DATA
In this section, we present four case studies in the use of big data analytics: (1) access to credit, (2) higher education, (3) employment, and (4) criminal justice. We describe the opportunities each case study presents for algorithmic systems to support personal, commercial, and
10
organizational missions, as well as the challenges that arise in utilizing the data without adverse impacts.
Big Data and Access to Credit
The Problem: Many Americans lack access to affordable credit due to thin or non- existent credit files.
Access to fairly-priced and affordable credit is an important factor in enabling Americans to thrive economically, especially those working to enter the middle class. For decades, lenders have largely relied on credit scores, such as the FICO score, to decide whether and on what terms to make a loan. Credit scores represent a prediction of the likelihood that someone will have a negative financial event, such as defaulting on a loan, within a specific period of time. Traditionally, this prediction is made based on actual data about a particular person’s credit history, and turned into a score using algorithms developed from past lending experiences. While traditional credit scores serve many Americans well, according to a study by the Consumer Financial Protection Bureau (CFPB), as many as 11 percent of consumers are “credit invisible”—they simply do not have enough up-to-date credit repayment history for the algorithm to produce a credit score.28 In addition, the CFPB found a strong relationship between income and a scorable credit record—30 percent of consumers in low-income neighborhoods are “credit invisible” and the credit records of another 15 percent are unscorable.29 According to the CFPB, African-Americans and Latinos are more likely to be credit invisible, at rates of around 15 percent in comparison to 9 percent for whites.30 The CFPB also found that an additional 13 percent of African-Americans and 12 percent of Latinos are unscorable, compared to 7 percent for whites.31
The Big Data Opportunity: Use of big data in lending can increase access to credit for the financially underserved.
One possible approach to this problem is to use data analytics drawing on multiple sources of information to create more opportunity for consumers to gain access to better credit. As companies collect information and score individuals, especially those without sufficient or updated credit information, data may be useful in assessing credit risk. Some companies look at previously untapped data, such as phone bills, public records, previous addresses, educational background, and tax records, while others may consider less conventional sources, such as location data derived from use of cellphones, information gleaned from social media platforms,
11
purchasing preferences via online shopping histories, and even the speeds at which applicants scroll through personal finance websites. New tools designed with big data have potential to create alternative scoring mechanisms and new opportunities for access to credit for the tens of millions of Americans who do not have enough information in their credit files to receive a traditional credit score or who have an undeservedly low score. For example, many Americans regularly pay their phone and utility bills—payment records that predict creditworthiness. But phone and utility payment information is generally only reported as part of a credit history if a customer falls far behind, and therefore it only ever serves to penalize a consumer’s perceived creditworthiness. One study by the Policy and Economic Research Council looked at more than four million credit files and found that if both positive and negative utility and telecom payments were included, over 70 percent of the unscorable files would become scorable and 64 percent of the “thin files” (files with very little other credit history) would see improved scores.32 The study also found that this change especially benefits low-income borrowers.33
The Big Data Challenge: Expanding access to affordable credit while preserving consumer rights that protect against discrimination in credit eligibility decisions.
While big data has the ability to increase American’s access to affordable credit, if not used with care, it also has the potential to perpetuate, exacerbate, or mask discrimination. For example, consider a technology that would glean information from an individual’s social media connections and use social analytic systems to create an alternative credit score. While such a tool might expand access to credit for those underserved by the traditional market, it could also function to reinforce disparities that already exist among those whose social networks are, like them, largely disconnected from everyday lending.34 Such tools could also raise questions about the ability of consumers to dispute an adverse decision or to correct inaccurate information. When such decisions are made within computationally-driven 'black box' systems, traditional notions of transparency may fail to fully capture and disclose the information consumers need to understand the basis of such decisions and the role that various data played in determining their credit eligibility. The right to be informed about and to dispute the accuracy of the underlying data used to create a credit score is particularly important because credit bureaus have significant data accuracy issues, which are likely to be exacerbated by the use of new, fast-changing data sources. An FTC study found that 21 percent of its sample of consumers had a confirmed error on at least one of their three credit bureau reports.35 Expanding the data sources for credit scoring systems from long-collected items like collections notices and credit card payments to fast-changing, large-volume data like social media usage and GPS location information would 12
most likely increase the presence of factually inaccurate data, leading to scores that are not based on a consumers’ likelihood of delinquency.36 Additionally, as the number of data sources increases, the relationship to creditworthiness becomes more complex and dynamic, and therefore consumers may have more difficulty interpreting the notices required under FRCA and ECOA and identifying problems.37 Consumers with less experience dealing with large institutions or complex data products may be particularly vulnerable to these data accuracy and transparency challenges. These concerns are not necessarily unique to the emerging data-analytics approaches to credit scoring. For example, in 2007 the FTC released a study of credit-based insurances scores finding that although there are substantial score disparities among ethnic groups, the scores are effective predictors of risk under automobile-insurance policies and are not simply proxies for race, ethnicity, sex, or other prohibited bases.38 As algorithms develop to measure creditworthiness in new ways, it will be critical to design and test them with similar concerns in mind and to guard against unintentionally using information that is a proxy for race, gender, or other protected characteristics .39 The limited research that does publicly exist has looked at whether the scores are effective predictors of delinquency, it has not examined whether new ways of evaluating credit worthiness adequately avoid considering proxies for traits like race or ethnicity.40 The shortage of studies on these new scoring products is a potential cause for concern because of the complexity and proprietary nature of these new products. If poorly implemented, algorithmic systems that utilize new scoring products to connect targeted marketing of credit opportunities with individual credit determinations could produce discriminatory harms. This is particularly concerning because the rapid pace of evolution in the credit sector, especially combined with ongoing advances in data science, makes it difficult for researchers and consumers alike to identify discrimination and take steps to prevent it.41
Big Data and Employment
The Problem: Traditional hiring practices may unnecessarily filter out applicants whose skills match the job opening.
Beginning in the 1990s, a growing number of companies realized there was a new way to access and analyze a larger pool of applicants for a job opening rather than simply reviewing paper files.42 Resume-database websites provided a place where individuals and companies could gain access to opportunities and talent. To deal with the sudden influx of candidates, companies looking to hire also turned to new ways of rating applicants, using analytical tools to 13
automatically sort and identify the preferred candidates to move forward in a hiring process. With this change, the task of identifying and scoring applicants began to shift from industrial psychologists and recruiting specialists to computer scientists, through the use of algorithms and large data sets.43 Yet even as recruiting and hiring managers look to make greater use of algorithmic systems and automation, the inclination remains for individuals to hire someone similar to themselves, an unconscious phenomenon often referred to as “like me” bias, which can impede diversity.44 Algorithmic systems can be designed to help prevent this bias and increase diversity in the hiring process. Yet despite these goals, because they are built by humans and rely on imperfect data, these algorithmic systems may also be based on flawed judgments and assumptions that perpetuate bias as well. Because these technologies are new, rapidly changing, difficult to decipher, and often subject to proprietary protections, their determinations can be even more difficult to challenge.
The Big Data Opportunity: Big data can be used to uncover or possibly reduce employment discrimination.
Just as with credit scoring, data analytics can be beneficial to the workplace in helping match people with the right jobs. As discussed above, research has documented a “like me bias” or “affinity bias” in hiring; even well-intentioned hiring managers often choose candidates with whom they share characteristics.45 By contrast, algorithmically-driven processes have the potential to avoid individual biases and identify candidates who possess the skills that fit the particular job.46 Companies can use data-driven approaches to find potential employees who otherwise might have been overlooked based on traditional educational or workplace-experience requirements. Data-analytics systems allow companies to objectively consider experiences and skill sets that have a proven correlation with success. By looking at the skills that have made previous employees successful, a human-resources data system can “pattern match” in order to recognize the characteristics the next generation of hires should have.47 When fairness, ethics, and opportunity are a core part of the original design, large-scale data systems can help combat the implicit and explicit bias often seen in traditional hiring practices that can lead to problematic discrimination.48 Beyond hiring decisions, properly deployed, advanced algorithmic systems present the possibility of tackling age-old employment discrimination challenges, such as the wage gap or occupational segregation.49
14
The Big Data Challenge: Promoting fairness, ethics, and mechanisms for mitigating discrimination in employment opportunity.
Data-analytics companies are creating new kinds of “candidate scores” by using diverse and novel sources of information on job candidates. These sources, and the algorithms used to develop them, sometimes use factors that could closely align with race or other protected characteristics, or may be unreliable in predicting success of an individual at a job. For example, workers who were unemployed for long periods during the recent economic downturn may have a harder time re-entering the workforce because candidate-scoring systems that consider “length of time since last job” can generate scores that send negative signals to potential employers that are unrelated to job performance. Similarly, one employment research firm found commuting distance to be one of the strongest predictors of how long a customer service employee will stay with a job.50 If algorithmic systems were trained to rely heavily on this factor without further consideration, they could end up discriminating against the candidates who, while otherwise qualified, happen to live in areas that are further away from the job than other candidates. While the factor of commuting distance was ultimately disregarded in this particular study out of concern for how highly it might correlate with race,51 other employers might overlook such important factors. Other common hiring criteria, such as credit-worthiness (also the work of algorithms) and criminal records, compromise the validity of these tools if they inaccurately or inadequately reflect an individual’s qualifications. Implementation of such systems with an eye to their broader effects on fairness and equal opportunity is therefore essential. Finally, as described earlier, machine-learning algorithms can help determine what kinds of employees are likely to be successful by reviewing the past performance of existing employees or by analyzing the preferences of hiring managers as shown by their past decisions.52 But if those sources themselves contain historical biases, the scores may well replicate those same biases. For example, if machine-learning algorithms emphasize the age that a candidate became interested in computing compared to their peers, cultural messages and assumptions that associate computing with boys more often than with girls could promote environments where more boys than girls are exposed to computers at an earlier age, thereby skewing later hiring patterns toward more male hires, even though a company’s hiring goals may be focused on gender equality. 53 Similar concerns could emerge regarding age discrimination, since older workers may be less likely to have grown up with home computers. Further, hiring algorithms that emphasize the need for a four-year college degree, or even a particular field of study or degree can leave out highly qualified, talented individuals who might not have those specific qualifications and could instead come into the job opportunity through on-the-job training or emerging alternative training and apprenticeship models—or who might have a four-year degree but in a different field than the ones sought by the algorithmic systems. These are the 15
types of factors that engineers and managers need to consider at the inception of designing analytical hiring systems and incorporate into the machine learning design work. “Equal opportunity by design” approaches are one way to promote these considerations. Companies have begun to filter their applicant pools for job openings using various human resources analytics platforms. It is critical to the fairness of American workplaces that all companies continue to promote fairness and ethical approaches to the use of data tools and ensure against the perpetuation of biases that could disfavor certain groups. Businesses also stand to benefit, because those that do not look beyond historical hiring patterns (even as mediated by an algorithm) will miss great candidates for important jobs.
Big Data and Higher Education
The Problem: Students often face challenges accessing higher education, finding information to help choose the right college, and staying enrolled.
Prospective students and their families must grapple with assessing which of the many institutions of higher education available will best prepare them to achieve their goals. The decisions to pursue a degree, at which degree level, and at which institution all have a lasting impact on students and their futures. For example, obtaining a bachelor’s degree can increase total earnings by 84 percent over a lifetime relative to expected earnings for someone with a high school diploma.54 Similarly, differences in the price of attendance across institutions affect financial returns, and may lead to differences in the amount that students have to borrow, which may also affect their career decisions and personal lives in meaningful ways. Despite the importance of this decision, there is a surprising lack of clear, easy to use, and accessible information available to guide the students making these choices. At the same time, institutions of higher education collect and analyze tremendous amounts of data about their students and applicants. Before students arrive on campus, colleges use student information in recruitment, admissions, and financial aid decisions. After students enroll, some schools are using the data collected through the application process and in the classroom to tailor their students’ educational experiences.55 The opportunities to use big data in higher education can either produce or prevent discrimination—the same technology that can help identify and serve students who are more likely to be in need of extra help can also be used to deny admissions or other opportunities based on the very same characteristics.
16
The Big Data Opportunity: Using big data can increase educational opportunities for the students who most need them.
To address the lack of information about college quality and costs, the Administration has created a new College Scorecard to provide reliable information about college performance.56 The College Scorecard is a large step toward helping students and their families evaluate college choices. Never-before-released national data about post-college outcomes—including the most comparable and reliable data on the earnings of colleges’ alumni and new data on student debt—and student-loan repayment provides students, families, and their advisers with a more accurate picture of college cost and value. The data also encourages colleges to strengthen support that helps students persist in and complete college, and to provide increased opportunities for disadvantaged students to get a college education. Armed with this information, students and their families can make more informed decisions and better understand the opportunities and tradeoffs of their choices. In addition to data that the Department of Education has made available through the College Scorecard, institutions of higher education also present a unique environment to utilize innovations in big data for students once enrolled. Schools can use data they are already collecting to help track student progress. The opportunity for innovation lies in how schools use that existing data to create a tailored learning experience. Big data techniques are already helping students learn more effectively through tailored instruction, which can help overcome continuing disparities in learning outcomes, and providing extra help for those more likely to drop out or fail.57 Georgia State University is one example of a college using big data to drive student success. Its Graduation and Progression Success (GPS) Advising program, which started in 2013, is designed to keep the school’s more than 32,000 students on track for graduation. It tracks eight hundred different risk factors for each student on a daily basis. When a problem is detected, the university deploys proactive advising and timely interventions to provide the support that students need. At times the interventions are as simple—and essential—as ensuring the student has registered for the right courses; at other times, the system uses predictive analytics to make sure that the student's performance in a prerequisite course makes success likely at the next level. Since the GPS Advising initiative began in 2013, there have been nearly 100,000 proactive interventions with Georgia State students based on the analytics-based alerts coming from the system. Over the past three years, Georgia State’s graduation rate has increased by 6 percentage points, from 48 percent to 54 percent, when compared to the baseline year. The biggest gains have been enjoyed by at-risk populations. This year for the first time in Georgia State's history, first-generation, black, and Latino students as well as those on federally-funded Pell grants all graduated at rates at or above that of the student body overall, and Georgia State
17
now awards more Bachelor's degrees to black students than any non-profit college or university in the United States. Over the last two-years alone, Georgia State has reduced time-to-degree by an average of half a semester per student, saving students more than $10 million in tuition and fees.58 The Big Data Challenge: Administrators must be careful to address the possibility of discrimination in higher education admissions decisions.
Though data can help high school students choose the right college, there are several challenges involved in accurately estimating the extent to which the specific school a student attends makes causal contributions to student success. One important limitation of Federal data sources is the lack of individual student-level data indicating academic preparation for college, such as their high-school GPA or college admissions test scores (e.g., SAT or ACT scores). Since academic preparation is an important element that adds context to measures of college quality, omitting this variable may bias estimates of college quality. As the College Scorecard continues to be refined and developed, these are challenges that the Department of Education will continue to face. In making admissions decisions, institutions of higher education may use big data techniques to try to predict the likelihood that an applicant will graduate before they ever set foot on campus.59 Using these types of data practices, some students could face barriers to admission because they are statistically less likely to graduate. Institutions could also deny students from low-income families, or other students who face unique challenges in graduating, the financial support that they deserve or need to afford college. This, in turn, creates a concern that as schools rush to cut costs, some applicants might face greater barriers to admission if they are considered unworthy of the extra resources it would take to keep them enrolled.60 One significant predictor of whether or not a student will graduate from college is family income, and the use of big data in this case may discriminate against students from lower-income families.61 The same data used to help students succeed can also be used to discourage low- income students from enrolling, and while there may be ways to mitigate this, such as financial incentives, especially at less selective institutions, it remains a cautionary example of using data to perpetuate discrimination. On the other hand, some schools and states are actively using data to promote access and success, and to prevent discrimination. For example, the State of Tennessee’s outcomes-based funding formula for four-year institutions offers an illustration of how data can promote both student success and access.62 Tennessee’s model places extra value on the “credit accumulation” and “degree attainment” outcomes of both students eligible for Pell grant funding and adult students (those over the age of 24).63 In particular, these outcomes are valued 40 percent more than the same outcomes for non-Pell grant eligible traditional-age 18
students.64 By doing this, institutions have an incentive to enroll and promote the success of lower-income and adult students, who are traditionally under-represented in higher education.65 There are valuable opportunities for the use of big data in higher education, but they must be implemented with care. As learning itself is a process of trial and error, it is particularly important to use data in a manner that allows the benefits of those innovations, but still allows a safe space for students to explore, make mistakes, and learn without concern that there will be long term consequences for errors that are part of the learning process.
Big Data and Criminal Justice
The Problem: In a rapidly evolving world, law enforcement officials are looking for smart ways to use new technologies to increase community safety and trust.
Local, state, and federal law enforcement agencies are increasingly drawing on data analytics and algorithmic systems to further their mission of protecting America. Using information gathered from the field and through the use of new technologies, law enforcement officials are analyzing situations in order to determine the appropriate response. At the same time, law enforcement agencies are expected to be accountable at all times to the communities they serve and will continue to be so in the digital age. Similarly, the technologies that assist law enforcement’s decisions and actions should also be accountable to ensure they are used in a thoughtful manner that considers the impact on communities and promotes successful community partnerships built on trust.
The Big Data Opportunity: Data and algorithms can potentially help law enforcement become more transparent, effective, and efficient.
Law enforcement agencies have long attempted to identify patterns in criminal activity in order to allocate scarce resources more efficiently. New technologies are replacing manual techniques, and many police departments now use sophisticated computer modeling systems to refine their understanding of crime hot spots, linking offense data to patterns in temperature, time of day, proximity to other structures and facilities, and other variables. The President’s Task Force on 21st Century Policing recommended, among many other steps, that law enforcement agencies adopt model policies and best practices for technology-based engagement that increases community trust and access; work toward national standards on the issue of technology’s impact on privacy concerns; and develop best practices that can be 19
adopted by state legislative bodies to govern the acquisition, use, retention, and dissemination of auditory, visual, and biometric data by law enforcement.66 Since the Task Force released its recommendations, the White House and the Department of Justice have been engaged in several initiatives to ensure that the report’s recommendations are put into practice across the United States. As part of these efforts, the White House launched the Police Data Initiative to make policing data more transparent and improve community trust.67 More than 50 police departments throughout the nation have joined in this work to realize the benefits of better technology. Commitments from participating jurisdictions include: increased use of open policing data to build community trust and increase departmental transparency, and use of data to more effectively identify policies that could be improved or officers who may contribute to adverse public interactions so they can be linked with effective training and interventions.68 Consistent with these goals, several police departments in the United States have developed and deployed “early warning systems” to identify officers who may benefit from additional training, resources, or counseling to prevent excessive uses of force, citizen complaints and other problems.69 Using de-identified police data, as well as contextual data about local crime and demographics, these systems are designed to detect the factors most indicative of future problems by attempting to determine behavioral patterns that predict a higher risk of future adverse incidents. Detecting these patterns opens new opportunities to develop targeted interventions for officers to protect their safety and improve police/community interactions. Separately, some of the newest analytical modeling techniques, often called “predictive policing,” might provide greater precision in predicting locations and times at which criminal activity is likely to occur. Research demonstrates that a neighborhood that has recently been victimized by one or more burglaries is likely to be targeted for additional property crimes in the coming days. An analytical method known as “near-repeat modeling” attempts to predict crimes based on this insight.70 Similarly, a technique known as “risk terrain modeling” can identify specific locations where criminal activity often clusters, such as bars, motels or convenience stores, and can predict the specific social and physical factors that attract would- be offenders and create conditions ripe for criminal activity.71 Current Los Angeles Police Department (LAPD) Chief of Police, Charlie Beck, described predictive policing as enabling “directed, information-based patrol; rapid response supported by fact-based prepositioning of assets; and proactive, intelligence-based tactics, strategy, and policy.”72 In some instances these systems have shown significant promise. In experiments conducted by the LAPD’s Foothill Division in which large sets of policing data were analyzed to predict occurrences of crime, the Division experienced a larger reduction in reported crime than any other division in the Department.73
20
The Big Data Challenge: The law enforcement community can use new technologies to enhance trust and public safety in the community, especially through measures that promote transparency and accountability and mitigate risks of disparities in treatment and outcomes based on individual characteristics.
When designed and deployed carefully, data-based methodologies can help law enforcement make decisions based on factors and variables that empirically correlate with risk, rather than on flawed human instincts and prejudices. However, it is important that data and algorithmic systems not be used in ways that exacerbate unwarranted disparities in the criminal justice system. For example, unadjusted data could entrench rather than ameliorate documented racial disparities where they already exist, such as in traffic stops and drug arrest rates.74 Those leading efforts to use data analytics to create and implement predictive tools must work hard to ensure that such algorithms are not dependent on factors that disproportionately single out particular communities based on characteristics such as race, religion, income level, education, or other data inputs that may serve as proxies for characteristics with little or no bearing on an individual’s likelihood of association with criminal activity. For instance, when historical information is used with predictive algorithms to direct patrols, prior arrest data could be used to advise beat officers to patrol certain areas with greater frequency or intensity. If feedback loops are not thoughtfully constructed, a predictive algorithmic system built in this manner could perpetuate policing practices that are not sufficiently attuned to community needs and potentially impede efforts to improve community trust and safety. For example, machine learning systems that take into account past arrests could indicate that certain communities require more policing and oversight, when in fact the communities may be changing for the better over time. Moving forward, law enforcement agencies could work to account for these issues: transparency and accountability on data input and processes, a focus on eliminating data that could serve as proxies for race or poverty, and ensuring that bias is not replicated through these tools are key steps. It is also important to note that criminal justice data is notoriously poor.75 This is in part because one of the major data repositories, the Federal Bureau of Investigation’s Uniform Crime Report (UCR), is in need of modernization and relies on voluntary contributions that often do not capture data with the degree of richness and completeness needed for in-depth analysis. FBI Director James Comey has prioritized improving data collection and repeatedly called on communities across the United States to increase participation in the UCR’s National Incident-Based Reporting System (NIBRS), explaining that this method of collecting data enhances our understanding of crime because “[it] doesn’t just include statistics. It gives the full picture–the circumstances and the context involving each incident. It asks: What happened?
21
Where did it happen? What time did it occur? Who was there and what is their demographic information? What is the relationship between the perpetrator and the victim?”76 Even if crime reporting is improved, there will remain reasons to approach any crime dataset with care and caution. Many criminal-justice data inputs are inherently subjective. Officers use discretion in enforcement decisions (e.g., deciding whom to stop, search, question, and arrest) just as police officers and prosecutors use discretion in charging (e.g., simple assault vs. felonious assault). The underlying data reflects these judgement calls. Policymakers should also continue to look for ways to better use the increasing amount of data that law enforcement agencies now have in order to improve public safety and accountability. For example, the number of agencies presently using body cameras is expanding exponentially. With this expansion comes thousands of hours of video and audio. As part of the Police Data Initiative, the White House has engaged with academics and technologists to determine if there is a machine-readable way to review this video and audio to identify both beneficial and problematic interactions between law enforcement and civilian community members. On December 8, 2015, the White House Office of Science and Technology Policy participated in a workshop co-hosted by Stanford University and the City of Oakland Police Department focused on accelerating research and development to make body-worn camera data more searchable and interoperable with other systems, and on automating processes to reduce reporting burdens. More broadly, the conversation about ways to effectively use predictive analytics in law enforcement should continue, building on the work that has already begun between key stakeholders—ranging from law enforcement agencies to academics, community leaders and civil society groups.
LOOKING TO THE FUTURE
The use of big data can create great value for the American people, but as these technologies expand in reach throughout society, we must uphold our fundamental values so these systems are neither destructive nor opportunity limiting. Moving forward, it is essential that the public and private sectors continue to have collaborative conversations about how to achieve the most out of big data technologies while deliberately applying these tools to avoid—and when appropriate, address—discrimination. In order to ensure growth in the use of data analytics is matched with equal innovation to protect the rights of Americans, it will be important to: Support research into mitigating algorithmic discrimination, building systems that support fairness and accountability, and developing strong data ethics frameworks. The Networking and Information Technology Research and Development Program and 22
the National Science Foundation (NSF) are developing research strategy proposals that will incorporate these elements and encourage researchers to continue to look at these issues. Through its support of the Council for Big Data, Ethics, and Society, as well as other efforts, NSF will continue to work with scientific, technical, and academic leaders to encourage the inclusion of data ethics within both research projects and student coursework and to develop interdisciplinary frameworks to help researchers, practitioners, and the public understand the complex issues surrounding big data, including discrimination, disparate impact, and associated issues of transparency and accountability. In particular, it will be important to bring together computer scientists, social scientists, and those studying the humanities in order to understand these issues in their historical, social, and technological contexts.77 Encourage market participants to design the best algorithmic systems, including transparency and accountability mechanisms such as the ability for subjects to correct inaccurate data and appeal algorithmic-based decisions. Big data technologies can support the success of public and private institutions, but to do so, they must be implemented in a responsible and ethical manner. Organizations, institutions, and companies should be held accountable for the decisions they make with the aid of computerized decision-making systems and technology. The FTC’s recent big data report included considerations for companies using big data techniques, discussed potentially applicable laws, and suggested questions for legal compliance.78 Private companies using data analytics to expand opportunity should take these considerations into account in order to ensure that they treat consumers, students, job candidates, and the public fairly. Both private and public entities should also consider improved methods of providing individuals and communities with the means to access and correct their data, as well as better ways of providing notice about how their information is being used to inform decisions, such as those described in the case studies of this report. Recognizing the research issues outlined, experts from the data science and social science communities, among others, should continue to develop additional best practices for fair and ethical use of big data techniques and machine learning in the public and private sectors. Promote academic research and industry development of algorithmic auditing and external testing of big data systems to ensure that people are being treated fairly. One way these issues can be tackled is through the emerging field of algorithmic systems accountability, where stakeholders and designers of technology “investigate normatively significant instances of discrimination involving computer algorithms” and use nascent tools and approaches to proactively avoid discrimination through the use of new technologies employing research-based behavior science. 79 These efforts should 23
also include an analysis identifying the constituent elements of transparency and accountability to better inform the ethical and policy considerations of big data technologies. There are other promising avenues for research and development that could address fairness and discrimination in algorithmic systems, such as those that would enable the design of machine learning systems that constrain disparate impact or construction of algorithms that incorporate fairness properties into their design and execution. Broaden participation in computer science and data science, including opportunities to improve basic fluencies and capabilities of all Americans. Consistent with the goals of the President’s Computer Science for All and TechHire initiatives, educational institutions and employers can strive to broaden participation in these fields.80 In particular, they should look for ways to provide more Americans with opportunities to have greater fluency and awareness of how these issues impact them and to influence how these fields evolve going forward. Consider the roles of the government and private sector in setting the rules of the road for how data is used. As use of big data moves from new and novel to mainstream, the private sector, citizens, institutions, and the public sector are establishing expectations, norms, and standards that will serve as guides for the future. How big data is used ethically to reduce discrimination and advance opportunity, fairness, and inclusion should inform the development of both private sector standards and public policy making in this space.
24
1 The White House. “Big Data: Seizing Opportunities, Preserving Values.” May 2014. https://www.whitehouse.gov/sites/default/files/docs/big_data_privacy_report_may_1_2014.pdf. 2 The White House. “Big Data: Seizing Opportunities and Preserving Values: Interim Progress Report.” February 2015. https://www.whitehouse.gov/sites/default/files/docs/20150204_Big_Data_Seizing_Opportunities_Preserving_Val ues_Memo.pdf. 3 Executive Office of the President. President’s Council of Advisors on Science and Technology. “Report to the President: Big Data and Privacy: A Technological Perspective.” May 2014. https://www.whitehouse.gov/sites/default/files/microsites/ostp/PCAST/pcast_big_data_and_privacy_- _may_2014.pdf. 4 The Federal Trade Commission. “Big Data: A Tool for Inclusion or Exclusion? Understanding the Issues.” January 2016. https://www.ftc.gov/system/files/documents/reports/big-data-tool-inclusion-or-exclusion-understanding- issues/160106big-data-rpt.pdf. 5 Ibid. 6 This report uses the term “discrimination” in a very broad sense to refer to outsized harmful impacts—whether intended or otherwise—that the design, implementation, and utilization of algorithmic systems can have on discrete communities and other groups that share certain characteristics. This report does not intend to reach a legal conclusion about discrimination as defined under federal or other law. Some instances of discrimination may be unintentional and even unforeseen. Others may be the result of a deliberate policy decision to concentrate services or assistance on those who are most in need. Still others may create adverse consequences for particular populations that create or exacerbate inequality of opportunity for those already at a disadvantage. We discuss all of these types of discrimination throughout the report—our aim being to draw attention to the fact that, as big data algorithmic systems play an increasing role in shaping our experiences and environment, those who design and implement them must be increasingly thoughtful about the potential for discriminatory effects that can have long-term or systemic consequences. 7 The White House. “Big Data: Seizing Opportunities, Preserving Values.” May 2014. https://www.whitehouse.gov/sites/default/files/docs/big_data_privacy_report_may_1_2014.pdf 8 The White House. “Big Data: Seizing Opportunities and Preserving Values: Interim Progress Report.” February 2015. https://www.whitehouse.gov/sites/default/files/docs/20150204_Big_Data_Seizing_Opportunities_Preserving_Val ues_Memo.pdf. 9 Ibid. “Digital Ad Revenues Surge 19% to $27.5 Billion in First Half of 2015." IAB.com. 2015. January 27, 2016. http://www.iab.com/news/digital-ad-revenues-surge-19-climbing-to-27-5-billion-in-first-half-of-2015-according- to-iab-internet-advertising-revenue-report/. 10 The White House. “Big Data: Seizing Opportunities and Preserving Values: Interim Progress Report.” February 2015. https://www.whitehouse.gov/sites/default/files/docs/20150204_Big_Data_Seizing_Opportunities_Preserving_Val ues_Memo.pdf 11 Ibid. 12 The White House Council of Economic Advisers. "Big Data and Differential Pricing." February 2015. https://www.whitehouse.gov/sites/default/files/whitehouse_files/docs/Big_Data_Report_Nonembargo_v2.pdf; The Federal Trade Commission. “Big Data: A Tool for Inclusion or Exclusion?: Understanding the Issues.” January 2016. https://www.ftc.gov/system/files/documents/reports/big-data-tool-inclusion-or-exclusion- understanding-issues/160106big-data-rpt.pdf. 13 Crawford, K. “The Hidden Biases of Big Data.” Harvard Business Review. April 1, 2013, https://hbr.org/2013/04/the-hidden-biases-in-big-data. 14 “U.S. Smartphone Use in 2015.” Pew Research Center. March 31, 2015. http://www.pewinternet.org/2015/04/01/u-s-smartphone-use-in-2015/pi_2015-04-01_smartphones_07/.
25
15 Citron, D., and Pasquale, F. "The Scored Society: Due Process for Automated Predictions." 89 Washington L. Rev. 1, 2014. https://digital.law.washington.edu/dspace-law/bitstream/handle/1773.1/1318/89WLR0001.pdf?sequence=1. 16 Pasquale, F. The Black Box Society: The Secret Algorithms That Control Money and Information. Harvard University Press. 2015. 17 Dwork, C. and Mulligan, D. "It's Not Privacy, and It's Not Fair." 66 Stan. L. Rev. Online 35. September 3, 2013. http://www.stanfordlawreview.org/online/privacy-and-big-data/its-not-privacy-and-its-not-fair. 18 Taylor, F. “Hiring in the Digital Age: What’s Next for Business Recruiting?” Business News Daily. January 11, 2016. http://www.businessnewsdaily.com/6975-future-of-recruiting.html. 19 The Federal Trade Commission. “Big Data: A Tool for Inclusion or Exclusion?: Understanding the Issues.” January 2016. https://www.ftc.gov/system/files/documents/reports/big-data-tool-inclusion-or-exclusion-understanding- issues/160106big-data-rpt.pdf. 20 Kraemer, F., van Overveld, K., and Peterson, M. 2011. “Is There an Ethics of Algorithms?” Ethics and Information Technology 13 (3): 251–60. 21 Miller, C. “When Algorithms Discriminate.” The New York Times. July 9, 2015. http://www.nytimes.com/2015/07/10/upshot/when-algorithms-discriminate.html. 22 Sweeney, L. “Discrimination in Online Ad Delivery”. Harvard University. January 28, 2013 http://ssrn.com/abstract=2208240 or http://dx.doi.org/10.2139/ssrn.2208240; Datta, A., Tschantz, M. C., and Datta, A. “Automated experiments on ad privacy settings: A tale of opacity, choice, and discrimination.” In Proceedings on Privacy Enhancing Technologies (PoPETs). 2015. https://www.andrew.cmu.edu/user/danupam/dtd-pets15.pdf. 23 Ng, A. Machine Learning course. Stanford University. 2016. https://www.coursera.org/learn/machine-learning. 24 Citron, D. “Big Data Should Be Regulated by ‘Technological Due Process’.” The New York Times. August 6, 2014. http://www.nytimes.com/roomfordebate/2014/08/06/is-big-data-spreading-inequality/big-data-should-be- regulated-by-technological-due-process. 25 Nelson, B. “The Data on Diversity.” Communications of the ACM. Vol 57. No 11, Pages 86-95. http://cacm.acm.org/magazines/2014/11/179827-the-data-on-diversity/fulltext; Welle, B. Smith, M. “Time to Raise the Profile of Women and Minorities in Science.” Scientific American. October 1, 2014. http://www.scientificamerican.com/article/time-to-raise-the-profile-of-women-and-minorities-in-science/. 26 Self W., Mitchell G., Mellers B., Tetlock P., and Hildreth J. “Balancing Fairness and Efficiency: The Impact of Identity-Blind and Identity-Conscious Accountability on Applicant Screening.” PLoS ONE. December 14, 2015. http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0145208. 27 Hardy, Q. “Why Data is Not the Truth.” The New York Times, June 1, 2013. http://bits.blogs.nytimes.com/2013/06/01/why-big-data-is-not-truth/. 28 The Consumer Financial Protection Bureau Office of Research. “Data Point: Credit Invisibles.” May 2015. http://files.consumerfinance.gov/f/201505_cfpb_data-point-credit-invisibles.pdf. 29 Ibid. 30 Ibid. 31 Ibid. 32 Turner, M. Walker, P., Chaudhuri, S., and Varghese, R. “A New Pathway to Financial Inclusion: Alternative Data, Credit Building, and Responsible Lending in the Wake of the Great Recession.” Policy and Economic Research Council. 2012. http://www.perc.net/wp-content/uploads/2013/09/WEB-file-ADI5-layout1.pdf 33 Ibid. 34 Wei, Y., Yildirim, P., Van den Bulte, C. and Dellarocas, C., “Credit Scoring with Social Network Data.” Marketing Science. 2015. http://pubsonline.informs.org/doi/abs/10.1287/mksc.2015.0949. 35 The Federal Trade Commission. “Big Data: A Tool for Inclusion or Exclusion? Understanding the Issues.” January 2016. https://www.ftc.gov/system/files/documents/reports/big-data-tool-inclusion-or-exclusion-understanding- issues/160106big-data-rpt.pdf. 36 Javelin Strategy and Research. “Evaluating the Viability of Alternative Credit Decisioning Tools.” LexisNexis. May 2013. http://www.lexisnexis.com/risk/downloads/whitepaper/alternative-credit-decisioning.pdf.
26
37 Talbot, D. "Data Discrimination Means the Poor May Experience a Different Internet." MIT Technology Review. October 9, 2013. http://www.technologyreview.com/news/520131/data-discrimination-means-the-poor-may- experience-a-different-internet/. 38 The Federal Trade Commission. “Credit-Based Insurance Scores: Impacts on Consumers of Automobile Insurance: A Report to Congress.” July 2007. https://www.ftc.gov/sites/default/files/documents/reports/credit- based-insurance-scores-impacts-consumers-automobile-insurance-report-congress-federal- trade/p044804facta_report_credit-based_insurance_scores.pdf; The Federal Reserve Board. “Report to the Congress on Credit Scoring and Its Effects on the Availability and Affordability of Credit.” August 2007. http://www.federalreserve.gov/boarddocs/rptcongress/creditscore/. 39 Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R. “Fairness Through Awareness.” Proceedings of the 3rd ACM Innovations in Theoretical Computer Science Conference, 214-226. 2012. http://arxiv.org/pdf/1104.3913.pdf. 40 The Federal Trade Commission. “Credit-Based Insurance Scores: Impacts on Consumers of Automobile Insurance: A Report to Congress.” July 2007. https://www.ftc.gov/sites/default/files/documents/reports/credit- based-insurance-scores-impacts-consumers-automobile-insurance-report-congress-federal- trade/p044804facta_report_credit-based_insurance_scores.pdf; The Federal Reserve Board. “Report to the Congress on Credit Scoring and Its Effects on the Availability and Affordability of Credit.” August 2007. http://www.federalreserve.gov/boarddocs/rptcongress/creditscore/. 41 Robinson, D., and Yu, H. “Knowing the Score: New Data, Underwriting, and Marketing in the Consumer Credit Marketplace. A Guide for Financial Inclusion Stakeholders.” October 2014. https://www.teamupturn.com/static/files/Knowing_the_Score_Oct_2014_v1_1.pdf. 42 Capelli, P. “Making the Most of Online Recruiting.” Harvard Business Review. March 2001. https://hbr.org/2001/03/making-the-most-of-on-line-recruiting. 43 Ibid. 44 Goldberg, C. “Relational Demography and Similarity-Attraction in Interview Assessments and Subsequent Offer Decisions.” Group and Organization Management 30.6: 597-624. 2005. http://gom.sagepub.com/content/30/6/597.short; Davidson, J. “Feds urged to fight ‘unconscious bias’ in hiring and promotions.” The Washington Post, April 14, 2016. https://www.washingtonpost.com/news/powerpost/wp/2016/04/14/feds-urged-to-fight-unconscious-bias-in- hiring-and-promotions/. 45 Goldberg, C. “Relational Demography and Similarity-Attraction in Interview Assessments and Subsequent Offer Decisions.” Group and Organization Management 30.6: 597-624, 2005. http://gom.sagepub.com/content/30/6/597.short. 46 "Robots Are Color Blind: How Big Data Is Removing Biases from the Hiring Process." Michaelhousman.com. http://michaelhousman.com/robots-are-color-blind/. 47 “Why Machines Discrimination and How to Fix Them.” Science Friday. November 20, 2015. http://www.sciencefriday.com/segments/why-machines-discriminate-and-how-to-fix-them/. 48 The Federal Trade Commission. “Big Data: A Tool for Inclusion or Exclusion? Understanding the Issues.” January 2016. https://www.ftc.gov/reports/big-data-tool-inclusion-or-exclusion-understanding-issues-ftc-report. 49 U.S. Equal Employment Opportunity Commission. Notice of Proposed Changes to the EEO-1 to Collect Pay Data from Certain Employers. https://www.eeoc.gov/employers/eeo1survey/2016_eeo-1_proposed_changes_qa.cfm 50 “Robot Recruiters: Big Data and Hiring.” The Economist. April 6, 2013. http://www.economist.com/news/business/21575820-how-software-helps-firms-hire-workers-more-efficiently- robot-recruiters. 51 Peck, D. “They're Watching You at Work.” The Atlantic. December 2013. http://www.theatlantic.com/magazine/archive/2013/12/theyre-watching-you-at-work/354681/. 52 Barocas, S. and Selbst, A. “Big Data’s Disparate Impact.” 104 Calif. L. Rev. 2016. http://ssrn.com/abstract=2477899. 53 “Why Machines Discrimination and How to Fix Them.” Science Friday. November 20, 2015. http://www.sciencefriday.com/segments/why-machines-discriminate-and-how-to-fix-them/.
27
54 Carnevale, A., Rose, S., and Cheah, B. “The College Payoff.” Georgetown Center on Education and the Workforce. August 5, 2011. https://cew.georgetown.edu/report/the-college-payoff/. 55 The Federal Trade Commission. “Big Data: A Tool for Inclusion or Exclusion? Understanding the Issues.” January 2016. https://www.ftc.gov/reports/big-data-tool-inclusion-or-exclusion-understanding-issues-ftc-report. 56 U.S. Department of Education. College Scorecard. https://collegescorecard.ed.gov/. 57 Nelson, L. “Big Data 101.” Vox. July 14, 2014. http://www.vox.com/2014/7/14/5890403/colleges-are-hoping- predictive-analytics-can-fix-their-graduation-rates. 58 Kurweil, M. and Wu, D. “Building a Pathway to Student Success at Georgia State University.” Ithaka S&R Case Study. 2015. http://www.sr.ithaka.org/publications/building-a-pathway-to-student-success-at-georgia-state- university/; Marcus, J. “Colleges Use Data to Predict Grades and Graduation.” The Hechinger Report. December 10, 2014. http://hechingerreport.org/like-retailers-tracking-trends-colleges-use-data-predict-grades-graduations/. 59 Ungerleider, N. “Colleges Are Using Big Data To Predict Which Students Will Do Well—Before They Accept Them.” Fast Company, October 21, 2013. http://www.fastcoexist.com/3019859/futurist-forum/colleges-are-using- big-data-to-predict-which-students-will-do-well-before-the. 60 Nelson, L. “Big Data 101.” Vox. July 14, 2014. http://www.vox.com/2014/7/14/5890403/colleges-are-hoping- predictive-analytics-can-fix-their-graduation-rates. 61 Ibid. 62 Tennessee Higher Education Commission. Outcomes Based Funding Formula Resources. https://www.tn.gov/thec/topic/funding-formula-resources. 63 Ibid. 64 Tennessee Higher Education Commission. 2015-20 Outcomes Based Funding Formula Overview. https://www.tn.gov/thec/article/2015-20-funding-formula. 65 Ness, E., Deupree, M., and Gándara M. “Campus Responses to Outcomes-Based Funding in Tennessee: Robust, Aligned, and Contested.” https://www.tn.gov/assets/entities/thec/attachments/FordFoundationPaper.pdf. 66 The U.S. Department of Justice’s Office of Community Oriented Policing Services. “Final Report of the President’s Task Force on 21st Century Policing.” May 2015. http://www.cops.usdoj.gov/pdf/taskforce/taskforce_finalreport.pdf. 67 “The Police Data Initiative: A 5-Month Update.” October 27, 2015. https://www.whitehouse.gov/blog/2015/10/27/police-data-initiative-5-month-update; White House Police Data Initiative Highlights New Commitments. April 21, 2016. https://www.whitehouse.gov/the-press- office/2016/04/22/fact-sheet-white-house-police-data-initiative-highlights-new-commitments. 68 Ibid. 69 Abdollah, T. “‘Early Warning Systems’ aim to ID Troubled Policy Officers.” Los Angeles Daily News. September 7, 2014. http://www.dailynews.com/government-and-politics/20140907/early-warning-systems-aim-to-id-troubled- police-officers. 70 Haberman, C. and Ratcliffe, J. “The Predictive Policing Challenges of Near Repeat Armed Street Robberies.” Policing, Vol 6, No. 2, 151-166. May 2, 2012. http://www.jratcliffe.net/wp-content/uploads/Haberman-Ratcliffe- 2012-The-predictive-policing-challenges-of-near-repeat-armed-street-robberies.pdf. 71 Kennedy, L., Caplan, J., and Piza, E. “Risk Clusters, Hotspots, and Spatial Intelligence: Risk Terrain Modeling as an Algorithm for Policy Resource Allocation Strategies.” Journal of Quantitative Criminology. September 2010. https://www.researchgate.net/profile/Eric_Piza/publication/226431177_Risk_Clusters_Hotspots_and_Spatial_Inte lligence_Risk_Terrain_Modeling_as_an_Algorithm_for_Police_Resource_Allocation_Strategies/links/543827f90cf2 4a6ddb92a4b9.pdf. 72 Beck, C. and McCue, C. “Predictive Policing: What Can We Learn from Wal-Mart and Amazon about Fighting Crime in a Recession?” Police Chief, Vol. 76, No. 11. November 2009. http://www.policechiefmagazine.org/magazine/index.cfm?fuseaction=display_arch&article_id=1942&issue_id=11 2009. 73 Friend, Z. “Predictive Policing: Using Technology to Reduce Crime.” FBI Law Enforcement Bulletin. October 4, 2013. https://leb.fbi.gov/2013/april/predictive-policing-using-technology-to-reduce-crime.
28
74 The U.S. Department of Justice, Office of Justice Programs, National Institute of Justice. “Racial Profiling and Traffic Stops.” January 10, 2013. http://www.nij.gov/topics/law-enforcement/legitimacy/pages/traffic-stops.aspx; Department of Justice, Bureau of Justice Statistics. “Racial Disparity in US Drug Arrests.” October 1, 1995. http://www.bjs.gov/content/pub/pdf/rdusda.pdf. 75 The U.S. Department of Justice, Bureau of Justice Statistics. “Bridging Gaps in Police Crime Data.” September 1999. http://www.bjs.gov/content/pub/pdf/bgpcd.pdf. 76 Comey, J. “Uniform Crime Reporting Program Message from the Director.” Fall 2015. https://www.fbi.gov/about-us/cjis/ucr/crime-in-the-u.s/2014/crime-in-the-u.s.-2014/resource-pages/message- from-director. 77 “Big Data, Big Questions.” International Journal of Communication, Vol. 8. 2014. http://ijoc.org/index.php/ijoc/article/view/2169/1162. 78 The Federal Trade Commission. “Big Data: A Tool for Inclusion or Exclusion? Understanding the Issues.” January 2016. https://www.ftc.gov/reports/big-data-tool-inclusion-or-exclusion-understanding-issues-ftc-report. 79 Sandvig, C., Hamilton, K., Karahalios, K., and Langbort, C. “Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms.” Center for People and Infrastructures, May 22, 2014 http://www-personal.umich.edu/~csandvig/research/Auditing%20Algorithms%20--%20Sandvig%20-- %20ICA%202014%20Data%20and%20Discrimination%20Preconference.pdf; Pope, D. and Syndor, J. “Implementing Anti-Discrimination Policies in Statistical Profiling Models.” American Economic Journal: Economic Policy, Vol. 3, No. 3, pp. 206-231. 2011. http://faculty.chicagobooth.edu/devin.pope/research/pdf/website_antidiscrimination%20models.pdf. 80 The White House. Computer Science for All. January 30, 2016. https://www.whitehouse.gov/blog/2016/01/30/computer-science-all; The White House. Tech Hire Initiative. https://www.whitehouse.gov/issues/technology/techhire.
29
3/5/2018 Can AI make your company more diverse? Learn from Pymetrics | SmartRecruiters
FUTURE OF WORK Can AI make your company more diverse? Learn from Pymetrics
Alex Cresswell January 30, 2018 4 min read
There are many stories detailing a future where algorithms dictate our decisions. Some of this fear is justified. But what if algorithms also present a powerful solution for removing our biases?
We all know that human bias exists in the hiring process and that this bias disadvantages women and minorities. Recent Facebook, Google and Uber lawsuits belie the consequences of this reality. Technology, though, is neither good or bad; it only knows what we teach it. So we shouldn’t be surprised that unchecked technology – created and trained by inherently biased humans – o en perpetuates our biases.
The good news is that AI is actually much more powerful than technologies we’ve created in the past.
With humans, it’s virtually impossible to truly remove our inherent biases. We have attempted to improve this through unconscious bias training, and while this e ort is well-intentioned, it has https://www.smartrecruiters.com/blog/can-ai-make-your-company-more-diverse-learn-from-pymetrics/ 1/11 3/5/2018 Can AI make your company more diverse? Learn from Pymetrics | SmartRecruiters been shown to be patently ine ective. While removing human bias is close to impossible, removing it from an algorithm is not only possible but already a reality. We can engineer and train algorithms to be bias-free. We can ensure that they treat individuals from every background equally.
How is this possible? Because the output of algorithms can be trained to be replicable and pre- tested to be bias-free. Humans, on the other hand, are both unpredictable and impossible to pre- test.
The potential for AI to promote equality is profound.
Yet all we hear about is AI creating evil robot overlords that instigate dystopian futures full of inequity. Why are we so focused on portraying AI as ubiquitously negative? Look around – humanity is doing a fantastic job of perpetuating inequality without robot assistance. Actually, we could really use some AI help!
AI can be an incredibly powerful force for good. Instead of viewing AI as a dystopian overlord, think of it as a bias-reversal mechanism that can undo the powerful biases introduced by even the most well-intentioned humans. This makes it one of our most powerful tools for promoting diversity, equality, and inclusion and we are starting to see this tool being put to good use. Applied, Pymetrics, Textio, Talent Sonar are all examples of platforms using technology to reduce bias. Each of these platforms has tangibly improved diversity outcomes for companies, helping them achieve record levels of diversity across gender, ethnicity and socioeconomic status.
Pymetrics co-founder Dr. Frida Polli explains, “We started the company to bring technology to the challenge of matching people to jobs, and so take human bias out of the process”. Dr. Polli had been struck by the fact that people use great matching technology to decide what to listen to next on Spotify, or what to buy next on Amazon, and yet the technology to decide on a company’s most important asset – its people – hadn’t evolved for decades.
As Managing Director for EMEA I can tell you that Pymetrics tests all our algorithms for accuracy and bias before they ever go near a real candidate, and adjust them to ensure we are passing through all groups equally. In this way, we have helped a major financial services company move their gender balance from 25 percent to 50 percent, and Unilever has achieved record highs for diversity across gender, ethnicity and socioeconomic status in their graduate program. These results are not at the expense of talent prediction; Pymetrics clients are also claiming substantial improvements in retention and performance, and all the while the process is much more e icient meaning that candidates get feedback much more quickly, and clients have a shorter time to hire and lower costs to do so.
Blind auditions for orchestras moved the needle in female hiring from 5 percent to 25 percent.
https://www.smartrecruiters.com/blog/can-ai-make-your-company-more-diverse-learn-from-pymetrics/ 2/11 3/5/2018 Can AI make your company more diverse? Learn from Pymetrics | SmartRecruiters We currently stand at 5 percent for many female roles – women CEOs, both Fortune 500 and startups, CTOs, engineers and a whole host of other male-dominated fields. What we need is the “blind audition” for the workplace.
AI can be scary, and when le unchecked it can definitely perpetuate existing disparities. However, if we are intentional about how we design it, AI can be our ultimate tool to create the kind of blindness that lets us see more clearly.
AI hiring bias hr technology inherent bias
Alex Cresswell
VIEW ALL POSTS
The TA Leader’s Top 5 Reasons to Attend Top 5 Sessions at Hiring Success 18 Hiring Success 18
You may also like
FUTURE OF WORK Hire18 Speaker Interview: Lou Adler
Kaya Payseno
https://www.smartrecruiters.com/blog/can-ai-make-your-company-more-diverse-learn-from-pymetrics/ 3/11 3/5/2018 Can AI make your company more diverse? Learn from Pymetrics | SmartRecruiters
FUTURE OF WORK PRODUCT TIPS 10 Signs Your Applicant Tracking System is Stuck in the 90’s
Jason Buss
FUTURE OF WORK Hire18 Sponsor: Wepow
SmartRecruiters
FUTURE OF WORK Hire18 Speaker Interview: Kathryn Minshew
Kaya Payseno
https://www.smartrecruiters.com/blog/can-ai-make-your-company-more-diverse-learn-from-pymetrics/ 4/11 3/5/2018 Can AI make your company more diverse? Learn from Pymetrics | SmartRecruiters
FUTURE OF WORK Hiring Success 18 Sets the Agenda
SmartRecruiters
FUTURE OF WORK Hire18 Sponsor: Codefights
SmartRecruiters
Get the latest Journal content sent to your inbox.
Email Address
Daily Weekly
SUBSCRIBE https://www.smartrecruiters.com/blog/can-ai-make-your-company-more-diverse-learn-from-pymetrics/ 5/11 3/5/2018 Can AI make your company more diverse? Learn from Pymetrics | SmartRecruiters
Latest Posts
Recruiting Startup of the Year Nominee: Joonko
Hire18 Speaker Interview: Lou Adler
10 Signs Your Applicant Tracking System is Stuck in the 90’s
Hire18 Sponsor: Wepow
Hire18 Speaker Interview: Kathryn Minshew
Tags
candidate experience candidates
conference customer
customer story employer brand
employer branding Facebook
give labor day hire18 Hiring
hiring success hiring success 18
how to How to hire HR
hr tech hr technology
human resources interview
interviewing interview tips
job boards leadership
LinkedIn recruiting resume
RSOTY smartprofile
smartrecruiters smartup
Social Media
Social Media Recruiting
social networks Social Recruiting https://www.smartrecruiters.com/blog/can-ai-make-your-company-more-diverse-learn-from-pymetrics/ 6/11 3/5/2018 Can AI make your company more diverse? Learn from Pymetrics | SmartRecruiters
Hire better with the applicant tracking system that works.
High performance organizations use SmartRecruiters' recruiting so ware to find and hire the best talent.
About SmartRecruiters
About Us Talent Acquisition Suite Marketplace Resources Free Tour
Get the latest Journal content sent to your inbox.
Email Address
Daily Weekly https://www.smartrecruiters.com/blog/can-ai-make-your-company-more-diverse-learn-from-pymetrics/ 10/11 3/5/2018 Dell Technologies 2018 Predictions – Entering the Next Era of Human Machine Partnerships - Dell Technologies
EMERGING TECHNOLOGIES Dell Technologies 2018 Predictions – Entering the Next Era of Human Machine Partnerships
December 11, 2017
Our research partners at The Institute for the Future (IFTF) recently forecasted that we’re entering the next era of human machine partnership, and that between now and 2030 humans and machines will work in closer concert with each other, transforming our lives.
We’ve worked with machines for centuries, but we’re about to enter an entirely new phase – characterized by even greater efficiency, unity and possibility than ever before.
Emerging technologies, such as Artificial Intelligence (AI), Augmented Reality (AR), Virtual Reality (VR), and advances in Internet of Things (IoT) and cloud computing – made possible through exponential developments in software, analytics, and processing power – are augmenting and accelerating this direction.
This is evident in our connected cars, homes, business and banking transactions already; even transforming how farmers manage their crops and cattle. Given this dizzying pace of progress, Luminaries & Insights let’s take a look at what’s coming down the pike next.
“In the future it won’t be a question of people or machines, the magic is people and machines. We’re entering the next era of https://www.delltechnologies.com/en-us/perspectives/dell-technologies-2018-predictions-entering-the-next-era-of-human-machine-partnerships/ 1/10 3/5/2018 Dell Technologies 2018 Predictions – Entering the Next Era of Human Machine Partnerships - Dell Technologies human-machine partnership, a more integrated, personal relationship with technology that has the power to amplify exponentially the creativity, inspiration, intelligence and curiosity of the human spirit.” — Michael Dell, Chairman and CEO, Dell Technologies
Prediction 1: AI will do the ‘thinking tasks’ at speed.
Over the next few years, AI will change the way we spend our time acting on data, not just curating it. Businesses will harness AI to do data-driven “thinking tasks” for them, significantly reducing the time they spend scoping, debating, scenario planning and testing every new innovation. It will mercifully release bottlenecks and liberate people to make more decisions and move faster, in the knowledge that great new ideas won’t get stuck in the mire.
Some theorists claim AI will replace jobs, but these new technologies may also create new ones, unleashing new opportunities for humans. For example, we’ll see a new type of IT professional focused on AI training and fine-tuning. These practitioners will be responsible for setting the parameters for what should and shouldn’t be classified good business outcomes, determining the rules for engagement, framing what constitutes ‘reward’ and so on. Once this is in place, the technology will be able to recommend positive commercial opportunities at lightning speed.
2018 Prediction: AI Will Do the ‘Thinking Tasks’ at Speed
https://www.delltechnologies.com/en-us/perspectives/dell-technologies-2018-predictions-entering-the-next-era-of-human-machine-partnerships/ 2/10 3/5/2018 Dell Technologies 2018 Predictions – Entering the Next Era of Human Machine Partnerships - Dell Technologies Prediction 2: Embedding the IQ of Things
Starting in 2018, we’ll take gargantuan strides in embedding near-instant intelligence in IoT- enhanced cities, organizations, homes, and vehicles. With the cost of processing power decreasing and a connected node approaching $0, soon we’ll have 100 billion connected devices, and after that a trillion. The magnitude of all that data combined, processing power with the power of AI will help machines better orchestrate our physical and human resources. We’ll evolve into ‘digital conductors’ of the technology and environments surrounding us. Technology will function as an extension of ourselves. Every object will become smart and enable us to live smarter lives.
We’re seeing this in our cars – the “ultimate mobile device” – which are being fitted out with ultrasonic sensors, technology that makes use of light beams to measure distance between vehicles and gesture recognition. In time, these innovations will make autonomous driving an everyday reality. Well before, we’ll get used to cars routinely booking themselves in for a service, informing the garage what needs to be done and scheduling their own software updates.
“Harnessing the full potential of IoT requires adding intelligence to every stage. Smarter data gathering at the edge, smarter compute at the core and deeper learning at the cloud. All this intelligence will drive a new level of business innovation in every industry.”
— Ray O’Farrell, VMware EVP & CTO, and general manager for Dell Technologies IoT division
Prediction 3: We’ll don AR headsets.
It also won’t be long until the lines between ‘real’ reality and augmented reality begin to blur. AR’s commercial viability is already evident. For instance, teams of construction workers, architects and engineers are using AR headsets to visualize new builds, coordinate efforts based on a single view of a development and train on-the-job laborers when a technician can’t be on site that day.
Of course, VR has strong prospects too. It will undoubtedly transform the entertainment and gaming space in the near term, thanks to the immersive experiences it affords, but smart bets are on AR becoming the de facto way of maximizing human efficiency and leveraging the ‘tribal knowledge’ of an evolving workforce.
“It’s happening in pockets today. But over the next year, VR and AR will cogently bring world issues into our personal reality. Time, money, geography will no longer be an inhibitor to experiencing a different environment, community or perspective.”
—Trisa Thompson, Chief Responsibility Officer https://www.delltechnologies.com/en-us/perspectives/dell-technologies-2018-predictions-entering-the-next-era-of-human-machine-partnerships/ 3/10 3/5/2018 Dell Technologies 2018 Predictions – Entering the Next Era of Human Machine Partnerships - Dell Technologies Prediction 4: A Deeper Relationship with Customers
Dell Technologies’ Digital Transformation Index shows that 45% of leaders in mid to large organizations believe they could be obsolete within 5 years and 78% see start-ups as a threat to their business. It’s never been more important to put the customer experience first.
“Today upwards of 60% of businesses are not prepared to completely address their customers’ digital expectations, but forward-thinking businesses are piloting these efforts with an eye on operationalizing in the coming year.”
—Marius Haas, President & Chief Commercial Officer
Over the next year, with predictive analytics, machine learning (ML) and AI at the forefront, companies will better understand and serve customers at, if not before the point of need. Customer service will pivot on perfecting the blend between man and machine. So, rather than offloading customer interactions to first generation chatbots and predetermined messages, humans and automated intelligent virtual agents will work together, as one team.
“Stronger human-machine partnerships will result in stronger human-human relationships, as companies take a customer- first approach and lead with insights. By applying machine learning and AI to customer data, companies will be able to predict and understand customer behavior like never before.” — Karen Quintos, Chief Customer Officer
Prediction 5: Bias check will become the next spell check.
Over the next decade, emerging technologies such as VR, AI, will help people find and act on information without interference from emotions or external prejudice, while empowering them to exercise human judgment where appropriate. In the short-term, we’ll see AI applied to hiring and promotion procedures to screen for conscious and unconscious bias. Meanwhile VR will increasingly be used as an interview tool to ensure https://www.delltechnologies.com/en-us/perspectives/dell-technologies-2018-predictions-entering-the-next-era-of-human-machine-partnerships/ 4/10 3/5/2018 Dell Technologies 2018 Predictions – Entering the Next Era of Human Machine Partnerships - Dell Technologies opportunities are awarded on merit alone, e.g. by masking a prospective employee’s true identity with an avatar.
“AI and VR capabilities will be key to dismantling human predisposition. In the short-term we’ll apply VR in job interview settings to remove identifying traits, machine learning to indicate key points of discrimination or AI to program positive bias into our processes.”
— Brian Reaves, Chief Diversity & Inclusion Officer
By using emerging technologies to these ends, ‘bias check’ could one day become a routine sanitizer, like ‘spell check’- but with society-wide benefits.
Prediction 6: Media & Entertainment will break new ground with esports.
In 2018, we’ll see increasingly vast numbers of players sitting behind screens or wearing VR headsets to battle it out in a high-definition computer-generated universe. As hundreds of millions of players and viewers tune-in, esports will go mainstream.
Additionally traditional sports, like cycling, have upped their game by harvesting data to identify incremental but game-changing gains. In the future every business will be a technology business, and our leisure time will become a connected experience.
Prediction 7: We’ll journey toward the “mega-cloud.”
Cloud is not a destination. It’s an IT model where orchestration, automation and intelligence are embedded deeply into IT Infrastructure. In 2018, businesses are overwhelmingly moving toward a multi-cloud approach, taking advantage of the value of all models from public to private, hosted, managed and SaaS. However, as more applications and workloads move into various clouds, the proliferation of cloud siloes will become an inevitability, thus inhibiting the organization’s ability to fully exploit data analytics and AI initiatives. This may also result in applications and data landing in the wrong cloud leading to poor outcomes. As a next step, we’ll see the emergence of the “mega cloud”, which will weave together multiple private and public clouds to behave as a coherent, holistic system. The mega cloud will offer a federated, intelligent view of an entire IT environment. To make the mega cloud possible, we will need to create multi-cloud innovations in networking (to move data between clouds), storage (to place data in the right cloud), compute (to utilize the best processing and acceleration for the https://www.delltechnologies.com/en-us/perspectives/dell-technologies-2018-predictions-entering-the-next-era-of-human-machine-partnerships/ 5/10 3/5/2018 Dell Technologies 2018 Predictions – Entering the Next Era of Human Machine Partnerships - Dell Technologies workloads), orchestration (to link networking, storage and compute together across clouds) and, as a new opportunity, customers will have to incorporate AI and ML to bring automation and insight to a new level from this next generation IT environment.
“We now live in a world of distributed data – with much of it being underutilized. Organizations know that data is today’s currency of innovation and business growth. We’ll see organizations drive a demand for a holistic and harmonious view of all their data, and ultimately, an intelligent ‘mega cloud’.”
— Jeff Clarke, Vice Chairman, Product & Operations
Prediction 8: The Year to Sweat the Small Stuff
In this increasingly interconnected world, our reliance on third parties has never been greater. Organizations aren’t simple atomic instances; rather, they are highly interconnected systems that exist as part of something even bigger. The ripples of chaos spread farther and faster now that technology connects us in astonishing ways. Consider that one of the most substantial data breaches in history occurred because attackers used credentials to log into a third-party HVAC system.
“We live in an increasingly interconnected world. The cyber threat will take on new proportions and the next year will the year to ‘sweat the small stuff’.”
— Zulfikar Ramzan, Chief Technology Officer, RSA
Due to our increasingly interwoven relationship with machines, small subtle failures can lead to mega failures. Hence, next year will be a year of action for multinational corporations, further inspired by the onslaught of new regulations such as GDPR. Prioritizing the implementation of cybersecurity tools and technologies to effectively protect data and prevent threats will be a growing imperative.
“There’s immense responsibility that comes with technological developments that drive human progress and change how we do things, in IT, our workplace and, realistically, every aspect of our lives. Those companies that properly manage and integrate the explosion of intelligent, personal compute devices will win in
https://www.delltechnologies.com/en-us/perspectives/dell-technologies-2018-predictions-entering-the-next-era-of-human-machine-partnerships/ 6/10 3/5/2018 Dell Technologies 2018 Predictions – Entering the Next Era of Human Machine Partnerships - Dell Technologies 2018.” — Liam Quinn, Chief Technology Officer
Would you like to read more like this?
0 Comments Dell Technologies 1 Login
Recommend 6 ⤤ Share Sort by Newest
Start the discussion…
LOG IN WITH OR SIGN UP WITH DISQUS ?
Name
Be the first to comment.
✉ Subscribe Privacy
Tags: ARTIFICIAL INTELLIGENCE AUGMENTED REALITY CLOUD NATIVE
CUSTOMER EXPERIENCE DIGITAL TRANSFORMATION DIGITAL TRENDS
EMERGING TECHNOLOGIES INTERNET OF THINGS MACHINE LEARNING
TECH TRENDS
https://www.delltechnologies.com/en-us/perspectives/dell-technologies-2018-predictions-entering-the-next-era-of-human-machine-partnerships/ 7/10 3/5/2018 Dell Technologies 2018 Predictions – Entering the Next Era of Human Machine Partnerships - Dell Technologies
CYBERSECURITY ARTIFICIAL INTELLIGENCE The Framework for the Future: Cybersecurity and AI Innovations Unleashing Digital Transformation Automation in 2030 February 9, 2018 February 21, 2018 by Daniel Newman, principal analyst at Futurum By Eric Vanderburg Research and author of Futureproof It is healthy to take a step back from the I’ve long said digital transformation is a journey, technological changes of today to strategize on not a destination. Every journey has a catalyst how the technology and business landscape for forward motion—the fuel in the tank, if you will evolve in the future. Let’s take a look. will. Let’s examine a few ways AI is pushing digital transformation forward into the next phase.
LUMINARIES & INSIGHTS 3,800 Business Leaders Declare: It’s A Tale of Two Futures
January 30, 2018 Jeremy Burton, CMO of Dell, discusses the latest research in which 3,800 business leaders from around the world weigh in on what they think the next 10-15 years have in store. https://www.delltechnologies.com/en-us/perspectives/dell-technologies-2018-predictions-entering-the-next-era-of-human-machine-partnerships/ 9/10 3/5/2018 Dell Technologies 2018 Predictions – Entering the Next Era of Human Machine Partnerships - Dell Technologies
Privacy Statement
Ads & Emails
Manage Cookies
© 2018 Dell Inc.
https://www.delltechnologies.com/en-us/perspectives/dell-technologies-2018-predictions-entering-the-next-era-of-human-machine-partnerships/ 10/10 3/5/2018 Diversity in Artificial Intelligence
Problem Research Cases Priorities Press about us Join
http://diversity.ai/ 1/8 3/5/2018 Diversity in Artificial Intelligence
Problem Research Cases Priorities Press about us Join
A think tank for inclusive Arti cial Intelligence
Preventing racial, age, gender, disability and other discrimination by humans and A.I. using the latest advances in Arti cial Intelligence
http://diversity.ai/ 2/8 3/5/2018 Diversity in Artificial Intelligence
Problem Research Cases Priorities Press about us Join
The new issue of Science (Vol 357, Issue 6346) is devoted to AI.
We strongly suggest reading the editorial "AI, people, and society". (Science Editorial)
Publication Evaluating race and sex diversity in the world's largest companies using deep neural networks
Publication
Evaluating race and sex diversity in the world's largest companies using deep neural networks
http://diversity.ai/ 3/8 3/5/2018 Diversity in Artificial Intelligence
Problem Research Problem Cases de nitionPriorities Press about us Join
The "AI Spring" fueled by the major leaps in performance and applications of machine learning technologies and especially in deep learn triggered a wave of concerns regarding AI safety. Most of these concerns are focused on the possibility of intelligent machines causing p harm to or exterminating humans entirely. These concerns are important but are not timely.
Sentient machines are very unlikely to get out of control any time soon. Howev racial, gender, ethnic, and age discrimination by arti cially intelligent systems probably happening already in many aspects of our lives involving individual a group pro ling. Dataset
For example, as we learned through several experimental projects including Aging.AI and Beauty.AI when certain population groups are represented in the training sets, these populations are left out or may be subject to higher error rates. These are the types of problems this collective will try to address through meetings, seminars, hackathons, open database sharing and a arti cially intelligent systems to promote and encourage inclusion, equality, balance and diversity.
Research directions
AI for Diversity.
Using AI to detect, analyze, prevent and combat human Using AI to prevent discrimination by AI. biases and discrimination.
The title of our rst research project is "Evaluating race and sex diversity in the world's largest companies using Deep Neural Networks".
Here is the abstract of our rst paper: Diversity is one of the fundamental properties for survival of species, populations and organizations. Recent advances in deep learning a the rapid and automatic assessment of organizational diversity and possible discrimination by race, sex, age and other parameters. Auto the process of assessing the organizational diversity using the deep neural networks and eliminating the human factor may provide a se real-time unbiased reports to all stakeholders. In this pilot study we applied the deep learned predictors of race and sex to the executive http://diversity.ai/ 4/8 3/5/2018 Diversity in Artificial Intelligence management and board member pro les of the 500 largest companies from the 2016 Forbes Global 2000 list and compared the predic ratios to the ratios within eachProblem company's Research country of origin and Cases ranked themPriorities by the sex-,Press age- about and us race- Joindiversity index (DI). While the st many limitations and no claims are being made concerning the individual companies, it demonstrates a method for the rapid and impar assessment of organizational diversity using deep neural networks.
Click the button below to get access to the dataset that we used in this study.
Dataset
http://diversity.ai/ 5/8 3/5/2018 Diversity in Artificial Intelligence
Problem Research Cases Priorities Press about us Join
Immediate tactical goals:
“ Achievement has no color.
— Abraham Lincoln
http://diversity.ai/ 6/8 3/5/2018 Diversity in Artificial Intelligence
Problem Research Cases Priorities Press about us Join
Priorities
Healthcare Security Insurance Finance
http://diversity.ai/ 7/8 3/5/2018 Diversity in Artificial Intelligence
Problem Research Cases Priorities Press about us Join
Join our e ort
Nominate an advisor or become a volunteer
LinkedIn or Facebook
Full name
SUGGEST
http://diversity.ai/ 8/8
ARTICLE 29 DATA PROTECTION WORKING PARTY
17/EN
WP 251
Guidelines on Automated individual decision-making and Profiling for the purposes of Regulation 2016/679
Adopted on 3 October 2017
This Working Party was set up under Article 29 of Directive 95/46/EC. It is an independent European advisory body on data protection and privacy. Its tasks are described in Article 30 of Directive 95/46/EC and Article 15 of Directive 2002/58/EC.
The secretariat is provided by Directorate C (Fundamental Rights and Union Citizenship) of the European Commission, Directorate General Justice, B-1049 Brussels, Belgium, Office No MO-59 03/075.
Website: http://ec.europa.eu/justice/data-protection/index_en.htm THE WORKING PARTY ON THE PROTECTION OF INDIVIDUALS WITH REGARD TO THE
PROCESSING OF PERSONAL DATA
set up by Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995,
having regard to Articles 29 and 30 thereof,
having regard to its Rules of Procedure,
HAS ADOPTED THE PRESENT GUIDELINES:
2
TABLE OF CONTENTS
I. DEFINITIONS ...... 6
A. PROFILING ...... 6
B. AUTOMATED DECISION-MAKING ...... 7
C. HOW THE GDPR ADDRESSES THE CONCEPTS...... 8
II. SPECIFIC PROVISIONS ON AUTOMATED DECISION-MAKING AS DEFINED IN ARTICLE 22 9
A. ‘BASED SOLELY ON AUTOMATED PROCESSING’ ...... 9
B. ‘LEGAL’ OR ‘SIMILARLY SIGNIFICANT’ EFFECTS ...... 10
C. EXCEPTIONS FROM THE PROHIBITION ...... 12 1. Performance of a contract ...... 12 2. Authorised by Union or Member State law...... 12 3. Explicit consent ...... 13
D. RIGHTS OF THE DATA SUBJECT ...... 13 1. Articles 13(2) (f) and 14(2) (g) - Right to be informed ...... 13 2. Article 15(1) (h) - Right of access ...... 15 3. Article 22(1) – Right not to be subject to a decision based solely on automated decision-making ..... 15
E. SPECIAL CATEGORY DATA – ARTICLE 22(4) ...... 16
F. ESTABLISHING APPROPRIATE SAFEGUARDS ...... 16
III. GENERAL PROVISIONS ON PROFILING AND AUTOMATED DECISION-MAKING ...... 17
A. DATA PROTECTION PRINCIPLES ...... 17 1. Article 5(1) (a) - Lawful, fair and transparent ...... 17 2. Article 5(1) (b) Further processing and purpose limitation ...... 18 3. Article 5(1) (c) Data minimisation ...... 19 4. Article 5(1) (d) Accuracy ...... 19 5. Article 5(1) (e) Storage limitation ...... 19
B. LAWFUL BASES FOR PROCESSING ...... 20 1. Article 6(1) (a) consent ...... 20 2. Article 6(1) (b) – necessary for the performance of a contract ...... 20 3. Article 6(1) (c) – necessary for compliance with a legal obligation ...... 21 4. Article 6(1) (d) – necessary to protect vital interests...... 21 5. Article 6(1) (e) – necessary for the performance of a task carried out in the public interest or exercise of official authority ...... 21 6. Article 6(1) (f) – necessary for the legitimate interests pursued by the controller or by a third party 21
C. ARTICLE 9 – SPECIAL CATEGORIES OF DATA ...... 22
D. RIGHTS OF THE DATA SUBJECT ...... 22 1. Articles 13 and 14 – Right to be informed ...... 23 2. Article 15 – Right of access ...... 24 3
3. Article 16 - Right to rectification, Article 17 Right to erasure and Article 18 Right to restriction of processing ...... 24 4. Article 21 – Right to object ...... 25
IV. CHILDREN AND PROFILING ...... 26
V. DATA PROTECTION IMPACT ASSESSMENTS (DPIA) ...... 27
ANNEX 1 - GOOD PRACTICE RECOMMENDATIONS ...... 28
ANNEX 2 – KEY GDPR PROVISIONS ...... 31
KEY GDPR PROVISIONS THAT REFERENCE AUTOMATED DECISION-MAKING AS DEFINED IN ARTICLE 22 ...... 31
KEY GDPR PROVISIONS THAT REFERENCE GENERAL PROFILING AND AUTOMATED DECISION-MAKING ...... 32
ANNEX 3 - FURTHER READING ...... 33
4
INTRODUCTION
The General Data Protection Regulation (the GDPR), specifically addresses profiling and automated individual decision-making, including profiling.1
Profiling and automated decision-making are used in an increasing number of sectors, both private and public. Banking and finance, healthcare, taxation, insurance, marketing and advertising are just a few examples of the fields where profiling is being carried out more regularly to aid decision-making.
Advances in technology and the capabilities of big data analytics, artificial intelligence and machine learning have made it easier to create profiles and make automated decisions with the potential to significantly impact individuals’ rights and freedoms.
The widespread availability of personal data on the internet and from Internet of Things (IoT) devices, and the ability to find correlations and create links, can allow aspects of an individual’s personality or behaviour, interests and habits to be determined, analysed and predicted.
Profiling and automated decision-making can be useful for individuals and organisations as well as for the economy and society as a whole, delivering benefits such as:
increased efficiencies; and resource savings.
They have many commercial applications, for example, they can be used to better segment markets and tailor services and products to align with individual needs. Medicine, education, healthcare and transportation can also all benefit from these processes.
However, profiling and automated decision-making can pose significant risks for individuals’ rights and freedoms which require appropriate safeguards.
These processes can be opaque. Individuals might not know that they are being profiled or understand what is involved.
Profiling can perpetuate existing stereotypes and social segregation. It can also lock a person into a specific category and restrict them to their suggested preferences. This can undermine their freedom to choose, for example, certain products or services such as books, music or newsfeeds. It can lead to inaccurate predictions, denial of services and goods and unjustified discrimination in some cases.
The GDPR introduces new provisions to address the risks arising from profiling and automated decision-making, notably, but not limited to, privacy. The purpose of these guidelines is to clarify those provisions.
1 Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC. Profiling and automated individual decision-making are also covered by Directive (EU) 2016/680 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data by competent authorities for the purposes of the prevention, investigation, detection or prosecution of criminal offences or the execution of criminal penalties, and on the free movement of such data. While these guidelines focus on profiling and automated individual decision-making under the GDPR, the guidance is also relevant regarding the two topics under Directive 2016/680, with respect to their similar provisions. The analysis of specific features of profiling and automated individual decision-making under Directive 2016/680 is not included in these Guidelines.
5
This document covers:
Definitions of profiling and automated decision-making and the GDPR approach to these in general – Chapter II Specific provisions on automated decision-making as defined in Article 22 – Chapter III General provisions on profiling and automated decision-making - Chapter IV Children and profiling – Chapter V Data protection impact assessments – Chapter VI
The Annexes provide best practice recommendations, building on the experience gained in EU Member States.
The Article 29 Data Protection Working Party (WP29) will monitor the implementation of these guidelines and may complement them with further details as appropriate.
I. Definitions The GDPR introduces provisions to ensure that profiling and automated individual decision-making (whether or not this includes profiling) are not used in ways that have an unjustified impact on individuals’ rights; for example:
specific transparency and fairness requirements; greater accountability obligations; specified legal bases for the processing; rights for individuals to oppose profiling and specifically profiling for marketing; and if certain conditions are met, a need to carry out a data protection impact assessment.
The GDPR does not just focus on the decisions made as a result of automated processing or profiling. It applies to the collection of data for the creation of profiles, as well as the application of those profiles to individuals.
A. Profiling
The GDPR defines profiling in Article 4(4) as: any form of automated processing of personal data consisting of the use of personal data to evaluate certain personal aspects relating to a natural person, in particular to analyse or predict aspects concerning that natural person’s performance at work, economic situation, health, personal preferences, interests, reliability, behaviour, location or movements;
Profiling is composed of three elements:
it has to be an automated form of processing; it has to be carried out on personal data; and the objective of the profiling must be to evaluate personal aspects about a natural person.
Article 4(4) refers to any form of profiling, rather than ‘solely’ automated processing (which is referred to in Article 22). Profiling has to involve some form of automated processing – although human involvement does not necessarily take the activity out of the definition.
6
Profiling is a procedure which may involve a series of statistical deductions. It is often used to make predictions about people, using data from various sources to infer something about an individual, based on the qualities of others who appear statistically similar.
The GDPR says that profiling is automated processing of personal data for evaluating personal aspects, in particular to analyse or make predictions about individuals. Therefore simply assessing or classifying individuals based on characteristics such as their age, sex, and height could be considered profiling, regardless of any predictive purpose.
The GDPR is inspired by but is not identical to the definition of profiling in the Council of Europe Recommendation CM/Rec(2010)132 (Recommendation), as the Recommendation excludes processing that does not include inference. Nevertheless the Recommendation is a useful reference tool - specifically its description of the three distinct stages of profiling:
data collection; automated analysis to identify correlations; applying the correlation to an individual to identify characteristics of present or future behaviour.
Each of the above stages represents a process that falls under the GDPR definition of profiling.
Broadly speaking, profiling means gathering information about an individual (or group of individuals) and analysing their characteristics or behaviour patterns in order to place them into a certain category or group, and/or to make predictions or assessments about, for example, their: ability to perform a task; interests; or likely behaviour.
Example
A data broker collects data from different public and private sources, either on behalf of its clients or for its own purposes. The data broker compiles the data to develop profiles on the individuals and places them into segments. It sells this information to companies who wish to improve the targeting of their goods and services. The data broker carries out profiling by placing a person into a certain category according to their interests. Whether or not there is automated decision-making as defined in Article 22(1) will depend upon the circumstances.
B. Automated decision-making
Automated decision-making has a different scope and may partially overlap with profiling. Solely automated decision-making is the ability to make decisions by technological means without human involvement. Automated decisions can be based on any type of data, for example:
2 Council of Europe. The protection of individuals with regard to automatic processing of personal data in the context of profiling. Recommendation CM/Rec(2010)13 and explanatory memorandum. Council of Europe 23 November 2010. https://www.coe.int/t/dghl/standardsetting/cdcj/CDCJ%20Recommendations/CMRec(2010)13E_Profiling.pdf . Accessed 24 April 2017
7
data provided directly by the individuals concerned (such as responses to a questionnaire); data observed about the individuals (such as location data collected via an application); derived or inferred data such as a profile of the individual that has already been created (e.g. a credit score).
Automated decisions can be made with or without profiling; profiling can take place without making automated decisions. However, profiling and automated decision-making are not necessarily separate activities. Something that starts off as a simple automated decision-making process could become one based on profiling, depending upon how the data is used.
Example
Imposing speeding fines purely on the basis of evidence from speed cameras is an automated decision- making process that does not necessarily involve profiling.
It would, however, become a decision based on profiling if the driving habits of the individual were monitored over time, and, for example, the amount of fine imposed is the outcome of an assessment involving other factors, such as whether the speeding is a repeat offence or whether the driver has had other recent traffic violations.
Decisions that are not wholly automated might also include profiling. For example, before granting a mortgage, a bank may consider the credit score of the borrower, with additional meaningful intervention carried out by humans before any decision is applied to an individual.
C. How the GDPR addresses the concepts
There are potentially three ways in which profiling may be used:
(i) general profiling; (ii) decision-making based on profiling; and (iii) solely automated decision-making, including profiling (Article 22).
The difference between (ii) and (iii) is best demonstrated by the following two examples where an individual applies for a loan online:
a human decides whether to agree the loan based on a profile produced by purely automated means(ii); an algorithm decides whether the loan is agreed and the decision is automatically delivered to the individual, without any meaningful human input (iii).
Although there are three types of profiling, only two legal frameworks apply.
Chapter III of these guidelines explains the specific provisions that apply to solely automated individual decision-making, including profiling.3 A general prohibition on this type of processing exists to reflect the potentially adverse effect on individuals.
Chapter IV of these guidelines explains the legal framework for general profiling. This includes decision-making based on profiling that is not solely automated.
3 As defined in Article 22(1) of the GDPR.
8
Under Article 23 Member States can introduce legislation to restrict4 data subjects’ rights and data controllers’ obligations regarding profiling and automated decision making. This may be particularly relevant for processing that falls under Article 22.
II. Specific provisions on automated decision-making as defined in Article 22 Article 22(1) says
The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.
In summary, Article 22 provides that:
(i) as a rule, there is a prohibition on fully automated individual decision-making, including profiling that has a legal or similarly significant effect; (ii) there are exceptions to the rule; (iii) there should be measures in place to safeguard the data subject’s rights and freedoms and legitimate interests5.
These safeguards, discussed in more detail below, include the right to be informed (addressed in Articles 13 and 14 – specifically meaningful information about the logic involved, as well as the significance and envisaged consequences for the data subject ), the right to obtain human intervention and the right to challenge the decision (addressed in Article 22(3)).
The prohibition in Article 22(1) will only apply when a decision based solely on automated processing, including profiling has a legal effect on or similarly significantly affects someone.
A. ‘Based solely on automated processing’
Article 22(1) refers to decisions ‘based solely’ on automated processing. This means that there is no human involvement in the decision process.
Example
An automated process produces what is in effect a recommendation concerning a data subject. If a human being reviews and takes account of other factors in making the final decision, that decision would not be ‘based solely’ on automated processing.
4 ‘when such a restriction respects the essence of the fundamental rights and freedoms and is a necessary and proportionate measure in a democratic society’ for certain specifically listed purposes (e.g. crime prevention) or in certain specifically listed situations. Article 23 provides a full list. Specific provisions for some of these purposes are listed in Article 23(2) 5 Recital 71 says that such processing should be “subject to suitable safeguards, which should include specific information to the data subject and the right to obtain human intervention, to express his or her point of view, to obtain an explanation of the decision reached after such assessment and to challenge the decision.”
9
The controller cannot avoid the Article 22 provisions by fabricating human involvement. For example, if someone routinely applies automatically generated profiles to individuals without any actual influence on the result, this would still be a decision based solely on automated processing.
To qualify as human intervention, the controller must ensure that any oversight of the decision is meaningful, rather than just a token gesture. It should be carried out by someone who has the authority and competence to change the decision. As part of the analysis, they should consider all the available input and output data.
B. ‘Legal’ or ‘similarly significant’ effects The GDPR recognises that automated decision-making, including profiling can have serious consequences for individuals. However, the GDPR does not define ‘legal’ or ‘similarly significant’ effects.
‘Legal effects’
A legal effect suggests a processing activity that has an impact on someone’s legal rights, such as the freedom to associate with others, vote in an election, or take legal action. A legal effect may also be something that affects a person’s legal status or their rights under a contract. For example, automated decisions that mean someone is:
entitled to or denied a particular social benefit granted by law, such as child or housing benefit; refused entry at the border; subjected to increased security measures or surveillance by the competent authorities; or automatically disconnected from their mobile phone service for breach of contract because they forgot to pay their bill before going on holiday.
‘Similarly significantly affects him or her’
Even if a decision-making process does not have an effect on people’s legal rights it could still fall within the scope of Article 22 if it produces an effect that is equivalent or similarly significant in its impact.
In other words, even where no legal (statutory or contractual) rights or obligations are specifically affected, the data subjects could still be impacted sufficiently to require the protections under this provision. The GDPR introduces the word ‘similarly’ (not present in Article 15 of Directive 95/46/EC) to the phrase ‘significantly affects’. This suggests that the threshold for significance must be similar, whether or not the decision has a legal effect.
For data processing to significantly affect someone the effects of the processing must be more than trivial and must be sufficiently great or important to be worthy of attention. In other words, the decision must have the potential to significantly influence the circumstances, behaviour or choices of the individuals concerned. At its most extreme, the decision may lead to the exclusion or discrimination of individuals.
Recital 71 provides the following typical examples: ‘automatic refusal of an online credit application’ or ‘e-recruiting practices without any human intervention’. These suggest that it is difficult to be precise about what would be considered sufficiently significant to meet the threshold. For example, based on the recital each of the following credit decisions fall under Article 22, but with very different degrees of impact on the individuals concerned:
renting a city bike during a vacation abroad for two hours; 10
purchasing a kitchen appliance or a television set on credit; obtaining a mortgage to buy a first home.
This bring us also to the issue of online advertising, which increasingly relies on automated tools and involves solely automated individual decision-making.
In many typical cases targeted advertising does not have a significant effect on individuals, for example an advertisement for a mainstream online fashion outlet based on a simple demographic profile: ‘women in the Brussels region’.
However it is possible that it may do, depending upon the particular characteristics of the case, including:
the intrusiveness of the profiling process; the expectations and wishes of the individuals concerned; the way the advert is delivered; or the particular vulnerabilities of the data subjects targeted.
Processing that might have little impact on individuals generally may in fact have a significant effect on certain groups of society, such as minority groups or vulnerable adults. For example, someone in financial difficulties who is regularly shown adverts for on-line gambling may sign up for these offers and potentially incur further debt.
Even where advertising or marketing practices do not fall under Article 22, data controllers must comply with the general legal framework applicable to profiling under the GDPR, covered in Chapter IV. The provisions of the proposed ePrivacy Regulation may also be relevant in many cases. Furthermore, children require enhanced protection, as will be discussed below in Chapter V.
Automated decision-making that results in differential pricing could also have a significant effect if, for example, prohibitively high prices effectively bar someone from certain goods or services.
Although the context and wording differ, the concept ‘substantially affects’ is discussed in the WP29 guidelines on the lead supervisory authority. The examples in these guidelines6 may be helpful when considering the effects of automated decision-making on data subjects.
Similarly significant effects may be positive or negative. These effects may also be triggered by the actions of individuals other than the one to which the automated decision relates. An illustration of this is given below.
Example
Hypothetically, a credit card company might reduce a customer’s card limit, based not on that customer’s own repayment history, but on non-traditional credit criteria, such as an analysis of other customers living in the same area who shop at the same stores.
This could mean that someone is deprived of opportunities based on the actions of others.
In a different context using these types of characteristics might have the advantage of extending credit to those without a conventional credit history, who would otherwise have been denied.
6 Article 29 Data Protection Working Party. Guidelines for identifying a controller or processor’s lead supervisory authority. 5 April 2017, http://ec.europa.eu/newsroom/document.cfm?doc_id=44102 Accessed 24 April 2017
11
C. Exceptions from the prohibition
Article 22(1) sets out a general prohibition on solely automated individual decision with a significant effect, as described above.
This means that the controller should not undertake the processing described in Article 22(1) unless one of the following Article 22(2) exceptions applies:
(a) necessary for the performance of or entering into a contract; (b) authorised by Union or Member State law to which the controller is subject and which also lays down suitable measures to safeguard the data subject’s rights and freedoms and legitimate interests; or (c) based on the data subject’s explicit consent.
1. Performance of a contract
Controllers may wish to use automated decision-making, for example, because it:
potentially allows for greater consistency or fairness in the decision making process (e.g. it might reduce the potential for human error, discrimination and abuse of power); reduces the risk of customers failing to meet payments for goods or services (for example by using credit referencing); or enables them to deliver decisions within a shorter time frame and improves the efficiency of the process. Routine human involvement may sometimes also be impractical or impossible due to the sheer quantity of data being processed.
Regardless of the above, these considerations alone are not always sufficient to show that this type of processing is necessary under Article 22(2)(a) for entering into, or the performance of, a contract. As described in the WP29 Opinion on legitimate interest, necessity should be interpreted narrowly.
The controller must be able to show that this profiling is necessary, taking into account whether a less privacy-intrusive method could be adopted. 7 If other less intrusive means to achieve the same goal exist, then the profiling would not be ‘necessary’.
2. Authorised by Union or Member State law
Automated decision-making including profiling could potentially take place under 22(2)(b) if Union or Member State law authorised its use. Recital 71 says that this could include its use for monitoring and preventing fraud and tax-evasion or to ensure the security and reliability of a service provided by the controller.
7 Buttarelli, Giovanni. Assessing the necessity of measures that limit the fundamental right to the protection of personal data. AToolkit European Data Protection Supervisor, 11 April 2017, https://edps.europa.eu/sites/edp/files/publication/17-04-11_necessity_toolkit_en_0.pdf Accessed 24 April 2017
12
3. Explicit consent Article 22 requires explicit consent. Processing that falls within the definition of Article 22(1) poses significant data protection risks and a high level of individual control over personal data is therefore deemed appropriate.
‘Explicit consent’ is not defined in the GDPR but suggests that the consent must be specifically confirmed by an express statement rather than some other affirmative action.
Explicit consent will be addressed in the forthcoming consent guidelines. Chapter IV.B provides more information on consent generally.
D. Rights of the data subject8
1. Articles 13(2) (f) and 14(2) (g) - Right to be informed Given the potential risks and interference that profiling caught by Article 22 poses to the rights of data subjects, data controllers should be particularly mindful of their transparency obligations. This means ensuring that information about the profiling is not only easily accessible for a data subject but that it is brought to their attention.9
Articles 13(2) (f) and 14(2) (g) require controllers to provide specific information about automated decision-making, based solely on automated processing, including profiling, that produces legal or similarly significant effects.10
If the controller is making automated decisions as described in Article 22(1), they must:
tell the data subject that they are engaging in this type of activity; provide meaningful information about the logic involved; and explain the significance and envisaged consequences of the processing.
Meeting these three specified transparency requirements will (along with other suitable safeguards) help controllers to better inform data subjects about the Article 22 (1) type of processing and the consequences.
It is good practice to provide the above information whether or not the processing falls within the narrow Article 22(1) definition. In any event the controller must provide sufficient information to the data subject to make the processing fair,11 and meet all the other information requirements of Articles 13 and 14. These requirements are explained in more detail in Chapter IV(section D).
8 GDPR Article 12 provides for the modalities applicable for the exercise of the data subject’s rights
9 Recital 60 states that “Furthermore the data subject should be informed of the existence of profiling and the consequences of such profiling”.
10 Referred to in Article 22(1) and (4).The forthcoming WP Guidelines on transparency will cover the general information requirements of Articles 13 and 14.
11 GDPR Recital 60 “The controller should provide the data subject with any further information necessary to ensure fair and transparent processing taking into account the specific circumstances and context in which the personal data are processed. Furthermore the data subject should be informed of the existence of profiling and the consequences of such profiling.”
13
Meaningful information about the ‘logic involved’
The growth and complexity of machine-learning can make it challenging to understand how an automated decision-making process or profiling works.
The controller should find simple ways to tell the data subject about the rationale behind, or the criteria relied on in reaching the decision without necessarily always attempting a complex explanation of the algorithms used or disclosure of the full algorithm.12 The information provided should, however, be meaningful to the data subject.
Example
A controller uses credit scoring to assess and reject an individual’s loan application. The score may have been provided by a credit reference agency, or calculated directly based on information held by the controller.
Regardless of the source (and information on the source must be provided to the data subject under Article 14 (2) (f) where the personal data have not been obtained from the data subject), if the controller is reliant upon this score it must be able to explain it and the rationale, to the data subject.
The controller explains that this process helps them make fair and responsible lending decisions. It provides details of the main characteristics considered in reaching the decision, the source of this information and the relevance. This may include, for example: the information provided by the data subject on the application form; information about previous account conduct , including any payment arrears; and official public records information such as fraud record information and insolvency records.
The controller also includes information to advise the data subject that the credit scoring methods used are regularly tested to ensure they remain fair, effective and unbiased. The controller provides contact details for the data subject to request that any declined decision is reconsidered, in line with the provisions of Article 22(3).
‘Significance’ and ‘envisaged consequences’
This term suggests that information must be provided about intended or future processing, and how the automated decision-making might affect the data subject.13 In order to make this information meaningful and understandable, real, tangible examples of the type of possible effects should be given.
12Complexity is no excuse for failing to provide information to the data subject. Recital 58 states that the principle of transparency is “of particular relevance in situations where the proliferation of actors and the technological complexity of practice makes it difficult for the data subject to know and understand whether, by whom and for what purpose personal data relating to him are being collected, such as in the case of online advertising”.
13 Council of Europe. Draft Explanatory Report on the modernised version of CoE Convention 108, paragraph 75: “Data subjects should be entitled to know the reasoning underlying the processing of their data, including the consequences of such a reasoning, which led to any resulting conclusions, in particular in cases involving the use of algorithms for automated-decision making including profiling. For instance in the case of credit scoring, they should be entitled to know the logic underpinning the processing of their data and resulting in a ‘yes’ or ‘no’ decision, and not simply information on the decision itself. Without an understanding of these elements there could be no effective exercise of other essential safeguards such as the right to object and the right to complain to a competent authority.”
14
Example
An insurance company uses an automated decision making process to set motor insurance premiums based on monitoring customers’ driving behaviour. To illustrate the significance and envisaged consequences of the processing it explains that dangerous driving may result in higher insurance payments and provides an app comparing fictional drivers, including one with dangerous driving habits such as fast acceleration and last-minute braking.
It uses graphics to give tips on how to improve these habits and consequently how to lower insurance premiums.
Controllers can use similar visual techniques to explain how a past decision has been made.
2. Article 15(1) (h) - Right of access
Article 15(1) (h) entitles data subjects to have the same information about solely automated decision- making, including profiling, as required under Articles 13(2) (f) and 14(2) (g), namely:
the existence of automated decision making, including profiling; meaningful information about the logic involved; and the significance and envisaged consequences of such processing for the data subject.
The controller should have already given the data subject this information in line with their Article 13 obligations.14
3. Article 22(1) – Right not to be subject to a decision based solely on automated decision-making As explained earlier in this chapter, Article 22(1) acts as a prohibition on solely automated individual decision-making, including profiling with legal or similarly significant effects. Instead of the data subject having to actively object to the processing, the controller can only carry out the processing if one of the three exceptions covered in Article 22(2) applies.
Even where one of these exceptions does apply, the GDPR provides a further layer of protection for data subjects in Article 22(3)15 “at least the right to obtain human intervention on the part of the controller, to express his or her point of view and to contest the decision”. The controller must provide a simple way for the data subject to exercise these rights.
Human intervention is a key element. Any review must be carried out by someone who has the appropriate authority and capability to change the decision. The reviewer should undertake a thorough assessment of all the relevant data, including any additional information provided by the data subject.
https://rm.coe.int/CoERMPublicCommonSearchServices/DisplayDCTMContent?documentId=09000016806b6e c2 . Accessed 24 April 2017 14 GDPR Article 12(3) clarifies the timescales for providing this information
15 Where the basis for processing is Article 22(2)(a) or (c)
15
E. Special category data – Article 22(4) Automated decision-making (described in Article 22(1)) that involves special categories of personal data is only allowed under certain conditions provided for in the GDPR or by Union or Member State Law (Article 22(4), referring to Article 9(2), (a) or (g)), namely:
9(2) (a) - the explicit consent of the data subject; or
9(2) (g) - processing necessary for reasons of substantial public interest, on the basis of Union or Member State law which shall be proportionate to the aim pursued, respect the essence of the right to data protection and provide for suitable and specific measures to safeguard the fundamental rights and interests of the data subject.
F. Establishing appropriate safeguards If the basis for processing is 22(2)(a) or 22(2)(c), Article 22(3) requires controllers to implement suitable measures to safeguard data subjects’ rights freedoms and legitimate interests. Such measures should include as a minimum a way for the data subject to obtain human intervention, express their point of view, and contest the decision. Recital 71 highlights that in any case suitable safeguards should also include:
.. specific information to the data subject ………… to obtain an explanation of the decision reached after such assessment and to challenge the decision.
Similarly, Article 22(2) (b) requires that the Member State law which authorises the processing includes suitable protections for data subjects.
This emphasises the need for transparency about the processing. The data subject will only be able to challenge a decision or express their view if they fully understand how it has been made and on what basis. Transparency requirements are discussed in Chapter III(section D).
Errors or bias in collected or shared data or an error or bias in the automated decision-making process can result in: incorrect classifications; and assessments based on imprecise projections; that impact negatively on individuals.
Controllers should carry out frequent assessments on the data sets they process to check for any bias, and develop ways to address any prejudicial elements, including any over-reliance on correlations. Systems that audit algorithms and regular reviews of the accuracy and relevance of automated decision-making including profiling are other useful measures.
Controllers should introduce appropriate procedures and measures to prevent errors, inaccuracies or discrimination on the basis of special category data16. These should be used on a cyclical basis; not
16 GDPR Recital 71 says that: “In order to ensure fair and transparent processing in respect of the data subject, taking into account the specific circumstances and context in which the personal data are processed, the controller should use appropriate mathematical or statistical procedures for the profiling, implement technical and organisational measures appropriate to ensure, in particular, that factors which result in inaccuracies in personal data are corrected and the risk of errors is minimised,….”
16
only at the design stage, but also continuously, as the profiling is applied to individuals. The outcome of such testing should feed back into the system design.
Further examples of appropriate safeguards can be found in the Recommendations section.
III. General provisions on profiling and automated decision-making This overview of the provisions applies to both profiling and automated decision-making.
A. Data protection principles The principles are relevant for all profiling and automated decision-making involving personal data.17 To aid compliance, controllers should consider the following key areas:
1. Article 5(1) (a) - Lawful, fair and transparent Transparency of processing18 is a fundamental requirement of the GDPR.
The process of profiling is often invisible to the data subject. It works by creating derived or inferred data about individuals – ‘new’ personal data that has not been provided directly by the data subjects themselves. Individuals have differing levels of comprehension and may find it challenging to understand the complex techniques involved in profiling and automated decision-making processes.
Under Article 12.1 the controller must provide data subjects with concise, transparent, intelligible and easily accessible information about the processing of their personal data.19
For data collected directly from the data subject this should be provided at the time of collection (Article 13); for indirectly obtained data the information should be provided within the timescales set out in Article 14(3).
Example
Some insurers offer insurance rates and services based on an individual’s driving behaviour. Elements taken into account in these cases could include the distance travelled, the time spent driving and the journey undertaken as well as predictions based on other data collected by the sensors in a (smart) car. The data collected is used for profiling to identify bad driving behaviour (such as fast acceleration, sudden braking, and speeding). This information can be cross-referenced with other sources (for example the weather, traffic, type of road) to better understand the driver’s behaviour.
17 GDPR – Recital 72 “Profiling is subject to the rules of this Regulation governing the processing of personal data, such as the legal grounds for processing or data protection principles.”
18 The forthcoming WP29 Guidelines on transparency will cover transparency generally in more detail.
19 Office of the Australian Information Commissioner. Consultation draft: Guide to big data and the Australian Privacy Principles, 05/2016 says: “Privacy notices have to communicate information handling practices clearly and simply, but also comprehensively and with enough specificity to be meaningful. The very technology that leads to greater collection of personal information also presents the opportunity for more dynamic, multi- layered and user centric privacy notices.” https://www.oaic.gov.au/engage-with-us/consultations/guide-to-big- data-and-the-australian-privacy-principles/consultation-draft-guide-to-big-data-and-the-australian-privacy- principles . Accessed 24 April 2017
17
The controller must ensure that they have a lawful basis for this type of processing. The controller must also provide the data subject with information about the collected data, the existence of automated decision-making, the logic involved, and the significance and envisaged consequences of such processing.
The specific requirements surrounding information and access to personal data are discussed in Chapters III (section D) and IV (section D).
Processing also has to be fair, as well as transparent.
Profiling may be unfair and create discrimination, for example by denying people access to employment opportunities, credit or insurance, or targeting them with excessively risky or costly financial products. The following example illustrates how unfair profiling can lead to some consumers being offered less attractive deals than others.
Example
A data broker sells consumer profiles to financial companies without consumer permission or knowledge of the underlying data. The profiles define consumers into categories (carrying titles such as “Rural and Barely Making It,” “Ethnic Second-City Strugglers,” “Tough Start: Young Single Parents,”) or “score” them, focusing on consumers’ financial vulnerability. The financial companies offer these consumers payday loans and other “non-traditional” financial services (high-cost loans and other financially risky products).20
2. Article 5(1) (b) Further processing and purpose limitation Profiling can involve the use of personal data that was originally collected for something else.
Example
Some mobile applications provide location services allowing the user to find nearby restaurants offering discounts. However, the data collected is also used to build a profile on the data subject for marketing purposes - to identify their food preferences, or lifestyle in general. The data subject expects their data will be used to find restaurants, but not to receive adverts for pizza delivery just because the app has identified that they arrive home late. This further use of the location data may not be compatible with the purposes for which it was collected in the first place, and may thus require the consent of the individual concerned.21
20 This example is taken from: United States Senate, Committee on Commerce, Science, and Transportation. A Review of the Data Broker Industry: Collection, Use, and Sale of Consumer Data for Marketing Purposes, Staff Report for Chairman Rockefeller, December 18, 2013. https://www.commerce.senate.gov/public/_cache/files/0d2b3642-6221-4888-a631- 08f2f255b577/AE5D72CBE7F44F5BFC846BECE22C875B.12.18.13-senate-commerce-committee-report-on- data-broker-industry.pdf . See page ii of the Executive Summary and 12 of the main body of the document in particular. Accessed 21 July 2017
21 Note that the provisions of the future ePrivacy Regulation may also apply.
18
Whether this additional processing is compatible with the original purposes for which the data were collected will depend upon a range of factors22, including what fair processing information the controller initially provided to the data subject. These factors are reflected in the GDPR23and summarised below:
the relationship between the purposes for which the data have been collected and the purposes of further processing; the context in which the data were collected and the reasonable expectations of the data subjects as to their further use; the nature of the data and the impact of the further processing on the data subjects; and the safeguards applied by the controller to ensure fair processing and to prevent any undue impact on the data subjects.
3. Article 5(1) (c) Data minimisation The business opportunities created by profiling, cheaper storage costs and the ability to process large amounts of information can encourage organisations to collect more personal data than they actually need, in case it proves useful in the future.
Controllers should be able to clearly explain and justify the need to collect and hold personal data, or consider using aggregated or otherwise anonymised (not just pseudonymised) data for profiling.
4. Article 5(1) (d) Accuracy Controllers should consider accuracy at all stages of the profiling process, specifically when:
collecting data; analysing data; building a profile for an individual; or applying a profile to make a decision affecting the individual.
If the data used in an automated decision-making or profiling process is inaccurate, any resultant decision or profile will be flawed. Decisions may be made on the basis of outdated data or the incorrect interpretation of external data. Inaccuracies may lead to inappropriate predictions or statements about, for example, someone’s health, credit or insurance risk.
Even if raw data is recorded accurately, the dataset may not be fully representative or the analytics may contain hidden bias.
Controllers need to introduce robust measures to verify and ensure on an ongoing basis that data re- used or obtained indirectly is accurate and up to date. This reinforces the importance of providing clear information about the personal data being processed, so that the data subject can correct any inaccuracies and improve the quality of the data.
5. Article 5(1) (e) Storage limitation Machine-learning algorithms are designed to process large volumes of information and build correlations. Storing collected personal data for lengthy periods of time means that organisations will
22 Highlighted in the Article 29 Data Protection Working Party. Opinion 03/2013 on purpose limitation,2 April 2013. http://ec.europa.eu/justice/data-protection/article-29/documentation/opinion- recommendation/files/2013/wp203_en.pdf . Accessed 24 April 2017
23 GDPR Article 6(4)
19
be able to build up very comprehensive, intimate profiles of individuals, since there will be more data for the algorithm to learn. Even if collection of the information fulfils the requirements of purpose specification and relevance, storing it for a long time may conflict with the proportionality consideration, i.e. the method may be too intrusive in terms of the individual's right to privacy.24
Keeping personal data for too long also increases the risk of inaccuracies.
B. Lawful bases for processing For automated decision making under Article 22, please note that only some of the lawful bases listed here are available, as described in Chapter III (section C) above.
1. Article 6(1) (a) consent Consent as a basis for processing generally will be addressed in the upcoming WP29 Guidelines on consent. Explicit consent is one of the exceptions from the prohibition on automated decision-making and profiling defined in Article 22(1).
Profiling can be opaque. Often it relies upon data that is derived or inferred from other data, rather than data directly provided by the data subject.
Controllers seeking to rely upon consent as a basis for profiling will need to show that data subjects understand exactly what they are consenting to. In all cases, data subjects should have enough relevant information about the envisaged use and consequences of the processing to ensure that any consent they provide represents an informed choice.
Where the data subject has no choice, for example, in situations where consent to profiling is a pre- condition of accessing the controller’s services; or where there is an imbalance of power such as in an employer/employee relationship, consent is not an appropriate basis for the processing.
2. Article 6(1) (b) – necessary for the performance of a contract This is covered in more detail in Chapter III(section C). The following is an example of profiling that would not meet the Article 6(1)(b) basis for processing.
Example
A user buys some items from an on-line retailer. In order to fulfil the contract, the retailer must process the user’s credit card information for payment purposes and the user’s address to deliver the goods. Completion of the contract is not dependent upon building a profile of the user’s tastes and lifestyle choices based on his or her visits to the website. Even if profiling is specifically mentioned in the small print of the contract, this fact alone does not make it ‘necessary’ for the performance of the contract.
24 Norwegian Data Protection Authority. The Great Data Race – How commercial utilisation of personal data challenges privacy, Report, November 2015. Datatilsynet https://www.datatilsynet.no/English/Publications/The- Great-Data-Race/ Accessed 24 April 2017
20
3. Article 6(1) (c) – necessary for compliance with a legal obligation There may be instances where there will be a legal obligation25 to carry out profiling – for example in connection with fraud prevention or money laundering. For more detail, including the safeguards to be applied, see the WP29 Opinion on legitimate interests.
4. Article 6(1) (d) – necessary to protect vital interests This covers situations where the processing is necessary to protect an interest which is essential for the life of the data subject or that of another natural person.
Certain types of processing may serve important public interest grounds as well as the vital interests of the data subject. Examples of this may include profiling necessary to develop models that predict the spread of life-threatening diseases or in situations of humanitarian emergencies. In these cases, however, and in principle, the controller can only rely on vital interest grounds if no other legal basis for the processing is available.26 If the processing involves special category personal data the controller would also need to ensure that they meet the requirements of Article 9(2) (c).
5. Article 6(1) (e) – necessary for the performance of a task carried out in the public interest or exercise of official authority Article 6(1) (e) might be an appropriate basis for public sector profiling in certain circumstances.
6. Article 6(1) (f) – necessary for the legitimate interests27 pursued by the controller or by a third party Profiling is allowed if it is necessary for the purposes of the legitimate interests28 pursued by the controller or by a third party. However, Article 6(1) (f) does not automatically apply just because the controller has a legitimate interest. The controller must carry out a balancing exercise to assess whether their interests are overridden by the data subject’s interests or fundamental rights and freedoms.
The following are particularly relevant:
the level of detail of the profile (a data subject profiled within a broadly described cohort such as ‘native English teachers living in Paris’, or segmented and targeted on a granular level); the comprehensiveness of the profile (whether the profile only describes a small aspect of the data subject, or paints a more comprehensive picture); the impact of the profiling (the effects on the data subject); and the safeguards aimed at ensuring fairness, non-discrimination and accuracy in the profiling process.
The WP29 opinion on legitimate interests29 explains the more common contexts where this issue arises and gives examples that may be relevant to profiling.
25 GDPR Recitals 41 and 45 26 GDPR Recital 46 27 Legitimate interests listed in GDPR Recital 47 include processing for direct marketing purposes and processing strictly necessary for the purposes of preventing fraud. 28 The controller’s “legitimate interest” cannot render profiling lawful if the processing falls within the Article 22(1) definition. 29 Article 29 Data Protection Working Party. Opinion 06/2014 on the notion of legitimate interests of the data controller under Article 7 of Directive 95/46/EC. European Commission, 9 April 2014 ,P59 – examples 5and 7
21
The controller should also consider the future use or combination of profiles when assessing the validity of processing under Article 6(1) (f).
C. Article 9 – Special categories of data30
Controllers can only process special category personal data if they can meet one of the conditions set out in Article 9(2), as well as a condition from Article 6. This includes special category data derived or inferred from profiling activity.
Profiling can create special category data by inference from other data which is not special category data in its own right but becomes so when combined with other data. For example, it may be possible to infer someone’s state of health from the records of their food shopping combined with data on the quality and energy content of foods.
Correlations may be discovered that indicate something about individuals’ health, political convictions, religious beliefs or sexual orientation, as demonstrated by the following example:
Example
One study31 combined Facebook ‘likes’ with limited survey information and found that researchers accurately predicted a male user’s sexual orientation 88% of the time; a user’s ethnic origin 95% of the time; and whether a user was Christian or Muslim 82% of the time.
Informing data subjects is particularly important in the case of inferences about sensitive preferences and characteristics. The controller should make the data subject aware that not only do they process (non-special category) personal data collected from the data subject or other sources but also that they derive from such data other (and special) categories of personal data relating to them.
D. Rights of the data subject32 The GDPR introduces stronger rights for data subjects and creates new obligations for controllers.
In the context of profiling these rights are actionable against the controller creating the profile and the controller making an automated decision about a data subject (with or without human intervention), if these entities are not the same.
http://ec.europa.eu/justice/data-protection/article-29/documentation/opinion- recommendation/files/2014/wp217_en.pdf . Accessed 24 April 2017
30 This Section is relevant for both profiling and automated decision-making. For automated decision-making under Article 22, please note the additional, specific provisions, as described in Chapter IV.F above.
31 Michael Kosinski, David Stilwell and Thore Graepel. Private traits and attributes are predictable from digital records of human behaviour. Proceedings of the National Academy of Sciences of the United States of America, http://www.pnas.org/content/110/15/5802.full.pdf . Accessed 29 March 2017
32 This Section is relevant for both profiling and automated decision-making. For automated decision making under Article 22, please note that there are also additional requirements when it comes to special categories of data, as described in Chapter III(section E) above.
22
Example
A data broker undertakes profiling of personal data. In line with their Article 13 and 14 obligations the data broker should inform the individual about the processing, including whether it intends to share the profile with any other organisations. The data broker should also present separately details of the right to object under Article 21(1).
The data broker shares the profile with another company. This company uses the profile to send the individual direct marketing.
The company should inform the individual (Article 14(1) (c)) about the purposes for using this profile, and from what source they obtained the information (14(2) (f)). The company must also advise the data subject about their right to object to processing, including profiling, for direct marketing purposes (Article 21(2)).
The data broker and the company should allow the data subject the right to access the information used (Article 15) to correct any erroneous information (Article 16), and in certain circumstances erase the profile or personal data used to create it (Article 17). The data subject should also be given information about their profile, for example in which ‘segments’ or ‘categories’ they are placed. 33
If the company uses the profile as part of a solely automated decision-making process with legal or similarly significant effects on the data subject, the company is the controller subject to the Article 22 provisions. (This does not exclude the data broker from Article 22 if the processing meets the relevant threshold.)
1. Articles 13 and 14 – Right to be informed Given the core principle of transparency underpinning the GDPR, controllers must ensure they explain clearly and simply to individuals how the profiling or automated decision-making process works.
In particular, where the processing involves profiling-based decision making (irrespective of whether it is caught by Article 22), then the fact that the processing is for the purposes of both (a) profiling and (b) making a decision based on the profile generated, must be made clear to the data subject.34
Recital 60 states that giving information about profiling is part of the controller’s transparency obligations under Article 5(1) (a). The data subject has a right to be informed by the controller about and, in certain circumstances, a right to object to ‘profiling’, regardless of whether a fully automated individual decision based on profiling takes place.
Further information about the different approaches to transparency in the context of profiling is provided in the Recommendations section. More generally, the WP29 will also publish guidelines on transparency under the GDPR.
33 The Norwegian Data Protection Authority. The Great Data Race -How commercial utilisation of personal data challenges privacy. Report, November 2015. https://www.datatilsynet.no/English/Publications/The-Great-Data- Race/ Accessed 24 April 2017
34 GDPR – Article 13(1)(c) and Article 14(1)(c)
23
2. Article 15 – Right of access Article 15 gives the data subject the right to obtain details of any personal data used for profiling, including the categories of data used to construct a profile.
Article 15 implies a more general form of oversight, rather than a right to an explanation of a particular decision. Nevertheless, through the exercise of this right the data subject can become aware of a decision made, including one based on profiling him or her.
Recital 63 provides some protection for controllers concerned about revealing trade secrets or intellectual property, which may be particularly relevant in relation to profiling. It says that the right of access ‘should not adversely affect the rights or freedoms of others’. However, only under rare circumstances should these rights outweigh individuals’ rights of access; controllers should not use this as an excuse to deny access or refuse to provide any information to the data subject. These rights should be considered in context and balanced against individuals’ rights to have information.
Recital 63 also specifies that ‘where possible, the controller should be able to provide remote access to a secure system which would provide the data subject with direct access to his or her personal data.’
In addition to information about the profile, the controller should make available the data used as input to create the profile.
3. Article 16 - Right to rectification, Article 17 Right to erasure and Article 18 Right to restriction of processing Profiling can involve an element of prediction, which increases the risk of inaccuracy. The input data may be inaccurate or irrelevant, or taken out of context. There may be something wrong with the algorithm used to identify correlations.
The Article 16 right to rectification might apply where, for example, an individual is placed into a category that says something about their ability to perform a task, and that profile is based on incorrect information. Individuals may wish to challenge the accuracy of the data used and any grouping or category that has been applied to them.
Article 16 also provides a right for the data subject to complement the personal data with additional information.
Example
A local surgery’s computer system places an individual into a group that is most likely to get heart disease. This ‘profile’ is not necessarily inaccurate even if he or she never suffers from heart disease. The profile merely states that he or she is more likely to get it. That may be factually correct as a matter of statistics.
Nevertheless, the data subject has the right, taking into account the purpose of the processing, to provide a supplementary statement. In the above scenario, this could be based, for example, on a more advanced medical computer system (and statistical model) carrying out more detailed examinations and factoring in additional data than the one at the local surgery with more limited capabilities.
The right to rectification applies to the ‘input personal data’ (the personal data used to create the profile) and to the ‘output data’ (the profile itself or ‘score’ assigned to the person, which is personal data relating to the person concerned).
24
Similarly the right to erasure (Article 17) will apply to both the input and the output data. If the basis for profiling is consent and that consent is withdrawn, the controller must erase the relevant personal data – unless there is another legal basis for the profiling. 35 The right to restrict processing (Article 18) will apply to any stage of the profiling process.
4. Article 21 – Right to object The controller has to bring details of the right to object under Article 21(1) and (2) explicitly to the data subject’s attention, and present it clearly and separately from other information (Article 21(4)).
Under Article 21(1) the data subject can object to processing (including profiling), on grounds relating to his or her particular situation. Controllers are specifically required to provide this right in all cases where processing is based on Article 6(1) (e) or (f).
Once the data subject exercises this right, the controller must interrupt36 (or avoid starting) the profiling process unless it can demonstrate compelling legitimate grounds that override the interests or rights and freedoms of the data subject. The controller may also have to erase the relevant personal data.37
The GDPR does not provide any explanation of what would be considered compelling legitimate grounds. 38 It may be the case that, for example, the profiling is beneficial for society at large (or the wider community) and not just the business interests of the controller, such as the interest in carrying out scientific research, or profiling to predict the spread of contagious diseases.
The controller would also need to prove that:
the impact on data subjects is limited to the minimum necessary to meet the particular objective (i.e. the profiling is the least intrusive way to achieve this); and the objective is critical for the organisation .
There must always be a balancing exercise between the competing interests of the controller and the basis for the data subject’s objection (which may be for personal, social or professional reasons). Unlike in the Directive, the burden of proof to show compelling legitimate grounds lies with the controller rather than the data subject.
Article 21(2) grants an unconditional right for the data subject to object to the processing of their personal data for direct marketing purposes, including profiling to the extent that it is related to such direct marketing. This means that there is no need for any balancing of interests; the controller must respect the individual’s wishes without questioning the reasons for the objection. Recital 70 provides additional context to this right and says that it may be exercised at any time and free of charge.
35 GDPR – Article 17(1)(b)
36 GDPR- Article 18(1)(d)
37 GDPR – Article 17(1)(c)
38 See explanation on legitimacy p. 24, Article 29 Data Protection Working Party Opinion 06/2014 on the notion of legitimate interests of the data controller under Article 7 of Directive 95/46/EC. European Commission. 9 April 2014. Page 26 http://ec.europa.eu/justice/data-protection/article-29/documentation/opinion- recommendation/files/2014/wp217_en.pdf . Accessed 24 April 2017
25
IV. Children and profiling The GDPR creates additional obligations for data controllers when they are processing children’s personal data.
Article 22 itself makes no distinction as to whether the processing concerns adults or children. However, recital 71 says that solely automated decision-making, including profiling, with legal or similarly significant effects should not apply to children.39 Given that this wording is not reflected in the Article itself, WP29 does not consider that this represents an absolute prohibition on this type of processing in relation to children. However, in the light of this recital, WP29 recommends that, wherever possible, controllers should not rely upon the exceptions in Article 22(2) to justify it.
There may nevertheless be some circumstances in which it is necessary for controllers to carry out solely automated decision-making, including profiling, with legal or similarly significant effects in relation to children, for example to protect their welfare. If so, the processing may be carried out on the basis of the exceptions in Article 22(2)(a), (b) or (c) as appropriate.
In those cases there must be suitable safeguards in place, as required by Article 22(2)(b) and 22(3), and they must therefore be appropriate for children. The controller must ensure that these safeguards are effective in protecting the rights, freedoms and legitimate interests of the children whose data they are processing.
The need for particular protection for children is reflected in recital 38, which says:
Children merit specific protection with regard to their personal data, as they may be less aware of the risks, consequences and safeguards concerned and their rights in relation to the processing of personal data. Such specific protection should, in particular, apply to the use of personal data of children for the purposes of marketing or creating personality or user profiles and the collection of personal data with regard to children when using services offered directly to a child.
Article 22 does not prevent controllers from making solely automated decisions about children, if the decision will not have a legal or similarly significant effect on the child. However, solely automated decision making which influences a child’s choices and behaviour could potentially have a legal or similarly significant effect on them, depending upon the nature of the choices and behaviours in question.
Because children represent a more vulnerable group of society, organisations should, in general, refrain from profiling them for marketing purposes. Children can be particularly susceptible in the online environment and more easily influenced by behavioural advertising. For example, in online gaming, profiling can be used to target players that the algorithm considers are more likely to spend money on the game as well as providing more personalised adverts. The age and maturity of the child may affect their ability to understand the motivation behind this type of marketing or the consequences.40
Article 40(2) (g) explicitly refers to the preparation of codes of conduct incorporating safeguards for children; it may also be possible to develop existing codes.41
39 Recital 71 – “such measure should not concern a child”.
40 An EU study on the impact of marketing through social media, online games and mobile applications on children’s behaviour found that marketing practices have clear impacts on children’s behaviour. 41 One example of a code of conduct dealing with marketing to children is that produced by FEDMA Code of conduct, explanatory memorandum, available at: http://www.oecd.org/sti/ieconomy/2091875.pdf Accessed 15
26
V. Data protection impact assessments (DPIA) Accountability is an important area and an explicit requirement under the GDPR. 42
A DPIA enables the controller to assess the risks involved in automated decision-making, including profiling. It is a way of showing that suitable measures have been put in place to address those risks and demonstrate compliance with the GDPR.
Article 35(3) (a) highlights the need for the controller to carry out a DPIA in the case of: a systematic and extensive evaluation of personal aspects relating to natural persons which is based on automated processing, including profiling, and on which decisions are based that produce legal effects concerning the natural person or similarly significantly affect the natural person;
Article 35(3)(a) refers to evaluations including profiling and decisions that are ‘based’ on automated processing, rather than ‘solely’ automated processing. We take this to mean that Article 35(3) (a) will apply in the case of decision-making including profiling with legal or similarly significant effects that is not wholly automated, as well as solely automated decision-making defined in Article 22(1).
Controllers could consider additional measures43 such as:
informing the data subject about the existence of and the logic involved in the automated decision-making process; explaining the significance and envisaged consequences of the processing for the data subject; providing the data subject with the means to oppose the decision; and allowing the data subject to express their point of view.
Other profiling activities may warrant a DPIA, depending upon the specifics of the case. Controllers may wish to consult the WP29 guidelines on DPIAs44 for further information and to help determine the need to carry out a DPIA.
May 2017. See, in particular: “6.2 Marketers targeting children, or for whom children are likely to constitute a section of their audience, should not exploit children’s credulity, loyalty, vulnerability or lack of experience.; 6.8.5 Marketers should not make a child’s access to a website contingent on the collection of detailed personal information. In, particular, special incentives such as prize offers and games should not be used to entice children to divulge detailed personal information.” 42 As required by the GDPR Article 5(2) 43 Mirroring the requirements in Article 13(2)(f), Article 14(2)(g) and Article 22(3) 44 Article 29 Data Protection Working Party. Guidelines on Data Protection Impact Assessment (DPIA) and determining whether processing is “likely to result in a high risk” for the purposes of Regulation 2016/679. 4 April 2017. European Commission. http://ec.europa.eu/newsroom/document.cfm?doc_id=44137 Accessed 24 April 2017.
27
ANNEX 1 - Good practice recommendations The following good practice recommendations will assist data controllers in meeting the requirements of the GDPR provisions on profiling and automated decision making.45
Article Issue Recommendation
5(1)(a),12, Right to have Controllers may wish to consider: 13, 14 information layered notices, where data subjects are informed about the processing of their data on a step by step basis. This type of approach can work by providing the key privacy information in a short notice, with links to expand each section to its full version, and a just in time notification at the point where the data is collected; visualisation and interactive techniques to aid algorithmic transparency46; standardised icons47 to inform individuals about profiling and automated decision-making, for example: o The organisation shares their personal data with other organisations; o Details of the other organisations with whom their personal data is shared; o Whether this/these organisation(s) is/are using their personal data to profile them; o Whether the profile is being used to make decisions about them.
Meaningful information about the logic involved will in most cases require controllers to provide details such as:
the information used in the automated decision-making process, including the categories of data used in a profile; the source of that information; how any profile used in the automated decision-making process is built, including any statistics used in the analysis; why this profile is relevant to the automated decision-making process; and how it is used for a decision concerning the data subject.
5(1)(b) Further If the processing is outside what the individual concerned would processing reasonably expect, or would have an unjustified adverse effect on them, controllers should regard the use or disclosure as incompatible with the original purpose for obtaining the information.
45 Controllers also need to ensure they have robust procedures in place to ensure that they can meet their obligations under Articles 15 – 22 in the timescales provided for by the GDPR. 46 Information Commissioner’s Office .Big data, artificial intelligence, machine learning and data protection , page 87, paragraph 194. ICO, 1 March 2017. https://ico.org.uk/for-organisations/guide-to-data-protection/big- data/ Accessed 24 April 2017 47 As envisaged in Article 12(7)
28
Article Issue Recommendation
6(1)(a) Consent as a If controllers are relying upon consent as a basis for processing they basis for should: processing provide sufficiently clear and comprehensive information about the profiling to ensure that data subjects understand what they are consenting to ; consider introducing a process of granular consent where they provide a clear and simple way for data subjects to agree to different purposes for processing; actively seek consent from the data subject before any new processing takes place; inform the data subject that they can withdraw consent.
15 Right of Information about the categories of data that have been or will be used access in the profiling or decision making process and why these are considered pertinent will generally be more relevant than providing a complex mathematical explanation about how algorithms or machine- learning work, although the latter should also be provided if this is necessary to allow experts to further verify how the decision –making process works.
Controllers may want to consider implementing a mechanism for data subjects to check their profile, including details of the information and sources used to develop it.
16 Right to Controllers providing data subjects with access to their profile in rectification connection with their Article 15 rights should allow them the opportunity to update or amend any inaccuracies in the data or profile. This can also help them meet their Article 5(1) (d) obligations.
Controllers should consider introducing online preference management tools such as a privacy dashboard. This gives data subjects the option of managing what is happening to their information across a number of different services – allowing them to alter settings, update their personal details, and review or edit their profile to correct any inaccuracies.
21(1) and Right to The right to object in Article 21(1) and (2) has to be explicitly brought (2) object to the attention of the data subject and presented clearly and separately from other information (Article 21(4).
Controllers need to ensure that this right is prominently displayed on their website or in any relevant documentation and not hidden away within any other terms and conditions.
29
Article Issue Recommendation
22 and Appropriate The following list, though not exhaustive, provides some good practice Recital 71 safeguards suggestions for controllers to consider when profiling: regular quality assurance checks of their systems to make sure that individuals are being treated fairly and not discriminated against, whether on the basis of special categories of personal data or otherwise; algorithmic auditing – testing the algorithms used and developed by machine learning systems to prove that they are actually performing as intended, and not producing discriminatory, erroneous or unjustified results; specific measures for data minimisation to incorporate clear retention periods for profiles and for any personal data used when creating or applying the profiles; using anonymisation or pseudonymisation techniques in the context of profiling; ways to allow the data subject to express his or her point of view and contest the decision; and, a mechanism for human intervention in defined cases, for example providing a link to an appeals process at the point the automated decision is delivered to the data subject, with agreed timescales for the review and a named contact point for any queries .
Controllers can also explore options such as: certification mechanisms for processing operations; codes of conduct for auditing processes involving machine learning; ethical review boards to assess the potential harms and benefits to society of particular applications for profiling.
30
ANNEX 2 – Key GDPR provisions
Key GDPR provisions that reference automated decision-making as defined in Article 22 Article Recital Comments
13(2)(f) 61 Right to be informed about: and the existence of automated decision-making under A22(1) and (4); 14(2)(g) meaningful information about the logic involved; significance and envisaged consequences of such processing. 15(h) Specific access rights to information about the existence of solely automated decision-making, including profiling. 22(1) 71 Prohibition on decision-making based solely on automated processing, including profiling, which produces legal/similarly significant effects. Recital 71: “…Such processing includes ‘profiling’ that consists of any form of automated processing of personal data evaluating the personal aspects relating to a natural person, in particular to analyse or predict aspects concerning the data subject’s performance at work, economic situation, health, personal preferences or interests, reliability or behaviour, location or movements”………. “Such measure should not concern a child” 22(2)(a- 71 Article 22(2) lifts the prohibition for processing based on (a) the performance c) of or entering into a contract, (b) Union or Member state law, or (c) explicit consent. Recital 71 provides further context on 22(2)(b)and says that processing described in A22(1): “should be allowed where expressly authorised by Union or Member State law to which the controller is subject, including for fraud and tax-evasion monitoring and prevention purposes conducted in accordance with the regulations, standards and recommendations of Union institutions or national oversight bodies and to ensure the security and reliability of a service provided by the controller…” 22(3) 71 Article 22 (3) and Recital 71 also specify that even in the cases referred to in 22(2)(a) and (c) the processing should be subject to suitable safeguards. Recital 71: “which should include specific information to the data subject and the right to obtain human intervention, to express his or her point of view, to obtain an explanation of the decision reached after such assessment and to challenge the decision. Such measure should not concern a child.” 23 73 Recital 73: “Restrictions concerning specific principles and concerning …….the right to object and decisions based on profiling …….may be imposed by Union or Member State law as far as necessary and proportionate in a democratic society…” to safeguard specific objectives of general public interest. 35(3)(a) 91 Requirement to carry out a DPIA.
47(2)(e) Binding corporate rules referred to in 47(1) should specify at least “…….the right not to be subject to decisions based solely on automated processing, including profiling in accordance with Article 22...”
31
Key GDPR provisions that reference general profiling and automated decision-making Article Recital Comments
3(2)(b) 24 Monitoring behaviour of EU citizens. Recital 24 “….tracked on the internet ……use of personal data processing techniques which consist of profiling a natural person, particularly in order to take decisions concerning her or him or for analysing or predicting her or his personal preferences, behaviours or attitudes”. 4(4) 30 Article 4(4) definition of profiling Recital 30 “online identifiers …., such as Internet Protocol addresses, cookie identifiers or other identifiers such as radio frequency identification tags… may leave traces which, in particular when combined with unique identifiers and other information received by the servers, may be used to create profiles of the natural persons and identify them.” 5 and 6 72 Recital 72: “Profiling is subject to the rules of this Regulation governing the processing of personal data, such as the legal grounds for processing (Article 6) or data protection principles (Article 5).” 8 38 Use of children’s personal data for profiling. Recital 38: “Children merit specific protection ….. in particular,…to the use of personal data of children for the purposes of….creating personality or user profiles.”
13 and 60 Right to be informed. 14 Recital 60: “Furthermore, the data subject shall be informed of the existence of profiling and the consequences of such profiling.” 15 63 Right of access. Recital 63: “right to know and obtain communication…..with regard to the purposes for which the personal data are processed,…..and, at least when based on profiling, the consequences of such profiling”. 21(1)(2) 70 Right to object to profiling. and (3) Recital 70 “…the right to object to such processing, including profiling to the extent that it is related to such direct marketing.” 23 73 Recital 73: “Restrictions concerning specific principles and concerning …….the right to object and decisions based on profiling …….may be imposed by Union or Member State law as far as necessary and proportionate in a democratic society…” to safeguard specific objectives of general public interest. 35(3)(a) 91 A DPIA is required in the case of “a systematic and extensive evaluation of personal aspects relating to natural persons which is based on automated processing, including profiling, and on which decisions are based that produce legal effects concerning the natural person or similarly significantly affect the natural person;” Covers decision-making including profiling that is not solely automated.
32
ANNEX 3 - Further reading These Guidelines take account of the following:
- WP29 Advice paper on essential elements of a definition and a provision on profiling within the EU General Data Protection Regulation, adopted 13 May 2013; - WP29 Opinion 2/2010 on online behavioural advertising, WP171; - WP29 Opinion 03/2013 on Purpose limitation, WP 203; - WP29 Opinion 06/2014 on the Notion of legitimate interests of the data controller under Article 7 of Directive 95/46/EC, WP217 - WP29 Statement on the role of a risk-based approach to data protection legal frameworks, WP218; - WP29 Opinion 8/2014 on the Recent Developments on the Internet of Things, WP223; - WP29 Guidelines on Data Protection Officers (DPOs), WP243; - WP29 Guidelines on identifying a controller or processor’s lead supervisory authority, WP244;Council of Europe. Recommendation CM/Rec(2010)13 on the protection of individuals with regard to automatic processing of personal data in the context of profiling; - Council of Europe. Guidelines on the protection of individuals with regard to the processing of personal data in a world of Big Data, 01/2017 - Information Commissioner’s Office – Big data, artificial intelligence, machine learning and data protection version 2.0, 03/2017 - Office of the Australian Commissioner - Consultation draft: Guide to big data and the Australian Privacy Principles, 05/2016 - European Data Protection Supervisor (EDPS) Opinion 7/2015 – Meeting the challenges of big data, 19 November 2015 - Datatilsynet – Big Data – privacy principles under pressure 09/2013 - Council of Europe. Convention for the protection of individuals with regard to automatic processing of personal data - Draft explanatory report on the modernised version of CoE Convention 108, August 2016 - Datatilsynet – The Great Data Race – How commercial utilisation of personal data challenges privacy. Report, November 2015 - European Data Protection Supervisor – Assessing the necessity of measures that limit the fundamental right to the protection of personal data: A Toolkit - Joint Committee of the European Supervisory Authorities. Joint Committee Discussion Paper on the use of Big Data by financial institutions 2016-86. https://www.esma.europa.eu/sites/default/files/library/jc-2016-86_discussion_paper_big_data.pdf. - Commission de la protection de la vie privée. Big Data Rapport https://www.privacycommission.be/sites/privacycommission/files/documents/Big%20Data%20vo or%20MindMap%2022-02-17%20fr.pdf. - United States Senate, Committee on Commerce, Science, and Transportation. A Review of the Data Broker Industry: Collection, Use, and Sale of Consumer Data for Marketing Purposes, Staff Report for Chairman Rockefeller, December 18, 2013. https://www.commerce.senate.gov/public/_cache/files/0d2b3642-6221-4888-a631- 08f2f255b577/AE5D72CBE7F44F5BFC846BECE22C875B.12.18.13-senate-commerce- committee-report-on-data-broker-industry.pdf - Lilian Edwards & Michael Veale. Slave to the Algorithm? Why a ‘Right to an Explanation’ is probably not the remedy you are looking for. Research paper, posted 24 May 2017. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2972855 - NYTimes.com. Showing the Algorithms behind New York City Services. https://mobile.nytimes.com/2017/08/24/nyregion/showing-the-algorithms-behind-new-york-city- services.html?referer=https://t.co/6uUVVjOIXx?amp=1. Accessed 24 August 2017 33
- Council of Europe. Recommendation CM/REC(2018)x of the Committee of Ministers to Member States on Guidelines to promote, protect and fulfil children’s rights in the digital environment (revised draft, 25 July 2017). https://www.coe.int/en/web/children/-/call-for-consultation- guidelines-for-member-states-to-promote-protect-and-fulfil-children-s-rights-in-the-digital- environment?inheritRedirect=true&redirect=%2Fen%2Fweb%2Fchildren . Accessed 31 August 2017 - Unicef. Privacy, protection of personal information and reputation rights. Discussion paper series: Children’s Rights and Business in a Digital World. https://www.unicef.org/csr/files/UNICEF_CRB_Digital_World_Series_PRIVACY.pdf. Accessed 31 August 2017 - House of Lords. Growing up with the internet. Select Committee on Communications, 2nd Report of Sessions 2016 – 17. https://publications.parliament.uk/pa/ld201617/ldselect/ldcomuni/130/13002.htm. Accessed 31 August 2017
34
3/5/2018 Even artificial intelligence can acquire biases against race and gender | Science | AAAS
1K
Advertisement
Log in | My account | Contact us Become a member Renew my subscription | Sign up for newsletters
Computers learning from human writing automatically see certain occupational words as masculine and others as feminine. BENEDETTO CRISTOFANI/@SALZMANART Even artificial intelligence can acquire biases against race and gender
By Matthew Hutson Apr. 13, 2017 , 2:00 PM
One of the great promises of artificial intelligence (AI) is a world free of petty human biases. Hiring by algorithm would give men and women an equal chance at work, the thinking goes, and predicting criminal behavior with big data would sidestep racial prejudice in policing. But a new study shows that computers can be biased as well, http://www.sciencemag.org/news/2017/04/even-artificial-intelligence-can-acquire-biases-against-race-and-gender 1/6 3/5/2018 Even artificial intelligence can acquire biases against race and gender | Science | AAAS especially when they learn from us. When algorithms glean the meaning of words by gobbling up lots of human- 1K written text, they adopt stereotypes very similar to our own.
“Don’t think that AI is some fairy godmother,” says study co-author Joanna Bryson, a computer scientist at the University of Bath in the United Kingdom and Princeton University. “AI is just an extension of our existing culture.”
The work was inspired by a psychological tool called the implicit association test, or IAT. In the IAT, words flash on a computer screen, and the speed at which people react to them indicates subconscious associations. Both black and white Americans, for example, are faster at associating names like “Brad” and “Courtney” with words like “happy” and “sunrise,” and names like “Leroy” and “Latisha” with words like “hatred” and “vomit” than vice versa.
SIGN UP FOR OUR DAILY NEWSLETTER Get more great content like this delivered right to you!
Email Address Sign Up
By signing up, you agree to share your email address with the publication. Information provided here is subject to Science's privacy policy.
To test for similar bias in the “minds” of machines, Bryson and colleagues developed a word-embedding association test (WEAT). They started with an established set of “word embeddings,” basically a computer’s definition of a word, based on the contexts in which the word usually appears. So “ice” and “steam” have similar embeddings, because both often appear within a few words of “water” and rarely with, say, “fashion.” But to a computer an embedding is represented as a string of numbers, not a definition that humans can intuitively understand. Researchers at Stanford University generated the embeddings used in the current paper by analyzing hundreds of billions of words on the internet.
Instead of measuring human reaction time, the WEAT computes the similarity between those strings of numbers. Using it, Bryson’s team found that the embeddings for names like “Brett” and “Allison” were more similar to those for positive words including love and laughter, and those for names like “Alonzo” and “Shaniqua” were more similar to negative words like “cancer” and “failure.” To the computer, bias was baked into the words.
IATs have also shown that, on average, Americans associate men with work, math, and science, and women with family and the arts. And young people are generally considered more pleasant than old people. All of these associations were found with the WEAT. The program also inferred that flowers were more pleasant than insects and musical instruments were more pleasant than weapons, using the same technique to measure the similarity of their embeddings to those of positive and negative words.
The researchers then developed a word-embedding factual association test, or WEFAT. The test determines how strongly words are associated with other words, and then compares the strength of those associations to facts in the real world. For example, it looked at how closely related the embeddings for words like “hygienist” and “librarian” were to those of words like “female” and “woman.” For each profession, it then compared this computer-generated gender association measure to the actual percentage of women in that occupation. The results were very highly correlated. So embeddings can encode everything from common sentiments about flowers to racial and gender biases and even facts about the labor force, the team reports today in Science.
“It’s kind of cool that these algorithms discovered these,” says Tolga Bolukbasi, a computer scientist at Boston University who concurrently conducted similar work with similar results. “When you’re training these word embeddings, you never actually specify these labels.” What’s not cool is how prejudiced embeddings might be deployed—when sorting résumés or loan applications, say. For example, if a computer searching résumés for
http://www.sciencemag.org/news/2017/04/even-artificial-intelligence-can-acquire-biases-against-race-and-gender 2/6 3/5/2018 Even artificial intelligence can acquire biases against race and gender | Science | AAAS computer programmers associates “programmer” with men, mens’ résumés will pop to the top. Bolukbasi's work 1K focuses on ways to “debias” embeddings—that is, removing unwanted associations from them.
Bryson has another take. Instead of debiasing embeddings, essentially throwing away information, she prefers adding an extra layer of human or computer judgement to decide how or whether to act on such biases. In the case of hiring programmers, you might decide to set gender quotas.
People have long suggested that meaning could plausibly be extracted through word cooccurrences, “but it was a far from a foregone conclusion,” says Anthony Greenwald, a psychologist at the University of Washington in Seattle who developed the IAT in 1998 and wrote a commentary on the WEAT paper for this week’s issue of Science. He says he expected that writing—the basis of the WEAT measurements—would better reflect explicit attitudes than implicit biases. But instead, the WEAT embeddings more closely resemble IAT biases than surveys about racial and gender attitudes, suggesting that we may convey prejudice through language in ways we don’t realize. “That was a bit surprising,” he says. He also says the WEAT might be used to test for implicit bias in past eras by testing word embeddings derived from, say, books written in the 1800s.
In the meantime, Byron and her colleagues have also shown that even Google is not immune to bias. The company’s translation software converts gender-neutral pronouns from several languages into “he” when talking about a doctor, and “she” when talking about a nurse.
All of this work “shows that it is important how you choose your words,” Bryson says. “To me, this is actually a vindication of political correctness and affirmative action and all these things. Now, I see how important it is.”
Posted in: Technology doi:10.1126/science.aal1053
Matthew Hutson
Matthew Hutson is a freelance science journalist in New York City. Email Matthew Twitter
More from News
With better atomic clocks, scientists prepare to redefine the second
Artificial intelligence could identify gang crimes—and ignite an ethical firestorm
New algorithm can create movies from just a few snippets of text
News from Science has introduced metered access. Full access to all news content is included in AAAS membership.
Got a tip? How to contact the news team http://www.sciencemag.org/news/2017/04/even-artificial-intelligence-can-acquire-biases-against-race-and-gender 3/6 3/5/2018 Even artificial intelligence can acquire biases against race and gender | Science | AAAS Advertisement 1K
Advertisement
Related Jobs
http://www.sciencemag.org/news/2017/04/even-artificial-intelligence-can-acquire-biases-against-race-and-gender 4/6 3/5/2018 Even artificial intelligence can acquire biases against race and gender | Science | AAAS
1K Postdoctoral Fellow in Cancer Research Los Angeles, California Up to $60,000 with benefits Employer: Children's Hospital Los Angeles / University of Southern California
Postdoctoral Fellow San Antonio, Texas Commensurate with experience and accomplishments; starting at NIH pay scale + benefits
We are seeking a motivated, creative postdoctoral fellow to work on DNA damage response and transcription and metabolism in cancer.
Employer: UT Health Science Center at San Antonio Post Doctoral Scholar Charleston, South Carolina Commensurate with experience. Employer: Medical University of South Carolina Department of Biochemistry
MORE JOBS ▶
ScienceInsider New NSF rules on sexual harassment leave many questions unanswered By Jeffrey Mervis Mar. 3, 2018
Flagship U.S. space telescope facing further delays By Daniel Clery Mar. 1, 2018
European agency concludes controversial ‘neonic’ pesticides threaten bees By Erik Stokstad Feb. 28, 2018
Boston University rejects geologist David Marchant’s appeal of termination By Meredith Wadman Feb. 28, 2018
EPA plan to reorganize environmental science center raises questions By Corbin Hiar, E&E News Feb. 27, 2018
http://www.sciencemag.org/news/2017/04/even-artificial-intelligence-can-acquire-biases-against-race-and-gender 5/6 3/5/2018 Even artificial intelligence can acquire biases against race and gender | Science | AAAS
1K More ScienceInsider
Sifter Gun violence studies show few causal relationships Mar. 2, 2018
Satellites discover 1.5 million more penguins in Antarctica Mar. 2, 2018
Californian ravens are a fusion of two different species Mar. 2, 2018
Scientists haven’t seen a single baby right whale calf this season Mar. 1, 2018
Warming seas may force king penguins to find a new home by 2100 Feb. 27, 2018
More Sifter
Science ECOLOGY 2 March 2018 Nature's strategies: A plant that stands and fights
Vol 359, Issue 6379
ECOLOGY Nature's strategies: Squirrels with a rainy day fund
ANTHROPOLOGY Bending to the water's will
http://www.sciencemag.org/news/2017/04/even-artificial-intelligence-can-acquire-biases-against-race-and-gender 6/6 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center
NUMBERS, FACTS AND TRENDS SHAPING YOUR WORLD ABOUT FOLLOW MY ACCOUNT DONATE
Internet & Technology
MENU RESEARCH AREAS SEARCH
FEBRUARY 8, 2017 Code-Dependent: Pros and Cons of the Algorithm Age Algorithms are aimed at optimizing everything. They can save lives, make things easier and conquer chaos. Still, experts worry they can also put too much control in the hands of corporations and governments, perpetuate bias, create filter bubbles, cut choices, creativity and serendipity, and could result in greater unemployment
BY LEE RAINIE (HTTP://WWW.PEWRESEARCH.ORG/STAFF/LEE-RAINIE/) AND JANNA ANDERSON (HTTP://WWW.PEWINTERNET.ORG/AUTHOR/JANDERSON/)
(filistimlyanin/iStock.com)
Algorithms are instructions for solving a problem (https://www.theguardian.com/science/2013/jul/01/how-algorithms-rule- world-nsa) or completing a task. Recipes are algorithms, as are math equations. Computer code is algorithmic. The internet runs on algorithms and all online searching is accomplished through them. Email knows where to go thanks to algorithms. Smartphone apps are nothing but algorithms. Computer and video games are algorithmic storytelling. Online dating and book-recommendation and travel websites would not function without algorithms. GPS mapping systems get people from point A to point B via algorithms. Artificial intelligence (AI) is naught but algorithms. The http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 1/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center material people see on social media is brought to them by algorithms. In fact, everything people see and do on the web is a product of algorithms. Every time someone sorts a column in a spreadsheet, algorithms are at play, and most financial transactions today are accomplished by algorithms. Algorithms help gadgets respond to voice commands, recognize faces, sort photos and build and drive cars. Hacking, cyberattacks and cryptographic code-breaking exploit algorithms. Self-learning and self-programming algorithms (https://erc.europa.eu/projects-and-results/erc-stories/self- learning-ai-emulates-human-brain) are now emerging, so it is possible that in the future algorithms will write many if not most algorithms.
Algorithms are often elegant and incredibly useful tools used to accomplish tasks. They are mostly invisible aids, augmenting human lives in increasingly incredible ways. However, sometimes the application of algorithms created with good intentions leads to unintended consequences. Recent news items tie to these concerns:
The British pound dropped 6.1% in value in seconds on Oct. 7, 2016, partly because of currency trades triggered by algorithms.
Microsoft engineers created a Twitter bot named “Tay” this past spring in an attempt to chat with Millennials by responding to their prompts, but within hours it was spouting racist, sexist, Holocaust-denying tweets based on algorithms that had it “learning” how to respond to others based on what was tweeted at it.
Facebook tried to create a feature to highlight Trending Topics from around the site in people’s feeds. First, it had a team of humans edit the feature, but controversy erupted when some accused the platform of being biased against conservatives. So, Facebook then turned the job over to algorithms only to find that they could not discern real news from fake news.
Cathy O’Neil, author of Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, pointed out that predictive analytics based on algorithms tend to punish the poor, using algorithmic hiring practices as an example.
Well-intentioned algorithms can be sabotaged by bad actors. An internet slowdown swept the East Coast of the U.S. on Oct. 21, 2016, after hackers bombarded Dyn DNS, an internet traffic handler, with information that overloaded its circuits, ushering in a new era of internet attacks powered by internet-connected devices. This after internet security expert Bruce Schneier warned in September that “Someone Is Learning How to Take Down the Internet.” And the abuse of Facebook’s News Feed algorithm and general promulgation of fake news online became controversial as the 2016 U.S. presidential election proceeded.
Researcher Andrew Tutt called for an “FDA for Algorithms,” noting, “The rise of increasingly complex algorithms calls for critical thought about how to best prevent, deter and compensate for the harms that they cause …. Algorithmic regulation will require federal uniformity, expert judgment, political independence and pre-market review to prevent – without stifling innovation – the introduction of unacceptably dangerous algorithms into the market.”
The White House released two reports in October 2016 detailing the advance of algorithms and artificial intelligence and plans to address issues tied to it, and it issued a December report outlining some of the potential effects of AI-driven automation on the U.S. job market and economy.
On January 17, 2017, the Future of Life Institute published a list of 23 Principles for Beneficial Artificial Intelligence, created by a gathering of concerned researchers at a conference at Asimolar, in Pacific Grove, California. The more than 1,600 signatories included Steven Hawking, Elon Musk, Ray Kurzweil and hundreds of the world’s foremost AI researchers.
http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 2/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center
The use of algorithms is spreading (https://en.wikipedia.org/wiki/List_of_algorithms) as massive amounts of data are being created, captured and analyzed by businesses and governments. Some are calling this the Age of Algorithms (https://newrepublic.com/article/133472/life-age-algorithms) and predicting that the future of algorithms (http://news.stanford.edu/2016/09/01/ai-might-affect-urban-life-2030/) is tied to machine learning and deep learning (https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/) that will get better and better at an ever-faster pace.
While many of the 2016 U.S. presidential election post-mortems noted the revolutionary impact of web-based tools in influencing its outcome, XPrize Foundation CEO Peter Diamandis predicted that “five big tech trends will make this election look tame (http://singularityhub.com/2016/11/07/5-big-tech-trends-that-will-make-this-election-look-tame/) .” He said advances in quantum computing and the rapid evolution of AI and AI agents embedded in systems and devices in the Internet of Things will lead to hyper-stalking, influencing and shaping of voters, and hyper-personalized ads, and will create new ways to misrepresent reality and perpetuate falsehoods.
http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 3/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center Analysts like Aneesh Aneesh of Stanford University foresee algorithms taking over public and private activities in a new era of “algocratic governance” (http://web.stanford.edu/class/sts175/NewFiles/Algocratic%20Governance.pdf) that supplants “bureaucratic hierarchies.” Others, like Harvard’s Shoshana Zuboff, describe the emergence of “surveillance capitalism” (https://cryptome.org/2015/07/big-other.pdf) that organizes economic behavior in an “information civilization.”
To illuminate current attitudes about the potential impacts of algorithms in the next decade, Pew Research Center and Elon University’s Imagining the Internet Center conducted a large-scale canvassing of technology experts, scholars, corporate practitioners and government leaders. Some 1,302 responded to this question about what will happen in the next decade:
Will the net overall effect of algorithms be positive for individuals and society or negative for individuals and society?
The non-scientific canvassing found that 38% of these particular respondents predicted that the positive impacts of algorithms will outweigh negatives for individuals and society in general, while 37% said negatives will outweigh positives; 25% said the overall impact of algorithms will be about 50-50, positive-negative. [See “About this canvassing of experts (http://www.pewinternet.org/2017/02/08/algorithms-about-this-canvassing-of-experts/) ” for further details about the limits of this sample.]
Participants were asked to explain their answers, and most wrote detailed elaborations that provide insights about hopeful and concerning trends. Respondents were allowed to respond anonymously; these constitute a slight majority of the written elaborations. These findings do not represent all the points of view that are possible to a question like this, but they do reveal a wide range of valuable observations based on current trends.
In the next section we offer a brief outline of seven key themes found among the written elaborations. Following that introductory section there is a much more in-depth look at respondents’ thoughts (http://www.pewinternet.org/2017/02/08/theme-1-algorithms-will-continue-to-spread-everywhere/) tied to each of the themes. All responses are lightly edited for style.
Theme 1: Algorithms will continue to spread everywhere
There is fairly uniform agreement among these respondents that algorithms are generally invisible to the public and there will be an exponential rise in their influence in the next decade.
A representative statement of this view came from Barry Chudakov, founder and principal at Sertain Research and StreamFuzion Corp. He replied:
“‘If every algorithm suddenly stopped working, it would be the end of the world as we know it.’ (Pedro Domingo’s The Master Algorithm (https://www.amazon.com/Master-Algorithm-Ultimate-Learning-Machine/dp/0465065708) ). Fact: We have already turned our world over to machine learning and algorithms. The question now is, how to better understand and manage what we have done?
“Algorithms are a useful artifact to begin discussing the larger issue of the effects of technology-enabled assists in our lives. Namely, how can we see them at work? Consider and assess their assumptions? And most importantly for those who don’t create algorithms for a living – how do we educate ourselves about the way they work, where they are in operation, what assumptions and biases are inherent in them, and how to keep them transparent? Like fish in a tank, we can see them swimming around and keep an eye on them.
http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 4/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center “Algorithms are the new arbiters of human decision-making in almost any area we can Fact: We have already turned our world imagine, from watching a movie (Affectiva emotion recognition) to buying a house (Zillow.com) to self- over to machine learning and driving cars (Google). Deloitte Global predicted algorithms. The question now is, how to more than 80 of the world’s 100 largest enterprise better understand and manage what we software companies will have cognitive technologies – mediated by algorithms – have done? integrated into their products by the end of 2016. As Brian Christian and Tom Griffiths write in BARRY CHUDAKOV Algorithms to Live By, (https://www.amazon.com/Algorithms-Live-Computer- Science-Decisions/dp/1627790365/ref=sr_1_1?s=books&ie=UTF8&qid=1476814852&sr=1- 1&keywords=algorithms+to+live+by) algorithms provide ‘a better standard against which to compare human cognition itself.’ They are also a goad to consider that same cognition: How are we thinking and what does it mean to think through algorithms to mediate our world?
“The main positive result of this is better understanding of how to make rational decisions, and in this measure a better understanding of ourselves. After all, algorithms are generated by trial and error, by testing, by observing, and coming to certain mathematical formulae regarding choices that have been made again and again – and this can be used for difficult choices and problems, especially when intuitively we cannot readily see an answer or a way to resolve the problem. The 37% Rule (https://en.wikipedia.org/wiki/Secretary_problem) , optimal stopping and other algorithmic conclusions are evidence-based guides that enable us to use wisdom and mathematically verified steps to make better decisions.
“The secondary positive result is connectivity. In a technological recapitulation of what spiritual teachers have been saying for centuries, our things are demonstrating that everything is – or can be – connected to everything else. Algorithms with the persistence and ubiquity of insects will automate processes that used to require human manipulation and thinking. These can now manage basic processes of monitoring, measuring, counting or even seeing. Our car can tell us to slow down. Our televisions can suggest movies to watch. A grocery can suggest a healthy combination of meats and vegetables for dinner. Siri reminds you it’s your anniversary.
“The main negative changes come down to a simple but now quite difficult question: How can we see, and fully understand the implications of, the algorithms programmed into everyday actions and decisions? The rub is this: Whose intelligence is it, anyway? … Our systems do not have, and we need to build in, what David Gelernter called ‘topsight,’ (http://www.peachpit.com/articles/article.aspx?p=29658&seqNum=3) the ability to not only create technological solutions but also see and explore their consequences before we build business models, companies and markets on their strengths, and especially on their limitations.”
Chudakov added that this is especially necessary because in the next decade and beyond, “By expanding collection and analysis of data and the resulting application of this information, a layer of intelligence or thinking manipulation is added to processes and objects that previously did not have that layer. So prediction possibilities follow us around like a pet. The result: As information tools and predictive dynamics are more widely adopted, our lives will be increasingly affected by their inherent conclusions and the narratives they spawn.”
“The overall impact of ubiquitous algorithms is presently incalculable because the presence of algorithms in everyday processes and transactions is now so great, and is mostly hidden from public view. All of our extended thinking systems (algorithms fuel the software and connectivity that create extended thinking systems) demand more thinking – not less – and a more global perspective than we have previously managed. The expanding collection and analysis of data and the resulting application of this information can cure diseases, decrease http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 5/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center poverty, bring timely solutions to people and places where need is greatest, and dispel millennia of prejudice, ill- founded conclusions, inhumane practice and ignorance of all kinds. Our algorithms are now redefining what we think, how we think and what we know. We need to ask them to think about their thinking – to look out for pitfalls and inherent biases before those are baked in and harder to remove.
“To create oversight that would assess the impact of algorithms, first we need to see and understand them in the context for which they were developed. That, by itself, is a tall order that requires impartial experts backtracking through the technology development process to find the models and formulae that originated the algorithms. Then, keeping all that learning at hand, the experts need to soberly assess the benefits and deficits or risks the algorithms create. Who is prepared to do this? Who has the time, the budget and resources to investigate and recommend useful courses of action? This is a 21st-century job description – and market niche – in search of real people and companies. In order to make algorithms more transparent, products and product information circulars might include an outline of algorithmic assumptions, akin to the nutritional sidebar now found on many packaged food products, that would inform users of how algorithms drive intelligence in a given product and a reasonable outline of the implications inherent in those assumptions.”
Theme 2: Good things lie ahead
A number of respondents noted the many ways in which algorithms will help make sense of massive amounts of data, noting that this will spark breakthroughs in science, new conveniences and human capacities in everyday life, and an ever-better capacity to link people to the information that will help them. They perform seemingly miraculous tasks humans cannot and they will continue to greatly augment human intelligence and assist in accomplishing great things. A representative proponent of this view is Stephen Downes, a researcher at the National Research Council of Canada, who listed the following as positive changes:
“Some examples: Banks. Today banks provide loans based on very incomplete data. It is true that many people who today qualify for loans would not get them in the future. However, many people – and arguably many more people – will be able to obtain loans in the future, as banks turn away from using such factors as race, socio-economic background, postal code and the like to assess fit. Moreover, with more data (and with a more interactive relationship between bank and client) banks can reduce their risk, thus providing more loans, while at the same time providing a range of services individually directed to actually help a person’s financial state.
“Health care providers. Health care is a significant and growing expense not because people are becoming less healthy (in fact, society-wide, the opposite is true) but because of the significant overhead required to support increasingly complex systems, including prescriptions, insurance, facilities and more. New technologies will enable health providers to shift a significant percentage of that load to the individual, who will (with the aid of personal support systems) manage their health better, coordinate and manage their own care, and create less of a burden on the system. As the overall cost of health care declines, it becomes increasingly feasible to provide single-payer health insurance for the entire population, which has known beneficial health outcomes and efficiencies.
“Governments. A significant proportion of government is based on regulation and monitoring, which will no longer be required with the deployment of automated production and transportation systems, along with sensor networks. This includes many of the daily (and often unpleasant) interactions we have with government today, from traffic offenses, manifestation of civil discontent, unfair treatment in commercial and legal processes, and the like. A simple example: One of the most persistent political problems in the United States is the gerrymandering of political boundaries to benefit incumbents. Electoral divisions created by an algorithm to a large degree eliminate gerrymandering (and when open and debatable, can be modified to improve on that result).” http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 6/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center A sampling of additional answers, from anonymous respondents:
“Algorithms find knowledge in an automated way much faster than traditionally feasible.” The efficiencies of algorithms will lead “Algorithms can crunch databases quickly enough to more creativity and self-expression. to alleviate some of the red tape and bureaucracy that currently slows progress down.”
“We will see less pollution, improved human health, less economic waste.”
“Algorithms have the potential to equalize access to information.”
“The efficiencies of algorithms will lead to more creativity and self-expression.”
“Algorithms can diminish transportation issues; they can identify congestion and alternative times and paths.”
“Self-driving cars could dramatically reduce the number of accidents we have per year, as well as improve quality of life for most people.”
“Better-targeted delivery of news, services and advertising.”
“More evidence-based social science using algorithms to collect data from social media and click trails.”
“Improved and more proactive police work, targeting areas where crime can be prevented.”
“Fewer underdeveloped areas and more international commercial exchanges.”
“Algorithms ease the friction in decision-making, purchasing, transportation and a large number of other behaviors.”
“Bots will follow orders to buy your stocks. Digital agents will find the materials you need.”
“Any errors could be corrected. This will mean the algorithms only become more efficient to humanity’s desires as time progresses.”
Themes illuminating concerns and challenges
Participants in this study were in substantial agreement that the abundant positives of accelerating code-dependency will continue to drive the spread of algorithms; however, as with all great technological revolutions, this trend has a dark side. Most respondents pointed out concerns, chief among them the final five overarching themes of this report; all have subthemes.
Theme 3: Humanity and human judgment are lost when data and predictive modeling become paramount
Advances in algorithms are allowing technology corporations and governments to gather, store, sort and analyze massive data sets. Experts in this canvassing noted that these algorithms are primarily written to optimize efficiency and profitability without much thought about the possible societal impacts of the data modeling and analysis. These respondents argued that humans are considered to be an “input” to the process and they are not seen as real, thinking, feeling, changing beings. They say this is creating a flawed, logic-driven society and that as the process evolves – that is, as algorithms begin to write the algorithms – humans may get left out of the loop, letting “the robots decide.” Representative of this view:
http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 7/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center Bart Knijnenburg, assistant professor in human-centered computing at Clemson University, replied, “Algorithms will capitalize on convenience and profit, thereby discriminating [against] certain populations, but also eroding the experience of everyone else. The goal of algorithms is to fit some of our preferences, but not necessarily all of them: They essentially present a caricature of our tastes and preferences. My biggest fear is that, unless we tune our algorithms for self-actualization, it will be simply too convenient for people to follow the advice of an algorithm (or, too difficult to go beyond such advice), turning these algorithms into self-fulfilling prophecies, and users into zombies who exclusively consume easy-to-consume items.”
An anonymous futurist said, “This has been going on since the beginning of the industrial revolution. Every time you design a human system optimized for efficiency or profitability you dehumanize the workforce. That dehumanization has now spread to our health care and social services. When you remove the humanity from a system where people are included, they become victims.”
Another anonymous respondent wrote, “We simply can’t capture every data element that represents the vastness of a person and that person’s needs, wants, hopes, desires. Who is collecting what data points? Do the human beings the data points reflect even know or did they just agree to the terms of service because they had no real choice? Who is making money from the data? How is anyone to know how his/her data is being massaged and for what purposes to justify what ends? There is no transparency, and oversight is a farce. It’s all hidden from view. I will always remain convinced the data will be used to enrich and/or protect others and not the individual. It’s the basic nature of the economic system in which we live.”
A sampling of excerpts tied to this theme from other respondents (for details, read the fuller versions (http://www.pewinternet.org/2017/02/08/theme-1-algorithms-will-continue-to-spread-everywhere/) in the full report):
“The potential for good is huge, but the potential for misuse and abuse – intentional, and Algorithms have the capability to shape inadvertent – may be greater.” individuals’ decisions without them “Companies seek to maximize profit, not maximize societal good. Worse, they repackage profit-seeking even knowing it, giving those who have as a societal good. We are nearing the crest of a control of the algorithms an unfair wave, the trough side of which is a new ethics of position of power. manipulation, marketing, nearly complete lack of privacy.”
“What we see already today is that, in practice, stuff like ‘differential pricing’ does not help the consumer; it helps the company that is selling things, etc.”
“Individual human beings will be herded around like cattle, with predictably destructive results on rule of law, social justice and economics.”
“There is an incentive only to further obfuscate the presence and operations of algorithmic shaping of communications processes.”
“Algorithms are … amplifying the negative impacts of data gaps and exclusions.”
“Algorithms have the capability to shape individuals’ decisions without them even knowing it, giving those who have control of the algorithms an unfair position of power.”
“The fact the internet can, through algorithms, be used to almost read our minds means [that] those who have access to the algorithms and their databases have a vast opportunity to manipulate large population groups.”
“The lack of accountability and complete opacity is frightening.” http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 8/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center “By utilitarian metrics, algorithmic decision-making has no downside; the fact that it results in perpetual injustices toward the very minority classes it creates will be ignored. The Common Good has become a discredited, obsolete relic of The Past.”
“In an economy increasingly dominated by a tiny, very privileged and insulated portion of the population, it will largely reproduce inequality for their benefit. Criticism will be belittled and dismissed because of the veneer of digital ‘logic’ over the process.”
“Algorithms are the new gold, and it’s hard to explain why the average ‘good’ is at odds with the individual ‘good.’”
“We will interpret the negative individual impact as the necessary collateral damage of ‘progress.’”
“This will kill local intelligence, local skills, minority languages, local entrepreneurship because most of the available resources will be drained out by the global competitors.”
“Algorithms in the past have been created by a programmer. In the future they will likely be evolved by intelligent/learning machines …. Humans will lose their agency in the world.”
“It will only get worse because there’s no ‘crisis’ to respond to, and hence, not only no motivation to change, but every reason to keep it going – especially by the powerful interests involved. We are heading for a nightmare.”
“Web 2.0 provides more convenience for citizens who need to get a ride home, but at the same time – and it’s naive to think this is a coincidence – it’s also a monetized, corporatized, disempowering, cannibalizing harbinger of the End Times. (I exaggerate for effect. But not by much.)”
Theme 4: Biases exist in algorithmically-organized systems
Two strands of thinking tie together here. One is that the algorithm creators (code writers), even if they strive for inclusiveness, objectivity and neutrality, build into their creations their own perspectives and values. The other is that the datasets to which algorithms are applied have their own limits and deficiencies. Even datasets with billions of pieces of information do not capture the fullness of people’s lives and the diversity of their experiences. Moreover, the datasets themselves are imperfect because they do not contain inputs from everyone or a representative sample of everyone. The two themes are advanced in these answers:
Justin Reich, executive director at the MIT Teaching Systems Lab, observed, “The algorithms will be primarily designed by white and Asian men – with data selected by these same privileged actors – for the benefit of consumers like themselves. Most people in positions of privilege will find these new tools convenient, safe and useful. The harms of new technology will be most experienced by those already disadvantaged in society, where advertising algorithms offer bail bondsman ads that assume readers are criminals, loan applications that penalize people for proxies so correlated with race that they effectively penalize people based on race, and similar issues.”
Dudley Irish, a software engineer, observed, “All, let me repeat that, all of the training data contains biases. Much of it either racial- or class-related, with a fair sprinkling of simply punishing people for not using a standard dialect of English. To paraphrase Immanuel Kant, out of the crooked timber of these datasets no straight thing was ever made.”
A sampling of quote excerpts tied to this theme from other respondents (for details, read the fuller versions (http://www.pewinternet.org/2017/02/08/theme-1-algorithms-will-continue-to-spread-everywhere/) in the full report):
“Algorithms are, by definition, impersonal and based on gross data and generalized assumptions. The people writing algorithms, even those grounded in data, are a non-representative subset of the population.” http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 9/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center “If you start at a place of inequality and you use algorithms to decide what is a likely outcome for a One of the greatest challenges of the person/system, you inevitably reinforce inequalities.” next era will be balancing protection of
“We will all be mistreated as more homogenous intellectual property in algorithms with than we are.” protecting the subjects of those “The result could be the institutionalization of algorithms from unfair discrimination biased and damaging decisions with the excuse of, and social engineering. ‘The computer made the decision, so we have to accept it.’”
“The algorithms will reflect the biased thinking of people. Garbage in, garbage out. Many dimensions of life will be affected, but few will be helped. Oversight will be very difficult or impossible.”
“Algorithms value efficiency over correctness or fairness, and over time their evolution will continue the same priorities that initially formulated them.”
“One of the greatest challenges of the next era will be balancing protection of intellectual property in algorithms with protecting the subjects of those algorithms from unfair discrimination and social engineering.”
“Algorithms purport to be fair, rational and unbiased but just enforce prejudices with no recourse.”
“Unless the algorithms are essentially open source and as such can be modified by user feedback in some fair fashion, the power that likely algorithm-producers (corporations and governments) have to make choices favorable to themselves, whether in internet terms of service or adhesion contracts or political biases, will inject both conscious and unconscious bias into algorithms.”
Theme 5: Algorithmic categorizations deepen divides
Two connected ideas about societal divisions were evident in many respondents’ answers. First, they predicted that an algorithm-assisted future will widen the gap between the digitally savvy (predominantly the most well-off, who are the most desired demographic in the new information ecosystem) and those who are not nearly as connected or able to participate. Second, they said social and political divisions will be abetted by algorithms, as algorithm-driven categorizations and classifications steer people into echo chambers of repeated and reinforced media and political content. Two illustrative answers:
Ryan Hayes, owner of Fit to Tweet, commented, “Twenty years ago we talked about the ‘digital divide’ being people who had access to a computer at home vs. those that didn’t, or those who had access to the internet vs. those who didn’t …. Ten years from now, though, the life of someone whose capabilities and perception of the world is augmented by sensors and processed with powerful AI and connected to vast amounts of data is going to be vastly different from that of those who don’t have access to those tools or knowledge of how to utilize them. And that divide will be self- perpetuating, where those with fewer capabilities will be more vulnerable in many ways to those with more.”
Adam Gismondi, a visiting scholar at Boston College, wrote, “I am fearful that as users are quarantined into distinct ideological areas, human capacity for empathy may suffer. Brushing up against contrasting viewpoints challenges us, and if we are able to (actively or passively) avoid others with different perspectives, it will negatively impact our society. It will be telling to see what features our major social media companies add in coming years, as they will have tremendous power over the structure of information flow.”
http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 10/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center A sampling of quote excerpts tied to this theme from other respondents (for details, read the fuller versions (http://www.pewinternet.org/2017/02/08/theme-1-algorithms-will-continue-to-spread-everywhere/) in the full report):
“If the current economic order remains in place, then I do not see the growth of data-driven The overall effect will be positive for algorithms providing much benefit to anyone outside of the richest in society.” some individuals. It will be negative for
“Social inequalities will presumably become the poor and the uneducated. As a reified.” result, the digital divide and wealth “The major risk is that less-regular users, especially disparity will grow. It will be a net those who cluster on one or two sites or platforms, negative for society. won’t develop that navigational and selection facility and will be at a disadvantage.”
“Algorithms make discrimination more efficient and sanitized. Positive impact will be increased profits for organizations able to avoid risk and costs. Negative impacts will be carried by all deemed by algorithms to be risky or less profitable.”
“Society will be stratified by which trust/identity provider one can afford/qualify to go with. The level of privacy and protection will vary. Lois McMaster [Bujold]’s Jackson’s Whole suddenly seems a little more chillingly realistic.”
“We have radically divergent sets of values, political and other, and algos are always rooted in the value systems of their creators. So the scenario is one of a vast opening of opportunity – economic and otherwise – under the control of either the likes of Zuckerberg or the grey-haired movers of global capital or ….”
“The overall effect will be positive for some individuals. It will be negative for the poor and the uneducated. As a result, the digital divide and wealth disparity will grow. It will be a net negative for society.”
“Racial exclusion in consumer targeting. Gendered exclusion in consumer targeting. Class exclusion in consumer targeting …. Nationalistic exclusion in consumer targeting.”
“If the algorithms directing news flow suppress contradictory information – information that challenges the assumptions and values of individuals – we may see increasing extremes of separation in worldviews among rapidly diverging subpopulations.”
“We may be heading for lowest-common-denominator information flows.”
“Efficiency and the pleasantness and serotonin that come from prescriptive order are highly overrated. Keeping some chaos in our lives is important.”
A number of participants in this canvassing expressed concerns over the change in the public’s information diets, the “atomization of media,” an over-emphasis of the extreme, ugly, weird news, and the favoring of “truthiness” over more- factual material that may be vital to understanding how to be a responsible citizen of the world.
Theme 6: Unemployment will rise
The spread of artificial intelligence (AI) has the potential to create major unemployment and all the fallout from that.
http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 11/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center An anonymous CEO said, “If a task can be effectively represented by an algorithm, then it can be easily performed by a machine. The negative trend I see here is that – with the rise of the algorithm – humans will be replaced by machines/computers for many jobs/tasks. What will then be the fate of Man?”
A sampling of quote excerpts tied to this theme from other respondents (for details, read the fuller versions (http://www.pewinternet.org/2017/02/08/theme-1-algorithms-will-continue-to-spread-everywhere/) in the full report):
“AI and robots are likely to disrupt the workforce to a potential 100% human unemployment. They will I foresee algorithms replacing almost all be smarter more efficient and productive and cost less, so it makes sense for corporations and workers with no real options for the business to move in this direction.” replaced humans. “The massive boosts in productivity due to automation will increase the disparity between workers and owners of capital.”
“Modern Western society is built on a societal model whereby Capital is exchanged for Labour to provide economic growth. If Labour is no longer part of that exchange, the ramifications will be immense.”
“No jobs, growing population and less need for the average person to function autonomously. Which part of this is warm and fuzzy?”
“I foresee algorithms replacing almost all workers with no real options for the replaced humans.”
“In the long run, it could be a good thing for individuals by doing away with low-value repetitive tasks and motivating them to perform ones that create higher value.”
“Hopefully, countries will have responded by implementing forms of minimal guaranteed living wages and free education past K-12; otherwise the brightest will use online resources to rapidly surpass average individuals and the wealthiest will use their economic power to gain more political advantages.”
Theme 7: The need grows for algorithmic literacy, transparency and oversight
The respondents to this canvassing offered a variety of ideas about how individuals and the broader culture might respond to the algorithm-ization of life. They argued for public education to instill literacy about how algorithms function in the general public. They also noted that those who create and evolve algorithms are not held accountable to society and argued there should be some method by which they are. Representative comments:
Susan Etlinger, industry analyst at Altimeter Group, said, “Much like the way we increasingly wish to know the place and under what conditions our food and clothing are made, we should question how our data and decisions are made as well. What is the supply chain for that information? Is there clear stewardship and an audit trail? Were the assumptions based on partial information, flawed sources or irrelevant benchmarks? Did we train our data sufficiently? Were the right stakeholders involved, and did we learn from our mistakes? The upshot of all of this is that our entire way of managing organizations will be upended in the next decade. The power to create and change reality will reside in technology that only a few truly understand. So to ensure that we use algorithms successfully, whether for financial or human benefit or both, we need to have governance and accountability structures in place. Easier said than done, but if there were ever a time to bring the smartest minds in industry together with the smartest minds in academia to solve this problem, this is the time.”
http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 12/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center Chris Kutarna, author of Age of Discovery and fellow at the Oxford Martin School, wrote, “Algorithms are an explicit form of heuristic, a way of routinizing certain choices and decisions so that we are not constantly drinking from a fire hydrant of sensory inputs. That coping strategy has always been co-evolving with humanity, and with the complexity of our social systems and data environments. Becoming explicitly aware of our simplifying assumptions and heuristics is an important site at which our intellects and influence mature. What is different now is the increasing power to program these heuristics explicitly, to perform the simplification outside of the human mind and within the machines and platforms that deliver data to billions of individual lives. It will take us some time to develop the wisdom and the ethics to understand and direct this power. In the meantime, we honestly don’t know how well or safely it is being applied. The first and most important step is to develop better social awareness of who, how, and where it is being applied.”
A sampling of quote excerpts tied to this theme from other respondents (for details, read the fuller versions (http://www.pewinternet.org/2017/02/08/theme-1-algorithms-will-continue-to-spread-everywhere/) in the full report):
“Who guards the guardians? And, in particular, which ‘guardians’ are doing what, to whom, using We need some kind of rainbow coalition the vast collection of information?” to come up with rules to avoid allowing “There are no incentives in capitalism to fight filter bubbles, profiling, and the negative effects, and inbuilt bias and groupthink to effect the governmental/international governance is virtually outcomes. powerless.”
“Oversight mechanisms might include stricter access protocols; sign off on ethical codes for digital management and named stewards of information; online tracking of an individual’s reuse of information; opt-out functions; setting timelines on access; no third-party sale without consent.”
“Unless there is an increased effort to make true information literacy a part of basic education, there will be a class of people who can use algorithms and a class used by algorithms.”
“Consumers have to be informed, educated, and, indeed, activist in their orientation toward something subtle. This is what computer literacy is about in the 21st century.”
“Finding a framework to allow for transparency and assess outcomes will be crucial. Also a need to have a broad understanding of the algorithmic ‘value chain’ and that data is the key driver and as valuable as the algorithm which it trains.”
“Algorithmic accountability is a big-tent project, requiring the skills of theorists and practitioners, lawyers, social scientists, journalists, and others. It’s an urgent, global cause with committed and mobilized experts looking for support.”
“Eventually, software liability law will be recognized to be in need of reform, since right now, literally, coders can get away with murder.”
“The Law of Unintended Consequences indicates that the increasing layers of societal and technical complexity encoded in algorithms ensure that unforeseen catastrophic events will occur – probably not the ones we were worrying about.”
“Eventually we will evolve mechanisms to give consumers greater control that should result in greater understanding and trust …. The pushback will be inevitable but necessary and will, in the long run, result in balances that are more beneficial for all of us.”
http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 13/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center “We need some kind of rainbow coalition to come up with rules to avoid allowing inbuilt bias and groupthink to effect the outcomes.”
“Algorithms are too complicated to ever be transparent or to ever be completely safe. These factors will continue to influence the direction of our culture.”
“I expect meta-algorithms will be developed to try to counter the negatives of algorithms.”
Anonymous respondents shared these one-liners on the topic:
“The golden rule: He who owns the gold makes the rules.”
“The bad guys appear to be way ahead of the good guys.”
“Resistance is futile.”
“Algorithms are defined by people who want to sell you something (goods, services, ideologies) and will twist the results to favor doing so.”
“Algorithms are surely helpful but likely insufficient unless combined with human knowledge and political will.”
Finally, this prediction from an anonymous participant who sees the likely endpoint to be one of two extremes:
“The overall impact will be utopia or the end of the human race; there is no middle ground foreseeable. I suspect utopia given that we have survived at least one existential crisis (nuclear) in the past and that our track record toward peace, although slow, is solid.”
Key experts’ thinking about the future impacts of algorithms
Following is a brief collection of comments by several of the many top analysts who participated in this canvassing:
‘Steering people to useful information’
Vinton Cerf, Internet Hall of Fame member and vice president and chief internet evangelist at Google: “Algorithms are mostly intended to steer people to useful information and I see this as a net positive.”
Beware ‘unverified, untracked, unrefined models’
Cory Doctorow, writer, computer science activist-in-residence at MIT Media Lab and co-owner of Boing Boing, responded, “The choices in this question are too limited. The right answer is, ‘If we use machine learning models rigorously, they will make things better; if we use them to paper over injustice with the veneer of machine empiricism, it will be worse.’ Amazon uses machine learning to optimize its sales strategies. When they make a change, they make a prediction about its likely outcome on sales, then they use sales data from that prediction to refine the model. Predictive sentencing scoring contractors to America’s prison system use machine learning to optimize sentencing recommendation. Their model also makes predictions about likely outcomes (on reoffending), but there is no tracking of whether their model makes good predictions, and no refinement. This frees them to make terrible predictions without consequence. This characteristic of unverified, untracked, unrefined models is present in many places: terrorist watchlists; drone-killing profiling models; modern redlining/Jim Crow systems that limit credit; predictive policing algorithms; etc. If we mandate, or establish normative limits, on practices that correct this sleazy conduct,
http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 14/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center then we can use empiricism to correct for bias and improve the fairness and impartiality of firms and the state (and public/private partnerships). If, on the other hand, the practice continues as is, it terminates with a kind of Kafkaesque nightmare where we do things ‘because the computer says so’ and we call them fair ‘because the computer says so.’”
‘A general trend toward positive outcomes will prevail’
Jonathan Grudin, principal researcher at Microsoft, said, “We are finally reaching a state of symbiosis or partnership with technology. The algorithms are not in control; people create and adjust them. However, positive effects for one person can be negative for another, and tracing causes and effects can be difficult, so we will have to continually work to understand and adjust the balance. Ultimately, most key decisions will be political, and I’m optimistic that a general trend toward positive outcomes will prevail, given the tremendous potential upside to technology use. I’m less worried about bad actors prevailing than I am about unintended and unnoticed negative consequences sneaking up on us.”
‘Faceless systems more interested in surveillance and advertising than actual service’
Doc Searls, journalist, speaker and director of Project VRM at Harvard University’s Berkman Center, wrote, “The biggest issue with algorithms today is the black-box nature of some of the largest and most consequential ones. An example is the one used by Dun & Bradstreet to decide credit worthiness. The methods behind the decisions it makes are completely opaque, not only to those whose credit is judged, but to most of the people running the algorithm as well. Only the programmers are in a position to know for sure what the algorithm does, and even they might not be clear about what’s going on. In some cases there is no way to tell exactly why or how a decision by an algorithm is reached. And even if the responsible parties do know exactly how the algorithm works, they will call it a trade secret and keep it hidden. There is already pushback against the opacity of algorithms, and the sometimes vast systems behind them. Many lawmakers and regulators also want to see, for example, Google’s and Facebook’s vast server farms more deeply known and understood. These things have the size, scale, and in some ways the importance of nuclear power plants and oil refineries, yet enjoy almost no regulatory oversight. This will change. At the same time, so will the size of the entities using algorithms. They will get smaller and more numerous, as more responsibility over individual lives moves away from faceless systems more interested in surveillance and advertising than actual service.”
A call for #AlgorithmicTransparency
Marc Rotenberg, executive director of the Electronic Privacy Information Center, observed, “The core problem with algorithmic-based decision-making is the lack of accountability. Machines have literally become black boxes – even the developers and operators do not fully understand how outputs are produced. The problem is further exacerbated by ‘digital scientism’ (my phrase) – an unwavering faith in the reliability of big data. ‘Algorithmic transparency’ should be established as a fundamental requirement for all AI-based decision-making. There is a larger problem with the increase of algorithm-based outcomes beyond the risk of error or discrimination – the increasing opacity of decision- making and the growing lack of human accountability. We need to confront the reality that power and authority are moving from people to machines. That is why #AlgorithmicTransparency is one of the great challenges of our era.”
The data ‘will be misused in various ways’
Richard Stallman, Internet Hall of Fame member and president of the Free Software Foundation, said, “People will be pressured to hand over all the personal data that the algorithms would judge. The data, once accumulated, will be misused in various ways – by the companies that collect them, by rogue employees, by crackers that steal the data from the company’s site, and by the state via National Security Letters. I have heard that people who refuse to be used by Facebook are discriminated against in some ways. Perhaps soon they will be denied entry to the U.S., for instance. Even if the U.S. doesn’t actually do that, people will fear that it will. Compare this with China’s social obedience score for internet users.”
http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 15/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center People must live with outcomes of algorithms ‘even though they are fearful of the risks’
David Clark, Internet Hall of Fame member and senior research scientist at MIT, replied, “I see the positive outcomes outweighing the negative, but the issue will be that certain people will suffer negative consequences, perhaps very serious, and society will have to decide how to deal with these outcomes. These outcomes will probably differ in character, and in our ability to understand why they happened, and this reality will make some people fearful. But as we see today, people feel that they must use the internet to be a part of society. Even if they are fearful of the consequences, people will accept that they must live with the outcomes of these algorithms, even though they are fearful of the risks.”
‘EVERY area of life will be affected. Every. Single. One.’
Baratunde Thurston, Director’s Fellow at MIT Media Lab, Fast Company columnist, and former digital director of The Onion, wrote: “Main positive changes: 1) The excuse of not knowing things will be reduced greatly as information becomes even more connected and complete. 2) Mistakes that result from errors in human judgment, ‘knowledge,’ or reaction time will be greatly reduced. Let’s call this the ‘robots drive better than people’ principle. Today’s drivers will whine, but in 50 years no one will want to drive when they can use that transportation time to experience a reality- indistinguishable immersive virtual environment filled with a bunch of Beyoncé bots.
“3) Corruption that exists today as a result of human deception will decline significantly—bribes, graft, nepotism. If the algorithms are built well and robustly, the opportunity to insert this inefficiency (e.g., hiring some idiot because he’s your cousin) should go down. 4) In general, we should achieve a much more efficient distribution of resources, including expensive (in dollars or environmental cost) resources like fossil fuels. Basically, algorithmic insight will start to affect the design of our homes, cities, transportation networks, manufacturing levels, waste management processing, and more. There’s a lot of redundancy in a world where every American has a car she never uses. We should become far more energy efficient once we reduce the redundancy of human-drafted processes.
“But there will be negative changes: 1) There will be an increased speed of interactions and volume of information processed—everything will get faster. None of the efficiency gains brought about by technology has ever lead to more leisure or rest or happiness. We will simply shop more, work more, decide more things because our capacity to do all those will have increased. It’s like adding lanes to the highway as a traffic management solution. When you do that, you just encourage more people to drive. The real trick is to not add more car lanes but build a world in which fewer people need or want to drive.
“2) There will be algorithmic and data-centric oppression. Given that these systems will be designed by demonstrably imperfect and biased human beings, we are likely to create new and far less visible forms of discrimination and oppression. The makers of these algorithms and the collectors of the data used to test and prime them have nowhere near a comprehensive understanding of culture, values, and diversity. They will forget to test their image recognition on dark skin or their medical diagnostic tools on Asian women or their transport models during major sporting events under heavy fog. We will assume the machines are smarter, but we will realize they are just as dumb as we are but better at hiding it.
“3) Entire groups of people will be excluded and they most likely won’t know about the parallel reality they don’t experience. Every area of life will be affected. Every. Single. One.”
A call for ‘industry reform’ and ‘more savvy regulatory regimes’
http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 16/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center Technologist Anil Dash said, “The best parts of algorithmic influence will make life better for many people, but the worst excesses will truly harm the most marginalized in unpredictable ways. We’ll need both industry reform within the technology companies creating these systems and far more savvy regulatory regimes to handle the complex challenges that arise.”
‘We are a society that takes its life direction from the palm of our hands’
John Markoff, author of Machines of Loving Grace: The Quest for Common Ground Between Humans and Robots and senior writer at The New York Times, observed, “I am most concerned about the lack of algorithmic transparency. Increasingly we are a society that takes its life direction from the palm of our hands – our smartphones. Guidance on everything from what is the best Korean BBQ to who to pick for a spouse is algorithmically generated. There is little insight, however, into the values and motives of the designers of these systems.”
Fix the ‘organizational, societal and political climate we’ve constructed’
danah boyd, founder of Data & Society, commented, “An algorithm means nothing by itself. What’s at stake is how a ‘model’ is created and used. A model is comprised of a set of data (e.g., training data in a machine learning system) alongside an algorithm. The algorithm is nothing without the data. But the model is also nothing without the use case. The same technology can be used to empower people (e.g., identify people at risk) or harm them. It all depends on who is using the information to what ends (e.g., social services vs. police). Because of unhealthy power dynamics in our society, I sadly suspect that the outcomes will be far more problematic – mechanisms to limit people’s opportunities, segment and segregate people into unequal buckets, and leverage surveillance to force people into more oppressive situations. But it doesn’t have to be that way. What’s at stake has little to do with the technology; it has everything to do with the organizational, societal and political climate we’ve constructed.”
We have an algorithmic problem already: Credit scores
Henning Schulzrinne, Internet Hall of Fame member and professor at Columbia University, noted, “We already have had early indicators of the difficulties with algorithmic decision-making, namely credit scores. Their computation is opaque and they were then used for all kinds of purposes far removed from making loans, such as employment decisions or segmenting customers for different treatment. They leak lots of private information and are disclosed, by intent or negligence, to entities that do not act in the best interest of the consumer. Correcting data is difficult and time-consuming, and thus unlikely to be available to individuals with limited resources. It is unclear how the proposed algorithms address these well-known problems, given that they are often subject to no regulations whatsoever. In many areas, the input variables are either crude (and often proxies for race), such as home ZIP code, or extremely invasive, such as monitoring driving behavior minute-by-minute. Given the absence of privacy laws, in general, there is every incentive for entities that can observe our behavior, such as advertising brokers, to monetize behavioral information. At minimum, institutions that have broad societal impact would need to disclose the input variables used, how they influence the outcome and be subject to review, not just individual record corrections. An honest, verifiable cost-benefit analysis, measuring improved efficiency or better outcomes against the loss of privacy or inadvertent discrimination, would avoid the ‘trust us, it will be wonderful and it’s AI!’ decision-making.”
Algorithms ‘create value and cut costs’ and will be improved
Robert Atkinson, president of the Information Technology and Innovation Foundation, said, “Like virtually all past technologies, algorithms will create value and cut costs, far in excess of any costs. Moreover, as organizations and society get more experience with use of algorithms there will be natural forces toward improvement and limiting any potential problems.”
‘The goal should be to help people question authority’ http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 17/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center Judith Donath of Harvard Berkman Klein Center for Internet & Society, replied, “Data can be incomplete, or wrong, and algorithms can embed false assumptions. The danger in increased reliance on algorithms is that is that the decision-making process becomes oracular: opaque yet unarguable. The solution is design. The process should not be a black box into which we feed data and out comes an answer, but a transparent process designed not just to produce a result, but to explain how it came up with that result. The systems should be able to produce clear, legible text and graphics that help the users – readers, editors, doctors, patients, loan applicants, voters, etc. – understand how the decision was made. The systems should be interactive, so that people can examine how changing data, assumptions, rules would change outcomes. The algorithm should not be the new authority; the goal should be to help people question authority.”
Do more to train coders with diverse world views
Amy Webb, futurist and CEO at the Future Today Institute, wrote, “In order to make our machines think, we humans need to help them learn. Along with other pre-programmed training datasets, our personal data is being used to help machines make decisions. However, there are no standard ethical requirements or mandate for diversity, and as a result we’re already starting to see a more dystopian future unfold in the present. There are too many examples to cite, but I’ll list a few: would-be borrowers turned away from banks, individuals with black-identifying names seeing themselves in advertisements for criminal background searches, people being denied insurance and health care. Most of the time, these problems arise from a limited worldview, not because coders are inherently racist. Algorithms have a nasty habit of doing exactly what we tell them to do. Now, what happens when we’ve instructed our machines to learn from us? And to begin making decisions on their own? The only way to address algorithmic discrimination in the future is to invest in the present. The overwhelming majority of coders are white and male. Corporations must do more than publish transparency reports about their staff – they must actively invest in women and people of color, who will soon be the next generation of workers. And when the day comes, they must choose new hires both for their skills and their worldview. Universities must redouble their efforts not only to recruit a diverse body of students –administrators and faculty must support them through to graduation. And not just students. Universities must diversify their faculties, to ensure that students see themselves reflected in their teachers.”
The impact in the short term will be negative; in the longer term it will be positive
Jamais Cascio, distinguished fellow at the Institute for the Future, observed, “The impact of algorithms in the early transition era will be overall negative, as we (humans, human society and economy) attempt to learn how to integrate these technologies. Bias, error, corruption and more will make the implementation of algorithmic systems brittle, and make exploiting those failures for malice, political power or lulz (http://knowyourmeme.com/memes/i-did-it-for-the-lulz) comparatively easy. By the time the transition takes hold – probably a good 20 years, maybe a bit less – many of those problems will be overcome, and the ancillary adaptations (e.g., potential rise of universal basic income) will start to have an overall benefit. In other words, shorter term (this decade) negative, longer term (next decade) positive.”
The story will keep shifting
Mike Liebhold, senior researcher and distinguished fellow at the Institute for the Future, commented, “The future effects of algorithms in our lives will shift over time as we master new competencies. The rates of adoption and diffusion will be highly uneven, based on natural variables of geographies, the environment, economies, infrastructure, policies, sociologies, psychology, and – most importantly – education. The growth of human benefits of machine intelligence will be most constrained by our collective competencies to design and interact effectively with machines. At an absolute minimum, we need to learn to form effective questions and tasks for machines, how to interpret responses and how to simply detect and repair a machine mistake.”
Make algorithms ‘comprehensible, predictable and controllable’
http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 18/19 3/5/2018 Experts on the Pros and Cons of Algorithms | Pew Research Center Ben Shneiderman, professor of computer science at the University of Maryland, wrote, “When well-designed, algorithms amplify human abilities, but they must be comprehensible, predictable and controllable. This means they must be designed to be transparent so that users can understand the impacts of their use and they must be subject to continuing evaluation so that critics can assess bias and errors. Every system needs a responsible contact person/organization that maintains/updates the algorithm and a social structure so that the community of users can discuss their experiences.”
In key cases, give the user control
David Weinberger, senior researcher at the Harvard Berkman Klein Center for Internet & Society, said, “Algorithmic analysis at scale can turn up relationships that are predictive and helpful even if they are beyond the human capacity to understand them. This is fine where the stakes are low, such as a book recommendation. Where the stakes are high, such as algorithmically filtering a news feed, we need to be far more careful, especially when the incentives for the creators are not aligned with the interests of the individuals or of the broader social goods. In those latter cases, giving more control to the user seems highly advisable.”
MAKE A FINANCIAL CONTRIBUTION TO SUPPORT OUR WORK
DONATE
http://www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age/ 19/19 3/5/2018 How AI is Helping HR Gauge Employee Sentiment in Real Time - Dell Technologies
ARTIFICIAL INTELLIGENCE Transformative THowechnology AI is Helping HR Gauge Employee Sentiment in Real Time
November 28, 2017
By Anne Shaw, Contributor
Employee retention, especially in the technology and healthcare sectors, has become a top concern for today’s organizations. One way for companies to stay on top of what matters most to their workforce, ensuring their employees come to work motivated and enthusiastic each day, is to better understand employee sentiment.
Today, many organizations use annual employee surveys to assess workplace satisfaction, engagement, and culture. And while comprehensive, these types of surveys eat up much of Human Resources’ time and, for all that HR departmental work, quickly became outdated.
By the time HR teams can scrutinize data and turn feedback into valuable insights, new issues arise that won’t show up until the next year’s annual survey.
To capture timely feedback and better understand real-time employee sentiment, many companies—from LinkedIn to Mercy Health—have begun to turn to artificial intelligence (AI) and machine learning (ML) solutions.
Spotting Intelligent Trends
Software suites, such as Glint and Culture Amp, offer automated and customized pulse surveys— that is, shorter and more frequent employee surveys that help executives obtain a holistic picture of their employees’ engagement levels. Data analytics – or people analytics – features included in https://www.delltechnologies.com/en-us/perspectives/how-ai-is-helping-hr-gauge-employee-sentiment-in-real-time/ 1/7 3/5/2018 How AI is Helping HR Gauge Employee Sentiment in Real Time - Dell Technologies these apps employ machine learning algorithms that categorize common feedback phrases to identify larger trends.
Algorithms comb through, organize and analyze survey data—including qualitative employee feedback. Leadership teams then get on-demand access to actionable insights on what employees both appreciate and criticize. Based on research by SAP, intelligent HR software creates an opportunity to map HR transactions to business value by giving leaders the insights necessary to proactively address issues and enhance positive sentiment.
Advanced intelligence companies like Glint, for example, identify which factors are linked to negative experiences and higher turnover, thereby reducing attrition.
Empowering the People
The key for companies tapping into sentiment technology lies in its data. Depending on the software type, the data analytics tools can layer in organizational and departmental context—for instance, tenure of employee groups, average time between promotions, and which leader a department reports to. The technology then identifies real-time trends in employee sentiment, for example, a desire for a more structured professional development program or a new type of benefit. With intelligently collected and organized data presented in an easy, interactive format, mid-level managers are empowered to address the issues most important to their specific teams.
“We want to make [HR] decisions based off of data,” Ben Hatch, the employee engagement manager at LinkedIn shared in a Glint testimonial. “We want to be able to not just say, ‘Hey, it feels like this is what our employees are telling us.’ We want to be able to know what our employees are telling us, and it’s X.'”
https://www.delltechnologies.com/en-us/perspectives/how-ai-is-helping-hr-gauge-employee-sentiment-in-real-time/ 2/7 3/5/2018 How AI is Helping HR Gauge Employee Sentiment in Real Time - Dell Technologies
LinkedIn's Innovative Approach to a Happy & Successful Workforce from Glint Inc.
02:05
For Michelle Deneau, Intuit’s director of HR business intelligence, Glint saves her team hours of sifting through innumerable employee comments to understand Intuit’s employee experience.
“It’s one of those technologies that once you start using it, you wonder what you did for so long without it,” she said in a recent Glint announcement.
Some companies with the right internal talent have even built their own AI tech solutions, allowing them to customize the software for their exact goals and culture. For instance, Loka, a company that helps other companies develop mobile apps, created Jane, an AI-powered chatbot to provide real-time answers to common HR questions.
With Jane in place, Loka could automate a bulk of its HR work at the same time it could monitor commonly-asked questions. By applying sentiment analysis to Jane, the bot was able to determine when to raise a red flag to Loka’s HR leaders.
With the help of the evolving sentiment applications, organizations are achieving a closer connection to their employee experience. They are then using this improved insight to empower employees, nurture their highest performers, and achieve higher productivity levels.
Would you like to read more like this?
https://www.delltechnologies.com/en-us/perspectives/how-ai-is-helping-hr-gauge-employee-sentiment-in-real-time/ 3/7 3/5/2018 How AI is Helping HR Gauge Employee Sentiment in Real Time - Dell Technologies
CHANGE MANAGEMENT VIRTUAL REALITY How One Doctor’s Approach to Change Management How Farmers Insurance Transformed Its Employee- Applies to Your Organization (and Saves Lives) Training Program Using Virtual Reality
February 15, 2018 January 16, 2018 By Marty Graham, Contributor When radiologist By harnessing virtual reality, businesses are and researcher Dr. Alexander Norbash made his saving thousands in training-related costs. Read first forays into leadership at pioneering how Farmers Insurance is using VR to train medical research hospitals, … claims representatives, giving them “hands-on” practice while saving thousands in travel costs.
ORGANIZATIONAL CULTURE The Employee Feedback Strategy Stirring a Revolution
December 5, 2017 Progressive companies are opting for continuous employee feedback models, where managers provide regular reviews vs. annual. Read the risks and rewards. https://www.delltechnologies.com/en-us/perspectives/how-ai-is-helping-hr-gauge-employee-sentiment-in-real-time/ 6/7 3/5/2018 How AI is Helping HR Gauge Employee Sentiment in Real Time - Dell Technologies
Privacy Statement
Ads & Emails
Manage Cookies
© 2018 Dell Inc.
https://www.delltechnologies.com/en-us/perspectives/how-ai-is-helping-hr-gauge-employee-sentiment-in-real-time/ 7/7
Sign In | Registe r FEATURE How arti cial intelligence can eliminate bias in hiring
AI and machine learning can help identify diverse candidates, improve the hiring pipeline and eliminate unconscious bias.
By Sharon Florentine Senior Writer, CIO DEC 22, 2016 2:00 AM PT
Diversity (or lack of it) is still a major challenge for tech companies. Poised to revolutionize the world of work in general, some organizations are leveraging technology to root out bias, better identify and screen candidates and help close the diversity gap.
That starts with understanding the nature of bias, and acknowledging that unconscious bias is a major problem, says Kevin Mulcahy, an analyst with Future Workplace and co-author of The Future Workplace Experience: 10 Rules for Managing Disruption in Recruiting and Engaging Employees. AI and machine learning can be an objective observer to screen for bias patterns, Mulcahy says.
"The challenge with unconscious bias is that, by definition, it is unconscious, so it takes a third- party, such as AI, to recognize those occurrences and point out any perceived patterns of bias. AI- enabled analysis of communication patterns about the senders or receivers -- like gender or age -- can be used to screen for bias patterns and present the pattern analysis back to the originators," Mulcahy says.
[ Related story: How gender-neutral job postings decrease time-to-hire ]
Setting strategy
Once bias is identified, though, how can companies address it? The first steps are to identify that bias is an issue and set strategies at all levels of the organization to address and change it, Mulcahy says. "Co mpanies deal with unconscious bias in various ways, beginning with recognizing that it exists, Sign In | Register stating a desire to change and setting strategies and goals to accomplish and track changes. One key to addressing observable bias is to tie examples of bias to their promotions and bonus payments. Simultaneously, you can explicitly demonstrate that examples of inclusion will help promotion and earn bonuses," Mulcahy says.
It's also important to measure and track improvement as well as negative patterns of behavior, Mulcahy says. AI interfaces can be a great source of objective measurements of patterns of behavior between workers. But it can't all be left to technology, Mulcahy says.
"You have to create a culture of 'If you see something, say something' that goes as high up as executive leadership. Do not expect lower-ranking people to call out examples of bias if senior people have not provided permission and led by example. You can still use AI to help; machines know no hierarchy, and can provide analytical reports back to workers of all levels, but there has to be a human element," he says.
Accountability, too, is key in changing patterns of bias. Everyone in an organization must take some responsibility for sourcing and promoting individuals who are different from themselves, and AI can help by analyzing patterns in new hires or promotions where workers are either recommending candidates that are measurably the same or measurably different than themselves, Mulcahy says.
"This type of exception reporting can objectively hold managers more accountable for explaining, in writing, the justification for any hire that is similar in age, gender and race to themselves. Shifting the burden of detecting patterns of sameness to a machine, while simultaneously placing the burden on the hiring managers for explaining sameness quickly exposes weak arguments for continued sameness and sends a more powerful message that diversity is valued," he says.
[ Related story: Battling gender bias in IT ]
Bias-free solutions
Of course, you have to make sure that any AI or machine-learning products used in this way are themselves free of bias -- or as free as possible, says Aman Alexander, product management director, CEB, an algorithmic assessment platform for recruiting and hiring. "AI /machine learning can help close the diversity gap, as long as it is not susceptible to human bias. Sign In | Register For example, recruiting contact center employees could provide AI/machine learning models with the historical application forms of hired contact center employees with high customer satisfaction scores. This allows the model to pick up on the subtle application attributes/traits and not be impacted by on-the-job, human biases," Alexander says.
By simply using an automated, objective process like this, it's possible to drastically reduce the scope for human bias. If, for example, fairly trained AI/machine learning tools are used to whittle an applicant pool down from 100 applicants to the final 10 interviewees, that means that 90 percent of the pool reduction would be done in a process immune to any human biases, Alexander explains.
[ Related story: 5 ways to reduce bias in your hiring practices ]
Unintentional outcomes
There are, however, some unintentional adverse impacts you must screen for andeliminate, Alexander says. "Say an AI/machine learning model could accurately pick up on a statistical relationship between college football quarterbacks and high performance in sales roles. That relationship could be predicated on entirely meritocratic causative factors -- the quarterback's role demands a combination of mental skill, decision making and influencing skills to lead a team, which translates well into high-pressure sales roles. Of course, an unintended consequence of that is excluding female applicants, by virtue of the fact that only men can be college football quarterbacks. The algorithm would then be favoring a trait which it would never find in women, thus disadvantaging them," he says.
The great thing about technology like this, though, is that unlike human biases, it's much easier to audit and remove those biases, he says. Algorithms can be tested by using them to score thousands of applicants, and then testing the demographic and/or gender breakdown of those applicants.
"If, on average, across all the traits the algorithm values it is disproportionately excluding women or any other group of people, the algorithm can be identified and corrected iteratively and seamlessly," Alexander says. In a ddition to the general potential for AI to identify unconscious bias, some technology Sign In | Register companies are leveraging AI and machine learning to address specific manifestations of bias, like gendered language in job descriptions, identifying diverse candidates at the top of the funnel and sourcing candidates that already exist in a company's applicant tracking system.
Watch your language
Companies often blame their lack of diversity on a shortage of qualified candidates. Using AI and machine learning to help organizations remove gender-biased language from job descriptions can diversify your applicant pool and therefore improve the chances of hiring female technologists, says Kieran Snyder, CEO and co-founder of Textio, a machine learning platform that analyzes language patterns.
The concept of "gendered" job listings refers to the use of male- or female-skewing terms within job descriptions. This idea has been gaining recognition since it was researched by the American Psychological Association, whose findings illustrated how some seemingly innocuous words could actually signal a gender bias in job ads, says Snyder, and that can impact the number of women who apply for open roles as well as the number who are eventually hired.
"It stands to reason that if you reach a wider pool of applicants, , you're much more likely to have more applicants which, in turn, improves diversity and speeds up the recruiting and hiring process," Snyder says.
Textio's predictive index uses data from more than 45 million job posts and combines that data with hiring outcomes to gauge the hiring performance and the gender tone of job postings, says Snyder. The tool is designed to make a quantitative judgment about the effectiveness of the language used in job postings and help companies tweak them to attract better, more diverse candidates, Snyder says.
Look internally rst
AI and machine learning can also help uncover candidates you may already have stored in your ATS to improve your pipeline and speed up search, says Steve Goodman, CEO and founder of talent rediscovery software company Restless Bandit. "W e're an AI and data analytics company, and our tools hook into your ATS or your HR systems, look Sign In | Register at the applications and résumés you've already got and then compare those to open job descriptions." Restless Bandit's product removes unconscious bias by eliminating names, geographical information and any other information about a candidate that could trigger bias, unconscious or otherwise, he says.
Companies are sitting on a treasure trove of applications, resumes and potential talent and yet, they continue to pay recruiters to search far and wide for candidates, Goodman says. Not only does Restless Bandit's tool help make hiring more efficient, it improves an employer's brand in the eyes of potential candidates, he says.
"How many times have you heard, 'Thanks, we'll keep your resume on file' and then never hear from that company again? It's a huge problem. Your application just disappears into the abyss. But imagine how the candidate feels about you if you can call them and say, 'let's look at these other jobs you're qualified for, even if they aren't what you applied to' -- it's showing you really pay attention to them," Goodman says.
Recruiting relevance
Finally, AI and machine learning are being applied to perform more targeted, relevant candidate searches, says Shon Burton, CEO of HiringSolved, a company that searches the web for publicly available candidate data, and then compiles that into candidate profiles, Burton says.
"The resume is dead. Ten years ago, we would get all the information about a candidate from a resume; now, we get it from the web, and it's constantly updated -- it's a high-frequency data stream. What we do is look at all publicly available data across the web and our software looks for certain relevance layers to find talent," Burton says.
It's different than a Boolean search filter, which requires much more manual input and effort and is based on eliminating "wrong" answers from a data set. HiringSolved's algorithms rank potential candidates based on information from their public profiles and how relevant that information is to the particular search parameters of the specific job description, he says.
"If you are going out and trying to identify candidates, you have a massive-scale data problem right off the bat -- you're looking at something like a billion social profiles, and you have to determine what's relevant, what's not, what is information about the same person, what's out of date, and ma ke inferences about that data," Burton says. Using AI and machine learning to search speeds up Sign In | Register the process and makes it more efficient, he says, and makes it easier to find diverse candidates at top of the hiring funnel.
If a company's looking for a female Linux systems administrator, for example, the software can search for and screen all potential female candidates with the relevant amount of experience, the right credentials and education and put them in front of a hiring manager or recruiter. It goes beyond just looking at job titles and past work experience, too, and can focus on specific skills that could make someone a great sysadmin with Linux experience, opening up the potential talent pool because you're opting people in instead of filtering people out, Burton says.
"What is then up to the humans involved is to make inferences. How old is the data? How reliable is it? That kind of thing. But the technology can greatly speed up the process," he says.
Future potential
One of the largest advantages AI/machine learning tools have over human-only processes is that they can be empirically trained and validated, says CEB's Alexander. Sometimes, the traits companies thought mattered most in quality hires turn out to be inconsequential, while traits considered "negative" actually make candidates more likely to succeed. With AI and machine learning technology, companies can rely solely on actual statistical relationships for these decisions, he says.
"The human mind is not designed for the type of pattern recognition that can be most helpful in making hiring decisions. For example, most people would be able to rattle off a list of the many traits they desire or avoid in an ideal candidate, but would have no idea what the relative success or failure rate is of people who exhibit those traits. They therefore don't have any data to justify their beliefs," he says. AI and machine learning analysis, however, can provide hard data that either confirms or denies recruiters', hiring managers' or executives' beliefs about the types of hires they should be making, he says.
Potential for these solutions
Many of these technologies are already deployed in the market and selling to Fortune 100 companies, and their potential scope and applicability is only going to increase, says Alexander. "Th ere are some major contributing factors at play. Greater data capture by companies' ATS and HR Sign In | Register information systems means greater capabilities for AI and machine learning tech to provide meaningful impact. More data accessibility through semantic and natural language programming tools have enabled us to break down and unlock the meaning and data within unstructured data like free form text, or data in messy data formats. This provides for richer data sets upon which to train algorithms. Cheaper compute power with scalable cloud computing and increased by use from the HR and recruiting community are also helping drive adoption," he says.
Whether by eliminating unconscious bias in general or attacking specific manifestations of bias in recruiting, screening and hiring talent, AI and machine learning has potential to level the playing field for women and other underrepresented minorities and provide a competitive advantage for companies, says Victoria Espinel, CEO of industry advocacy group BSA The Software Alliance. These technologies can play a major role in opening up career and economic opportunities to as many people as possible, and specifically compensating for biases in order to reduce or remove them, she says.
"I think there's enormous potential for these kinds of technologies, and it's really important to apply them in this way. Software can definitely play a role in mitigating any negative impact of our biases and making sure there's a wide range of perspectives and creative thinking that goes into designing products and solutions so they can be accessible to the widest range of people," Espinel says.
Related Video
BORN IN SILICON ALLEY Sign In | Register x.ai is an AI assistant that schedules your meetings (3:26)
Next read this:
The best ERP systems: 10 enterprise resource planning tools compared The death of long-term IT planning 10 old-school IT principles that still rule 9 warning signs of bad IT architecture The working dead: IT jobs bound for extinction The top 6 governance, risk and compliance (GRC) certifications 12 'best practices' IT should avoid at all costs Why IT projects still fail The skills and traits of a next-generation CIO 9 lies CIOs tell themselves The secret to being a superhero CIO? A kickass sidekick
Sharon Florentine covers IT careers, women in technology and diversity and inclusion for CIO.com.
Follow ✉
NEW! Download the Winter 2018 digital edition of CIO magazine
YOU MIGHT LIKE A Ads by Revcontent
It's Scary What This Crazy New Rule in This Site Knows Method "Ends" Washington, 7 T ime Lottery About You - Ear Ringing District of Nature's Best Sign In | Register Winner Says TruthFinder Silencis Complete Better Finances Kept Weight- "You're All loss Secret Has
Financial Review 247 Buzz Leaks
Brilliant The Stars of Penny Stock Strange Link Robin Williams' Startup Leaves Harry Potter: Formula Between Eggs Final Net Worth Washington, Where Are They Designed to Be and Diabetes Stuns His
EverQuote MoneyVersed Agora Financial Vedda Blood Sugar Reme… Interesticle
Copyright © 2018 IDG Communications, Inc. 3/5/2018 Inside the surprisingly sexist world of artificial intelligence — Quartz Your guide to intelligent enterprise is here. Introducing intelligent enterprise unleashed. #TechVision2018 new applied now Explore the trends #TechVision2018
ROAD TO BETTER ROBOTS Inside the surprisingly sexist world of artificial intelligence Sarah Todd October 25, 2015
China’s party summit, Oscar winners, penguin supercolony. All this and more in today's Daily Brief. https://qz.com/531257/inside-the-surprisingly-sexist-world-of-artificial-intelligence/ 1/7 3/5/2018 Inside the surprisingly sexist world of artificial intelligence — Quartz
We need more women working on the R2D2s of the future. (AP Photo/Kevin Wolf)
This post has been updated.
There’s no doubt Stephen Hawking is a smart guy. But the world-famous theoretical physicist recently declared that women leave him stumped.
“Women should remain a mystery,” Hawking wrote in response to a Reddit user’s question about the realm of the unknown that intrigued him most. While Hawking’s remark was meant to be light-hearted, he sounded quite serious discussing the potential dangers of arti cial intelligence during Reddit’s online Q&A session:
The real risk with AI isn’t malice but competence. A superintelligent AI will be extremely good at accomplishing its goals, and if those goals aren’t aligned with ours, we’re in trouble.
Hawking’s comments might seem unrelated. But according to some women at the forefront of computer science, together they point to an unsettling truth. Right now, the real danger in the world of arti cial intelligence isn’t the threat of robot overlords—it’s a startling lack of diversity.
China’s party summit, Oscar winners, penguin supercolony. All this and more in today's Daily Brief. https://qz.com/531257/inside-the-surprisingly-sexist-world-of-artificial-intelligence/ 2/7 3/5/2018 Inside the surprisingly sexist world of artificial intelligence — Quartz
I spoke with a few current and emerging female leaders in robotics and arti cial intelligence about how a preponderance of white men have shaped the elds—and what schools can do to get more women and minorities involved. Here’s what I learned:
1. Hawking’s offhand remark about women is indicative of the gender stereotypes that continue to ourish in science.
“Our culture perpetuates the idea that women are somehow mysterious and unpredictable and hard to understand,” says Marie desJardins, a professor of computer science at the University of Maryland, Baltimore County whose research focuses on machine learning and intelligent decision-making. “I think that’s nonsense. People are hard to understand.”
But outdated ideas about gender tend to take root in spaces that lack a diversity of perspectives. That’s certainly the case in computer science. Women received just 18% of undergraduate computer-science degrees in 2011, according to the most recent data available from the National Center for Education statistics.
That statistic is even more depressing when you consider that the number of women in computer science has been declining for the past few decades. In 1985, 37% of computer-science degrees went to women.
2. Fewer women are pursuing careers in arti cial intelligence because the eld tends to de-emphasize humanistic goals.
Over the course of desJardins’ multi-decade career, she’s noticed that arti cial intelligence research has drifted from a focus on how the technology can improve people’s lives.
“I was at a presentation recently where we were talking about the types of goals that people have for their careers,” she says. “There’s a difference between agentic China’s party summit, Oscar winners, penguin supercolony. All this and more in today's Daily Brief. https://qz.com/531257/inside-the-surprisingly-sexist-world-of-artificial-intelligence/ 3/7 3/5/2018 Inside the surprisingly sexist world of artificial intelligence — Quartz goals, which have to do with your personal goals and your desire to be intellectually challenged, and communal goals, which involve working with other people and solving problems.”
In general, many women are driven by the desire to do work that bene ts their communities, desJardins says. Men tend to be more interested in questions about algorithms and mathematical properties. Since men have come to dominate AI, she says, “research has become very narrowly focused on solving technical problems and not on the big questions.”
3. There may be a link between the homogeneity of AI researchers and public fears about scientists who lose control of superintelligent machines.
From Dr. Frankenstein to Jurassic Park’s misguided geneticists, popular culture is full of stories about scientists so absorbed by their creations that they neglect to consider the consequences for mankind. Today, arti cial intelligence and robotics are faced with a more realistic version of the problem. Scientists’ homogeneity may lead them to design intelligent machines without considering the effect on people different from themselves.
Tessa Lau, a computer scientist and cofounder of robotics company Savioke, says she frequently sees designs that neglect to take into account the perspectives of women and other minority groups.
Back when she worked for robotics research lab Willow Garage, she encountered “this enormous robot research prototype called PR2,” Lau recalls. “It weighs hundreds of pounds—it’s much larger than a smaller woman—and it has two big arms. It looks really scary. I didn’t even want one of those things near me if it wasn’t being controlled properly.”
China’s party summit, Oscar winners, penguin supercolony. All this and more in today's Daily Brief. https://qz.com/531257/inside-the-surprisingly-sexist-world-of-artificial-intelligence/ 4/7 3/5/2018 Inside the surprisingly sexist world of artificial intelligence — Quartz
At Savioke, Lau has made it her mission to design robots that are accessible to a diverse range of people. The Relay service robot, already in use at ve hotels including the Holiday Inn Express in Redwood City and Aloft locations in Cupertino and Silicon Valley, brings towels, snacks and other deliveries straight to guests’ doors. The robots come with a touch screen that’s easy to reach for children and people in wheelchairs, and obligingly perform what Lau calls a “happy dance” if they receive a ve-star satisfaction rating from guests.
“For most people, that’s the rst time they’ve had a robot come to the door,” Lau says. “We want the experience to be a pleasant one.”
4. To close the diversity gap, schools need to emphasize the humanistic applications of arti cial intelligence.
When Olga Russakovsky came up with the idea for the world’s rst arti cial intelligence summer camp for girls this year, she knew the key to holding campers’ interest would be to demonstrate how robots can bene t society.
“The way we teach technology and AI, we start from ‘Let us teach you how to code,” says Russakovsky, a postdoctoral research fellow at the Carnegie Mellon Robotics Institute. “So we wind up losing people who are interested in more-high level goals.”
At the Stanford Arti cial Intelligence Laboratory Outreach Summer, or SAILOR, 24 high-school students delved into research projects about practical applications of AI. One group learned about how natural language processing can be used to scan tweets during natural disasters, identifying the areas in need of relief. Another section focused on how computer vision can improve hospital sanitation by monitoring doctors’ hand-washing habits, while other campers researched self- driving cars and how computational biology could lead to cures for cancer.
China’s party summit, Oscar winners, penguin supercolony. All this and more in today's Daily Brief. https://qz.com/531257/inside-the-surprisingly-sexist-world-of-artificial-intelligence/ 5/7 3/5/2018 Inside the surprisingly sexist world of artificial intelligence — Quartz
“The camp aims to teach AI from a top-down perspective,” Russakovsky says, “and show kids, ‘here are some of the ways it can really be useful.’”
5. A number of women scientists are already advancing the range of applications for robotics and arti cial intelligence.
Cynthia Breazeal, who leads the personal robots group at MIT’s Media Labs, is the founder of Jibo—a crowd-funded robot for the home that can snap family photos, keep track of to-do lists and upcoming events, tell stories that keep the kids entertained and learn its owners’ preferences in order to become an ever-more indispensable part of their lives.
Also at MIT, aeronautics and astronautics professor Julie Shah has created kinder, gentler factory robots that can learn to cooperate with and adapt to their human colleagues. The director of MIT’s Computer Science and Arti cial Intelligence Laboratory, Daniela Rus, recently debuted a silicone robotic hand that can recognize objects by touch—a trait that could bring us one step closer to household robots that can help out in the kitchen.
Both Rus and Stanford Arti cial Intelligence Lab director Fei-Fei Li are overseeing research labs funded by Toyota that aim to create “intelligent” cars that can reduce accidents while keeping humans in the driver’s seat. And Martha Pollack, a provost and computer science professor at the University of Michigan, works on robotic assistants for the elderly and people with Alzheimer’s, dementia and other cognitive disabilities.
6. Robotics and arti cial intelligence don’t just need more women—they need more diversity across the board.
That means stepping up efforts to attract people from a wide range of racial and ethnic backgrounds as well as people with disabilities.
China’s party summit, Oscar winners, penguin supercolony. All this and more in today's Daily Brief. https://qz.com/531257/inside-the-surprisingly-sexist-world-of-artificial-intelligence/ 6/7 3/5/2018 Inside the surprisingly sexist world of artificial intelligence — Quartz
“Cultural diversity is big too,” says Heather Knight, a doctoral student at Carnegie Mellon’s Robotics Institute and founder of Marilyn Monrobot Labs in New York City. “Japan thinks robots are great. There’s no Terminator complex about arti cial intelligence taking over the world.”
Differences in cultural attitudes can prompt countries to carve out niches in particular areas. The US has a heavy focus on arti cial intelligence in the military, Knight says. And Europeans tend to be particularly interested in applications that support people with disabilities and the elderly.
“It’s important that we’re all talking to each other,” Knight says, “so ideas that are in the minority in one place can get fostered somewhere else.”
This post originally identi ed the Relay service robot as SaviONE, an earlier alpha version of the robot.
We welcome your ideas at [email protected].
China’s party summit, Oscar winners, penguin supercolony. All this and more in today's Daily Brief. https://qz.com/531257/inside-the-surprisingly-sexist-world-of-artificial-intelligence/ 7/7 3/5/2018 These Artificial Intelligence Startups Want to Fix Tech's Diversity Problem | WIRED
SIMON CHANDLER BACKCHANNEL 09.13.17 06:45 AM THE AI CHATBOT WILL HIRE YOU NOW
SUBSCRIBE
ALEUTIE/ISTOCK
Eyal Grayevsky has a plan to make Silicon Valley more diverse. Mya Systems, the San Francisco-based artificial intelligence company that he cofounded in 2012, has built its strategy on a single idea: Reduce the influence of humans in recruiting. “We’re taking out bias from the process,” he tells me.
Simon Chandler is a freelance journalist covering tech, politics, and music. ——— Sign up to get Backchannel's weekly newsletter, and follow us on Facebook and Twitter. Free Articles Left This Sign In or Register if you're already Get unlimited access. Try 3 close 3 Month a subscriber. months free. https://www.wired.com/story/the-ai-chatbot-will-hire-you-now/ 1/11 3/5/2018 These Artificial Intelligence Startups Want to Fix Tech's Diversity Problem | WIRED They’re doing this with Mya, an intelligent chatbot that, much like a recruiter, interviews and evaluates job candidates. Grayevsky argues that unlike some recruiters, Mya is programmed to ask objective, performance-based questions and avoid the subconscious judgments that a human might make. When Mya evaluates a candidate’s resume, it doesn’t look at the candidate’s appearance, gender, or name. “We’re stripping all of those components away,” Grayevsky adds.
Though Grayevsky declined to name the companies that use Mya, he says that it’s currently used by several large recruitment agencies, all of which employ the chatbot for “that initial conversation.” It filters applicants against the job’s core requirements, learns more about their educational and professional backgrounds, informs them about the specifics of the role, measures their level of interest, and answers questions on company policies and culture.
Everyone knows that the tech industry has a diversity problem, but attempts to rectify these imbalances have been disappointingly slow. Though some firms have blamed the “pipeline problem,” much of the slowness stems from recruiting. Hiring is an extremely complex, high-volume process, where human recruiters—with their all-too-human biases—ferret out the best candidates for a role. In part, this system is responsible for the uniform tech workforce we have today. But what if you could reinvent hiring—and remove people? A number of startups are building tools and platforms that recruit using artificial intelligence, which they claim will take human bias largely out of the recruitment process.
Another program that seeks to automate the bias out of recruiting is HireVue. Using intelligent video- and text-based software, HireVue predicts the best performers for a job by extracting as many as 25,000 data points from video interviews. Used by companies like Intel, Vodafone, Unilever and Nike, HireVue’s assessments are based on everything from facial expressions to vocabulary; they can even measure such abstract qualities as candidate empathy. HireVue's CTO Loren Larsen says that through HireVue, candidates are “getting the same shot regardless of gender, ethnicity, age, employment gaps, or college attended.” That’s because the tool applies the same process to all applicants, who in the past risked being evaluated by someone whose judgement could change based on mood and circumstance.
MORE FROM THIS EDITION
STEVEN LEVY Brain-Machine Interface Isn't Sci-Fi Anymore
JEREMY HSU Africa’s Delivery Drones Are Zipping Past the US
GLENN FLEISHMAN Meet the Font Detectives Who Ferret Out Fakery
STEVEN LEVY Apple is Defying History With Its Pricey iPhone X
Free Articles Left This Sign In or Register if you're already Get unlimited access. Try 3 close 3 Month a subscriber. months free. https://www.wired.com/story/the-ai-chatbot-will-hire-you-now/ 2/11 3/5/2018 These Artificial Intelligence Startups Want to Fix Tech's Diversity Problem | WIRED Though AI recruiters aren’t widely used, their prevalence in HR is increasing, according to Aman Alexander, a Product Management Director at consultancy firm CEB, which provides a wide range of HR tools to such corporations as AMD, Comcast, Philips, Thomson Reuters, and Walmart. “Demand has been growing rapidly,” he says, adding that the biggest users aren’t tech companies, but rather large retailers that hire in high volumes. Meaning that the main attraction of automation is efficiency, rather than a fairer system.
Yet the teams behind products such as HireVue and Mya believe that their tools have the potential to make hiring more equitable, and there are reasons to believe them. Since automation requires set criteria, using an AI assistant require companies to be conscious of how they evaluate prospective employees. In a best- case scenario, these parameters can be constantly updated in a virtuous cycle, in which the AI uses data it has collected to make its process even more bias-free.
Of course, there’s a caveat. AI is only as good as the data that powers it—data that’s generated by messy, disappointing, bias-filled humans.
TRENDING NOW
Science Neuroscientist Anil Seth Answers Neuroscience Questions From Twitter
Dig into any algorithm intended to promote fairness and you’ll find hidden prejudice. When ProPublica Free Articles Left This Sign In or Register if you're already examined police tools that predictG recti udinvlimsmit readt eascce, respsor. Tterrys 3found that the a lgorithm was biased agacilnosste 3 Month a subscriber. months free. https://www.wired.com/story/the-ai-chatbot-will-hire-you-now/ 3/11 3/5/2018 These Artificial Intelligence Startups Want to Fix Tech's Diversity Problem | WIRED African Americans. Or there’s Beauty.AI, an AI that used facial and age recognition algorithms to select the most attractive person from an array of submitted photos. Sadly, it exhibited a strong preference for light- skinned, light-haired entrants.
Even the creators of AI systems admit that AIs aren’t free of bias. “[There’s a] huge risk that using AI in the recruiting process is going to increase bias and not reduce it,” says Laura Mather, founder and CEO of AI recruitment platform Talent Sonar. Since AI is dependent on a training set generated by a human team, it can promote bias rather than eliminating it, she adds. Its hires might be “all be smart and talented, but are likely to be very similar to one another.”
And because AIs are being rolled out to triage high-volume hires, any bias could systematically affect who makes it out of a candidate pool. Grayevsky reports that Mya Systems is focusing on such sectors like retail, “where CVS Health are recruiting 120,000 people to fill their retail locations, or Nike is hiring 80,000 a year.” Any discrimination that seeps into the system would be practiced on an industrial scale. By quickly selecting say, 120,000 applicants from a pool of 500,000 or more, AI platforms could instantaneously skew the applicant set that makes it through to a human recruiter.
RELATED STORIES
ALEXIS SOBEL FITTS When Companies Get Serious About Diversity, They Call Her
KAREN WICKRE Surviving as an Old in the Tech World
MIRANDA KATZ This Twenty-Something Forced Silicon Valley to ‘Show Her the Numbers’
Then again, the huge capacity has a benefit: It frees up human recruiters to focus their energy on making well-informed final decisions. “I’ve spoken to thousands of recruiters in my life; every single one of them complains about not having enough time in their day,” Grayevsky says. Without time to speak to every candidate, gut decisions become important. Even though AI allows recruiters to handle greater volumes of candidates, it might also give recruiters the time to move from snap judgements.
Avoiding those pitfalls requires that engineers and programmers be hyper-aware. Grayevsky explains that Mya Systems “sets controls” over the kinds of data Mya uses to learn. That means that Mya’s behavior isn’t generated using raw, unprocessed recruitment and language data, but rather with data pre-approved by Mya Systems and is clients. This approach narrows Mya’s opportunity to learn prejudices in the manner of Tay—a chatbot that was released into the wilds by Microsoft last year and quickly became racist, thanks to trolls. This approach doesn’t eradicate bias, though, since any pre-approved data reflects the inclinations and preferences of the people selecting.
Free Articles Left This Sign In or Register if you're already Get unlimited access. Try 3 close 3 Month a subscriber. months free. https://www.wired.com/story/the-ai-chatbot-will-hire-you-now/ 4/11 3/5/2018 These Artificial Intelligence Startups Want to Fix Tech's Diversity Problem | WIRED This is why it’s a possibility that rather than eliminating biases, AI HR tools might perpetuate them. “We try not to see AI as a panacea,” says Y-Vonne Hutchinson, the executive director of ReadySet, an Oakland-based diversity consultancy. “AI is a tool, and AI has makers, and sometimes AI can amplify the biases of its makers and the blindspots of its makers.” Hutchinson adds that in order for tools to work, “the recruiters who are using these programs [need to be] trained to spot bias in themselves and others.” Without such diversity training, the human recruiters just impose their biases at a different point in the pipeline.
Some companies using AI HR tools are wielding them expressly to increase diversity. Atlassian, for example, is one of the many customers of Textio, an intelligent text editor that uses big data and machine learning to suggest alterations to a job listing that make it appeal to different demographics. According to Aubrey Blanche, Atlassian’s global head of diversity and inclusion, the text editor helped the company increase the percentage of women among new recruits from 18 percent to 57 percent.
“We’ve seen a real difference in the gender distribution of the candidates that we’re bringing in and also that we’re hiring,” Blanche explains. One of the unexpected benefits of using Textio is that, on top of diversifying Atlassian’s applicants, it made the company self-aware of its corporate culture. “It provokes a lot of really great internal discussion about how language affects how our brand is seen as an employer,” she says.
Ultimately, if AI recruiters result in improved productivity, they’ll become more widespread. But it won’t be enough for firms to simply adopt AI and trust in it to deliver fairer recruitment. It’s vital that the systems be complemented by an increasing awareness of diversity. AI may not become an antidote to the tech industry’s storied problems with diversity, but at best it might become an important tool in Silicon Valley’s fight to be better.
#BACKCHANNEL #ARTIFICIAL INTELLIGENCE #DIVERSITY #DIVERSITY IN TECH #WORK
VIEW COMMENTS
SPONSORED STORIES
POWERED BY OUTBRAIN
BABBEL App made by 100+ linguists gets you speaking a new language in 3 weeks - here's how!
TIME What the NAD+ Pill Could Mean for the Aging Process
Free Articles Left This Sign In or Register if you're already Get unlimited access. Try 3 close 3 Month PC WORLD a subscriber. 5 Reasons the Next Generationm ofon Andtrohids P fhroneees W. ill Be Hard to Beat https://www.wired.com/story/the-ai-chatbot-will-hire-you-now/ 5/11 3/5/2018 These Artificial Intelligence Startups Want to Fix Tech's Diversity Problem | WIRED
TOPHATTER It’s Like eBay, But Everything Sells in 90 Seconds.
AARP Love History? This Woman Turned That Into a Second Career
MORE BACKCHANNEL
BACKCHANNEL Mind Games: The Tortured Lives of ‘Targeted Individuals’ LAURA YAN
Free Articles Left This Sign In or Register if you're already Get unlimited access. Try 3 close 3 Month a subscriber. months free. https://www.wired.com/story/the-ai-chatbot-will-hire-you-now/ 6/11 3/5/2018 These Artificial Intelligence Startups Want to Fix Tech's Diversity Problem | WIRED
OUTRAGE TWEETING How Liberals Amped Up a Parkland Shooting Conspiracy Theory MOLLY MCKEW
SCI-FI How Ava DuVernay Became a Creator of Worlds ANGELA WATERCUTTER
Free Articles Left This Sign In or Register if you're already Get unlimited access. Try 3 close 3 Month a subscriber. months free. https://www.wired.com/story/the-ai-chatbot-will-hire-you-now/ 7/11 3/5/2018 These Artificial Intelligence Startups Want to Fix Tech's Diversity Problem | WIRED
BACKCHANNEL Inside the Mueller Indictment: A Russian Novel of Intrigue GARRETT M. GRAFF
GENE EDITING Crispr Will Transform the Way We Eradicate Invasive Species EMMA MARRIS
Free Articles Left This Sign In or Register if you're already Get unlimited access. Try 3 close 3 Month a subscriber. months free. https://www.wired.com/story/the-ai-chatbot-will-hire-you-now/ 8/11 3/5/2018 These Artificial Intelligence Startups Want to Fix Tech's Diversity Problem | WIRED
BACKCHANNEL How Sam Cossman Makes a Living Jumping into Volcanoes JOSH DEAN
GET BACKCHANNEL'S NEWSLETTER
We publish our biggest stories on Wednesdays. Be the first to know.
Enter your email
SUBMIT
FOLLOW US ON TWITTER
Backchannel is moving to Wired! Here's what that means: http://trib.al/Ar1TZSg
FOLLOW
FOLLOW US ON FACEBOOK
Get Backchannel's latest stories and live videos.
FOLLOW
Free Articles Left This Sign In or Register if you're already Get unlimited access. Try 3 close 3 Month a subscriber. months free. https://www.wired.com/story/the-ai-chatbot-will-hire-you-now/ 9/11 3/5/2018 These Artificial Intelligence Startups Want to Fix Tech's Diversity Problem | WIRED
SUBSCRIBE ADVERTISE
SITE MAP PRESS CENTER
FAQ ACCESSIBILITY HELP
CUSTOMER CARE CONTACT US
SECUREDROP T-SHIRT COLLECTION
NEWSLETTER WIRED STAFF
JOBS RSS
CNMN Collection
Use of this site constitutes acceptance of our user agreement (effective 3/21/12) and privacy policy (effective 3/21/12). Affiliate link policy. Your California privacy rights. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of Condé Nast.
Free Articles Left This Sign In or Register if you're already Get unlimited access. Try 3 close 3 Month a subscriber. months free. https://www.wired.com/story/the-ai-chatbot-will-hire-you-now/ 10/11